I usually have one hummingbird feeder near a thistle seed feeder, but I have noticed that if the hummers are feeding the other birds aren't around, and vice versa. Hummers have nearly attacked me before, flying within inches of my eyes and backing off just as quickly, so I suspect your feathered friends are a little intimidated by the pint size and speedy little bullies on the block. Some distance would probably do both the hummers and the other birds some good. I've never seen any aggressive behaviour turn dangerous, but it would certainly make the dining experience more stressful than most seed eaters would like.