I think i get what you are trying to say and i have talked with many people who have experiences like this, some of which have never watched an anime in their life. Ask yourself how much do these haters actually know about Japan other what they've heard or anime they happen to flip past on tv. Not everyone is going to be as intelligent and open minded as you and i have learned this painfully over the years. Im pretty much immune to most ignorance like this.
You shouldn't blame anime though as its a form of entertainment unique to Japan. It should not be the only focus but most of the time its all these people know. I mentioned to a co-worker that i was interested in Japanese tea ceremony and he gave the dirtiest look and it didn't bother me at all.
The best advice i could give is really to ignore them. They know nothing about you or the culture obviously and have no will to learn its not worth your time or aggravation get into with them in the end you will be the one who ends up getting upset and angry. Just focus on what you want and spend time with more like minded and open people.
Anime isn't the problem as usual people are the problem which is why my hope for mankind dwindles a little each day lol
