I think you have to differentiate between what people say out loud and what they really think when it comes to anime and manga. To be honest I think a lot of people say they dislike it just to "fit in", when secretly they quite enjoy it. It's the public stigma they want to steer clear of, not anime/manga itself.
Back in high school I was part of the so-called "popular crowd" and we used to make fun of the geeks in our class who wore star trek t-shirts and read spiderman magazines and stuff (very bad of me, I know
). Then I started dating a guy with all kinds of nerdy interests and my opinion of what it meant to be a geek immediately changed. He was one of the most open-minded and intellectual people I've ever met (and good-looking too!!) and I've had a thing for nerdy guys ever since.
~annelie