Hrm... I went to Japan with members of my doujinshi/anime art club, have cosplayed on occasion and do enjoy anime/manga. I don't consider myself an otaku, but I do think that an anime fan with realistic expectations can go to Japan and have a fantastic time. I don't think it's any less of a legitimate reason for going to Japan than say, going to China because of interest in kung fu (which I've also done
), as long as the person is approaching it in a realistic and intelligent way.
The problem is the number of self-proclaimed otaku, mainly quite young, who know next to nothing about Japan outside of what they've seen in anime and think its some kind of wonderland.