yeah those weeaboo comments piss me off too, but I agree with a previous comment..tougher skin is definitely a must. i mean I'm actually going to Japan this summer, to study for 2 years and i have a pretty good mastery of the language and I still get called that. I admit i do love Japanese pop culture and language and history, but no one understands that I want to do something different with my life. i feel your pain. People are so stupid.
as long as its not an obsession to the point that you go crazy from addiction,then i think you are in good shape.