I work for Toshiba, so I go to Japan for both play and work. Why did I avoid Azubu and the gaijin strongholds cause as you stated they seem to be full of the F-tards you spoke of... The guys that want to bag themselves a Japanese woman and the woman who go hang out there so they can meet a gaijin. To me those locations are like training wheels for gaijin...
Yes, there is good shopping to be had in those areas but seriously there is no shortage of good shopping in Tokyo...the Japanese thrive on it. Omote Sando/Harajuku, Shimokitazawa, Ginza...Kyoto has tons of great places, Osaka, etc. Pretty much every city in Japan has a great shopping arcade.
Those areas also seem much dirtier to me as many of the people are gaijin or US Military out for fun, they bring the same bad habits and attitudes have have in their respective nation.
It's kind of the same reason I have no urge to go to Australia, not that I don't like Australians, Great people! It's just that it seems very much like the US. I go to places to experience different things (food, fashion, environment, culture, and people) Going to a place that looks and feels just like SoCal but with a Aussie accent?? Pass, I could go to NY for less and still get a different accent.
That's me though, hope that covered it