[GoNative]

There's only about 100,000 westerners living in Japan. This is only around 1 westerner for every 1,300 Japanese. It's probably considerably higher in places like Tokyo and considerably less in much of the rural parts of the country. So yes you can stand out and many people have had little contact with westerners (in the flesh). It's certainly very different to Australia where I'm originally from where about 1 in every 4 people were born overseas.
If people actively avoid contact with you, which I've experienced on trains before, I don't really see it as racism. Just ignorance and the fear associated with that. It's a bit like if you got on a train and there's some old guy in a dress you're probably unlikely to rush over to sit next to him because he's obviously a bit weird. In Japan in many areas we foreigners are considered weird. It's because we're not Japanese and it is based on our race but it's not malicious in anyway.