Japan is one of the few countries in the world (at least the 1st world) that doesn't have an immigration policy whatsoever. They have immigration laws of course but no actual policy about an annual intake of immigrants like most other countries. Currently still only about 1% of the population were born overseas making it very much a monocultural society.
This lack of interaction with other cultures does mean there is a great deal of ignorance when it comes to foreigners. And of course like anywhere ignorance can lead to unfounded fears and certainly right wing nationalist groups and political parties do play upon those fears quite a lot. This has led to things like racial profiling by police and often a general distrust of foreigners.
From a western viewpoint many even extremely nice Japanese people can hold very racist attitudes but in my experience these attitudes rarely come out of any sort of malice or hatred. It's mostly just ignorance. Many wouldn't even realise what they are saying could be conceived as racist.
Also on the whole Japanese people are extremely polite and even when they do express something that is blatantly racist it will be done in a reasonably polite way. It would be extremely rare for it to become violent or in your face at all.
In my more than 5 years here I have personally only ever experienced a couple of occassions of verbal racism directly at myself. It does annoy me how difficult it is for someone like myself to get permanent residency not being married to a Japanese national but I still absolutely love living here!
