I found Japan to be one of the safest place I have ever been to
EVER!!
In the whole time I was there, the only place that I felt uncomfortable was in Roppongi, it was like being in a bad part of England again.
I hope the experience didn't ruin your perception of Japan and mess things up too much for you.