[Nyororin]

I also don`t get why this is a frustrating question.
It`s no different than moving to a new town and being asked "What brought you here?", etc. People are curious about why someone new came to their area. When you`re clearly different in appearance, it`s just easier for someone to know that you`re not from around there.
Japanese people ask other Japanese people this question ALL THE TIME when they find out that someone is from another area. The thing is, for foreigners, you`re always not from around there.

It`s small talk. People aren`t looking for your life story. The only time I can see it being stressful is if the reason you are there is either stressful or embarrassing to talk about.