some good advice in this thread... gotta admit I have had the same problem (I just tell them I am German and speak very little English which has always worked).
It can be a blessing and a curse to be foreign in this country... as an English native, it is VERY easy to make friends as so many people want to practice English, but also quite annoying when you want to relax in a coffee shop and people start talking to you in English when it is the last thing you want lol