[MMM]

I'll just give this take on it.

I have a Japanese wife and a 4-year-old son and we live in the US, but visit Japan at least once a year. Even though we live in the US, we maintain a pretty much Japanese-speaking household. My son has Japanese friends (large population here) and American friends, and "swirled" friends like him.

He was almost a total Japanese speaker up until he turned 4, but now he has really pushed himself (all on his own) to speak more English. He elects to watch more English videos and, as a result, is more vocal with his English-speaking friends.

I would say give your child the chances to speak both. (At the very least, the grandparents will be happy) and keep it up. I know more than a couple Japanese/American people that say "I wish my parents spoke more Japanese to me when I was a kid", so we are making a real try to help him be bilingual.
It is cool, because now he say "How do you say X in English" or "How do you say X in Japanese" even though we never made a distinction to him (He must have noticed from people around him who can speak one or the other).