When I lived in Japan, if I was out with my wife people would always speak to her (she was Japanese). My Japanese was reasonably good, and my wife said that there were no real issues with my grammar, vocabulary, or pronunciation, but still people didn’t understand me because I was a foreigner and they knew they couldn’t understand foreigners.