I know a handful of white Americans who at a glance you’d never guess they’re highly educated polyglots and would meet the requirements for entering this restaurant. The way the sign is worded, I think you’re right, only fluent Japanese people allowed here because we only speak Japanese here.