There are all sorts of filters:

• Expensive clubs. Members only associations like country clubs can skew towards the ultra rich. Yacht clubs and polo clubs are kinda an extreme version of this, but there are all sorts of organizations where the membership can be assumed to be rich.
• Expensive hobbies. Wine tasting, skiing, golfing, boating, horse stuff, biking, and traveling/vacations can range from the slightly expensive to prices that only the ultra rich can afford.
• Related to both of the above, expensive places. If you’re skiing in an expensive resort town, and hanging out in the lobby of a $2000/night hotel, you’ll probably only see employees of these places or other very rich people. Some have even layers beyond that, like an exclusive members only club in an expensive area, or a separate lounge for only people lodging in the most expensive rooms in the hotel. Or if you’re at a private jet airport, and weather causes delays and cancellations, standing around in the terminal might allow you to mingle with other private jet people. Or if you live in a crazy expensive neighborhood or building, your neighbors are pretty much guaranteed to be rich.
• Third party verification. Networking, introduction by mutual friends/acquaintances, even social media or dating apps where you have to prove your status/wealth.

It’s not all or nothing, either. Some places have a disproportionately high number of rich people but aren’t necessarily exclusive to the rich (private schools, certain types of clubs, certain types of activities/hobbies, public parks/restaurants/libraries/museums in rich areas). So a lot of rich people do mingle with the middle class, but often will feel comfortable letting their guard down more or less in particular places or in particular groups.