You need to read again the thing that was described, more carefully. Imagine for example that by “a page,” the person means a page called /juicy-content or something.
Comment on Microsoft and Reddit Are Fighting About Why Bing’s Crawler Is Blocked on Reddit
Moonrise2473@feddit.it 2 months agoonly if you don’t want any visits except from yourself, because this removes your site from any search engine
should write a “disallow: /juicy-content” and then block anything that tries to access that page (only bad bots would follow that path)
mozz@mbin.grits.dev 2 months ago
Miaou@jlai.lu 2 months ago
That’s exactly what was described…?
Moonrise2473@feddit.it 2 months ago
Oops. As a non-native English speaker I misunderstood what he meant. I understood wrongly that he set the server to ban everything that asked for robots.txt
Zoop@beehaw.org 2 months ago
Just in case it makes you feel any better: I’m a native English speaker who always aced the reading comprehension tests back in school, and I read it the exact same way. Lol! I’m glad I wasn’t the only one. :)