This is an automated archive made by the Lemmit Bot.
The original was posted on /r/technology by /u/No-Drawing-6975 on 2024-06-27 10:35:29+00:00.
Submitted 2 years ago by bot@lemmit.online [bot] to technology@lemmit.online
https://www.theverge.com/2024/6/25/24185984/reddit-robots-txt-fight-ai-bots-scraping-crawlers
The original was posted on /r/technology by /u/No-Drawing-6975 on 2024-06-27 10:35:29+00:00.