Comment on Black Mirror AI

<- View Parent
Zexks@lemmy.world ⁨1⁩ ⁨week⁩ ago

Lol. And they’ll delist you. Unless you’re really important, good luck with that.

robots.txt

Disallow: /some-page.html

If you disallow a page in robots.txt Google won’t crawl the page. Even when Google finds links to the page and knows it exists, Googlebot won’t download the page or see the contents. Google will usually not choose to index the URL, however that isn’t 100%. Google may include the URL in the search index along with words from the anchor text of links to it if it feels that it may be an important page.

source
Sort:hotnewtop