Comment

Comment on How did Google discover public links of Claude chats?

Ah ok, looks like I didn’t read the article carefully enough.

That still leaves two scenarios though:

Claude fucked up, and gave Google a way to index public links of people. After this news broke, they asked Google (and other search engines) to not show those urls anymore.
People fucked up and/or fell victim to Google’s the data hunger, which let Google to index their public links, after which Claude asked Google not to index such links.

I guess, in the end, I’m mostly wondering if the long random strings of public links are generally “brute-force-discovered” by crawlers like Google, or whether “discovery” of such links requires some kind of breach.

source

Sort:hotnew top

SnoopSqueak@lemmy.today ⁨3⁩ ⁨days⁩ ago

Claude fucked up, and gave Google a way to index public links of people.

Anthropic fucked up, Claude is not a person.

After this news broke, they asked Google (and other search engines) to not show those urls anymore.

Correct, that seems to be the case.

People fucked up and/or fell victim to Google’s the data hunger, which let Google to index their public links

Kind of. The user fuck up would be not realizing that Anthropic had no protections from search engines, and that by clicking “share” they were creating publicly accessible links.

after which Claude asked Google not to index such links.

Anthropic seems to have updated the domain those links are generated on so that crawlers do not index public chats, yes.

I guess, in the end, I’m mostly wondering if the long random strings of public links are generally “brute-force-discovered” by crawlers like Google, or whether “discovery” of such links requires some kind of breach.

I am also curious. I looked it up. developers.google.com/search/…/how-search-works

The first stage is finding out what pages exist on the web. There isn’t a central registry of all web pages, so Google must constantly look for new and updated pages and add them to its list of known pages. This process is called “URL discovery”. Some pages are known because Google has already visited them. Other pages are discovered when Google extracts a link from a known page to a new page: for example, a hub page, such as a category page, links to a new blog post.

If I had to guess based on this, Google had probably indexed the domain Claude or its frontend run on and automatically scooped up the public chats as they became available.

It is also possible that any Claude chat shared will still be accessible to web crawlers, if they just ignore the flags telling them not to scrape, crawl, access, index, etc.

source
- KurtVonnegut@mander.xyz ⁨2⁩ ⁨days⁩ ago
  Thank you this was very helpful.
  
  source