Creating a torrent that includes all of humanity's knowledge/art/entertainment?

Submitted ⁨⁨11⁩ ⁨months⁩ ago⁩ by ⁨AnarchistsForDemocracy@lemmy.world⁩ to ⁨[deleted]⁩

I know we can’t do this with any copyrighted materials. But a lot of books, music, art, knowledge is in the creative commons. Is it possible to create one massive torrent that includes all that can be legally included and then have people only download what they actually want to enjoy?

source

Comments

Sort:hotnew top

jeffw@lemmy.world ⁨11⁩ ⁨months⁩ ago
So like… a meta-torrent that is a torrent of all the other torrents?

source
- litchralee@sh.itjust.works ⁨11⁩ ⁨months⁩ ago
  With the way that the BitTorrent v2 protocol works, each file of the original, underlying torrents wouldn’t have to be re-seeded, but rather would reuse each file’s individual hash and thus incorporate those files into the meta torrent without necessarily having to download or even upload any part of the meta torrent.
  
  That said, the .bittorrent file would be massive and might run up against certain limits in the current protocol.
  
  source
- Unsustainable@lemmy.today ⁨11⁩ ⁨months⁩ ago
  But don’t lose the list. Losing the list would be bad. We’ll need to keep the list in a safe place.
  
  source
aelwero@lemmy.world ⁨11⁩ ⁨months⁩ ago
I mean… That pretty much describes torrents period… What is the functional difference between hosting a single torrent with everything, and hosting a torrent per item?

If the expectation is that you only include files you want when downloading the torrent, you’re only going to be seeding that portion.

Seems like it would just make the search function harder, and make it harder to determine the “health” of individual items…

I don’t understand the benefit…

source
- AnarchistsForDemocracy@lemmy.world ⁨11⁩ ⁨months⁩ ago
  For example zlibrary is 220TB of books and scientific articles, that included in the torrent would be great along with all the stuff that is arts and music.
  
  Basically it would be a way to combat media vanishing off of the net over time. Basically a noah’s arche for all of mankinds knowledge.
  
  It would be great to have everything in one single spot to make it easier to contribute and get stuff. We’d also be more easily capable of combining our forces to maintain/create the thing.
  
  source
  - aelwero@lemmy.world ⁨11⁩ ⁨months⁩ ago
    That makes a great focal point for what I was saying actually ;)
    
    It’s 220TB, so youll have incredibly few people who download the whole torrent. Most will open the torrent list and select a small number of items from it to download. The files selected the most will get seeded frequently, the ones that never get selected by anyone will have only the originator seeding it (if they continue to do so).
    
    It’s functionally no different than if each individual file is a torrent… Except that the seeding info is going to be wonky on the single 220TB torrent because nobody is downloading it intact, only in pieces.
    
    It’s also much easier to find a specific file if it is it’s own torrent vs. one of a billion files in a single mega torrent.
    
    Just because you put it on an index in a torrent doesn’t mean the file still exists somewhere. That media can still vanish…
    
    What would do what you’re suggesting this torrent would do (which a torrent cannot actually do) is a Yottabyte capable computer somewhere storing all those files… You’d need that to keep the seeding intact for the whole torrent…
    
    source
    -> View More Comments
Romkslrqusz@lemm.ee ⁨11⁩ ⁨months⁩ ago
Not quite what you’re getting at, but the entirety of Wikipedia without images is available as a 20-30GB download: en.wikipedia.org/…/Wikipedia:Database_download

source
PrinceWith999Enemies@lemmy.world ⁨11⁩ ⁨months⁩ ago
I think there’s a handful of problems with the idea. For starters (I’m just going with the first returned result because the actual numbers don’t matter as much as the magnitudes), there’s around 64 zetabytes comprising the internet as of 2020, 64 trillion GB. That’s going to be one hell of a zip file. In fact, pretty much the only thing capable of storing that much information is, well, the Internet itself.

Second is the rate of information being produced. These estimates vary wildly, but the rate of growth is increasing exponentially. We will soon be writing more data per day to the internet than is currently there from the very beginning until now.

So maybe we don’t need every product page from every store website around the world. Maybe we don’t need the tens of millions of pages of corporate training manuals. Maybe we need curation rather than SELECT * FROM INTERNET.

That’s what things like Gutenberg and the Internet Archive do. They’re very limited in what they catch, of course. It’s also sort of what Wikipedia does, although curation here includes summarization. It’s also a feature of historical archives from existing media - like New York Times records that go back a century (or wherever they’re at now), or back issues of Nature and Science going back to when they started publication. Those are obviously doable - people are doing them - but each alone is a microscopic piece of the puzzle.

So, given that those exist, alongside the rest of the internet, what value are we creating? Storing something digitally doesn’t give it permanence, and I have an 8” floppy disk for a cash register POS created by an unknown OS to prove it.

Someday (hopefully soon) PDFs will go away and nothing will read them. Hell, the concept of “file” could go away in 50 years. There are written texts from thousands of years ago that we cannot read, and others we’ve deciphered only very recently and imperfectly. All of that archived stuff will have to be ported over, and again that’s going to mean yet more curation. At the rate information is growing you’re going to make Sisyphus look like he’s on a vacation in Tahiti.

Does that mean it’s all one big library of Alexandria? Not necessarily.

Rather than thinking of all those data as a library, think of them as an ecosystem of knowledge. Once Amazon goes out of business, no one’s going to care about that one page of theirs with the nose hair trimmer. We will still have a copy of the NYT when we landed on the moon, or when Nazi Germany was defeated. We’ll also have other information about space programs and 20th century history. We probably won’t have my mom’s recipes or all those pictures I’ve taken of my pets over the years, and my MySpace page is thankfully gone forever. I even deleted all of my Reddit content before moving on.

Maybe my scientific publications will end up archived someplace, but there we get into the tree falling in the forest problem. If no one reads them from now to the end of time, are they really there? Maybe physically, but they’ve sort of passed out of the ecosystem of human knowledge and are now part of the fossil record, if anything.

We’ve also researched how to communicate over millennia. There’s the (kind of silly but a little cool) Long Now project. We’ve also tried to invent symbology that will allow us to put warning signs outside hazardous/nuclear waste storage facilities that will continue to communicate “Danger - Do Not Enter” for tens of thousands of years.

In short, I think that the problem you’re trying to solve is impermanence or entropy, which both Buddhists and physicists will tell you aren’t things we’re going to solve.

source
kpw@kbin.social ⁨11⁩ ⁨months⁩ ago
You could seed the Torrents by the internet archive.

source
CmdrShepard@lemmy.one ⁨11⁩ ⁨months⁩ ago
Pretty sure this stuff already exists in some form. /r/datahoarder people would probably be able to steer you in the right direction though you may need to lay out several thousand for enough HDDs to hold it.

source
ElderWendigo@sh.itjust.works ⁨11⁩ ⁨months⁩ ago
You’re describing leeching from something like Anna’s Archive datasets.

source
skillissuer@discuss.tchncs.de ⁨11⁩ ⁨months⁩ ago
Scihub + libgen archives (torrent list) come pretty close, even that it’s not what you’re asking about (100TB? total)

source
rufus@discuss.tchncs.de ⁨11⁩ ⁨months⁩ ago
“All” is impossible. You’re going to miss something. And it’s a lot of work. Maybe have a look at the datasets people/researchers use to train Artificial Intelligence. I think some people put in the effort to compile large datasets with just freely licensed data.

source
- AnarchistsForDemocracy@lemmy.world ⁨11⁩ ⁨months⁩ ago
  
  it’s a lot of work
  
  so per your suggestion using for example the zlibrary book/paper repo and training sets of openai as starting point one could maybe get around the brunt of the work.
  
  source
  - rufus@discuss.tchncs.de ⁨11⁩ ⁨months⁩ ago
    Well, this depends a bit on what you’re trying to archieve, here. If you want to learn about machine learning, you can do other things than train an LLM. And I think there are some online courses out there. I’ve seen one on Huggingface (NLP Course) and services like Lambda have lots of examples and tools available. There are other platforms with material and courses available. I don’t however think LLMs are something you want to get your feet wet, with.
    
    If you want to create something useful, maybe taking a base/foundation model and fine-tuning it would be the best approach. I think training a halfway-decent model from zero takes like a 5 figure amount of money. I’m not an expert, I don’t really know. But teaching a model to speak a new language is something that can be done after the fact in fine-tuning. And this saves quite some computing time and requires a smaller dataset. That would be the way to “get around the brunt of the work”.
    
    If you just want something that speaks French, I think there are already some models out there you could just use. For example Vigogne and also the big ones like Llama2 (and perhaps Mistral) should be able to do it to some degree.
    
    So in conclusion, I don’t really know what you’re trying to do, so it is difficult to give advice. And machine learning, natural language processing etc isn’t super simple. You need the background knowledge, some experience with the different choices available and access to some computing resources to do it on your own. Fortunately AI is a hype now. There is traditional literature available (aka. books), courses, and even tutorials and YouTube videos that explain details or the process how to fine-tune something. And most of the scientific papers are available to us, nowadays and they often contain useful details about dataset size and stuff like that. I think training something from zero is something few people do. Especially as a hobby.
    
    source
TropicalDingdong@lemmy.world ⁨11⁩ ⁨months⁩ ago
maybe

source
PeachMan@lemmy.world ⁨11⁩ ⁨months⁩ ago
No

source
Synthead@lemmy.world ⁨11⁩ ⁨months⁩ ago
Yes

source