Comment on Pandas

Kausta@lemm.ee ⁨1⁩ ⁨year⁩ ago

• thisfro@slrpnk.net ⁨1⁩ ⁨year⁩ ago

  I raise you thousands of gzipped files (total > 20GB) combined into one dataframe. Frankly, my work laptop did not like it all that much. But most basic operations still worked fine tho

  source
• QuizzaciousOtter@lemm.ee ⁨1⁩ ⁨year⁩ ago

  I really don’t think that’s a lot either. Nowadays we routinely process terabytes of data.

  source

  • Kausta@lemm.ee ⁨1⁩ ⁨year⁩ ago

    Yeah, it was just a simple example. Although using just pandas (without something like dask) for loading terabytes of data at once into a single dataframe may not be the best idea, even with enough memory.

    source
• whotookkarl@lemmy.world ⁨1⁩ ⁨year⁩ ago

  It’s good to see the occult is still alive and well

  source