Comment on (In Python) Can you save an object that is in memory to disk and reload it at a later time?
UnfortunateShort@lemmy.world 1 month ago
I think pickle is what you want.
Keep in mind that this might have a huge performance impact if you do it all the time - it’s still IO even when it’s not parsing.
spacemanspiffy@lemmy.world 1 month ago
My idea would be to load one larger file one time and not parse anything, and keep it in memory the entire time. Versus what it does now which is load the files and parse them and keep everything in memory.
But three people responding here so far with “pickle” so maybe that is the way.
UnfortunateShort@lemmy.world 1 month ago
You can stuff all the info into an object and use it this way, no problem. I just wanted to point out that this doesn’t have zero performance impact compared to what you currently have.
So (depending on how your OS caches files) you might not want to do this like twice in a lambda that you pass to an iterator over a huge slice or something.