Support for different pickle backend?

[landaire]

Just wanted to throw this out there since I saw there's a custom pickle backend, but I've made a lot of adjustments to the serde_pickle crate to facilitate my own needs here:

1. https://docs.rs/pickled/latest/pickled/
2. https://github.com/landaire/pickled

I have an issue in serde_pickle here describing some of the changes: birkenfeld/serde-pickle#34

But tl;dr I fixed memo'd objects, implemented refcounting, and support for custom objects (kind of). Right now custom objects just default to a dictionary (see: https://github.com/landaire/pickled/blob/da6e72ba4ea3d5b4dd87d8c269e918321fab564c/src/de.rs#L1176-L1193) but technically I don't think anything is stopping this from being extendable with custom callbacks for constructing an object implementing some interface.

I am not a pickle expert and semi-accidentally was forced to do these things for consuming a pickled data file outside of my control. But from my own experience I was able to significantly reduce deserialization time and memory overhead (refcounting helps a LOT here) and get a complete picture of the original object.

I'm really not sure if any of these changes would benefit this project, but again, just wanted to mention it.

[landaire]

I actually had to add support for completely custom objects for my other project, so 2.0.0-alpha4 now supports typed object construction and I implemented a bunch of fixes for difference in type comparisons between Rust / Python VM.

cc @LaurentMazare, I apologize for the ping, but seeing as you've done a lot of the work on the pickle module here I wanted to flag to you.

[ferris-joy]

This is super interesting, thanks for sharing this.

The refcounting and memoized object support especially sound huge. I can imagine how much memory that saves once you start dealing with large object graphs or real-world pickled datasets. The custom object construction support in 2.0.0-alpha4 also sounds really nice — mapping everything into generic dictionaries only gets you so far.

Also totally relate to the “accidentally became a pickle expert because production forced it” situation 😅

Really appreciate you pointing this out. I’m definitely going to dig through the implementation and issue thread because there’s a lot of useful work here that could help improve compatibility and performance quite a bit.