feat(cache): add cache_memory, an in-memory builder-driven cache - #27
Merged
Conversation
ptomecek
force-pushed
the
feat/in-memory-builder-cache
branch
2 times, most recently
from
August 28, 2026 17:09
605c65f to
26c88fc
Compare
Adds `cache_memory`, a third caching primitive alongside `cache` and `cache_parquet`, closing the gap described in issue #26: an in-memory, builder-driven cache that does not require the plan to be serializable. `cache` derives its key by serializing the plan, so it cannot key a LazyFrame backed by `register_io_source` whose generator closes over non-picklable state (a connection, a lock, an open iterator). `cache_parquet` accepts a builder + schema and needs no serialization, but only has a disk backend. `cache_memory` fills the remaining corner: builder-callable input, no plan serialization, in-memory backend. It collects the builder at most once into a closure-held buffer and replays it on every reference or collect, so a source read from many branches of a multi-branch evaluation (including `pl.collect_all`) executes upstream exactly once instead of once per reference. There is no key: reuse happens through the returned frame's identity, and the buffer is reclaimed by garbage collection when that frame is dropped. Semantics: - Construction and `collect_schema()` (with a callable schema) do no I/O and no collect; the builder runs only on first row demand. - The one-time build is thread-safe; concurrent first-collects fanned out across `pl.collect_all` share a single build. - The build outcome is terminal: any failure is recorded and re-raised on subsequent collects (as a neutral error carrying the original type and message), so a failing builder is not re-run per branch. Retry by constructing a fresh `cache_memory`. - The collected frame is reconciled against the declared schema once (missing column or dtype mismatch raises; extras are dropped). Exposed as a top-level function and as `lf.piot.cache_memory(schema=...)`. Adds unit tests and documents the primitive in the API reference and query-optimization guides. Signed-off-by: Pascal Tomecek <40371786+ptomecek@users.noreply.github.com> Co-authored-by: hintse <5867507+hintse@users.noreply.github.com>
ptomecek
force-pushed
the
feat/in-memory-builder-cache
branch
from
August 28, 2026 17:13
26c88fc to
c4462cd
Compare
Contributor
ptomecek
marked this pull request as ready for review
August 31, 2026 14:41
hintse
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
cache_memory, a third caching primitive alongsidecacheandcache_parquet, closing the gap described in #26: an in-memory, builder-driven cache that does not require the plan to be serializable.cacheLazyFrame(plan)cache_parquetschema)cache_path(explicit)cache_memoryschema)cachederives its key by serializing the plan, so it cannot key aLazyFramebacked byregister_io_sourcewhose generator closes over non-picklable state (a connection, a lock, an open iterator).cache_parquetaccepts a builder + schema and needs no serialization, but only has a disk backend.cache_memoryfills the remaining corner: builder-callable input, no plan serialization, in-memory backend.Behaviour
It collects the builder at most once into a closure-held buffer and replays it on every reference or collect, so a source read from many branches of a multi-branch evaluation (including
pl.collect_all) executes upstream exactly once instead of once per reference. There is no key — reuse happens through the returned frame's identity, and the buffer is reclaimed by garbage collection when that frame is dropped.collect_schema()(with a callableschema) do no I/O and no collect; the builder runs only on first row demand.pl.collect_allshare a single build.cache_memory.API
cache_memory(build_or_lf, *, schema)lf.piot.cache_memory(*, schema)Test plan
tests/io_sources/test_lazy_cache_memory.py(23 cases): collect-once across references andpl.collect_allfan-out, thread-safe single build, buffer released on GC, non-serializable plugin source, callable-schema resolution without building, terminal/no-amplification failure semantics, no frame/traceback retention on failure, schema reconciliation, and empty-projection cardinality.ruff checkandruff format --checkclean.io_sourcessuite passes.Closes #26.