Skip to content

feat(cache): add cache_memory, an in-memory builder-driven cache - #27

Merged
ptomecek merged 1 commit into
mainfrom
feat/in-memory-builder-cache
Aug 31, 2026
Merged

feat(cache): add cache_memory, an in-memory builder-driven cache#27
ptomecek merged 1 commit into
mainfrom
feat/in-memory-builder-cache

Conversation

@ptomecek

Copy link
Copy Markdown
Collaborator

Summary

Adds cache_memory, a third caching primitive alongside cache and cache_parquet, closing the gap described in #26: an in-memory, builder-driven cache that does not require the plan to be serializable.

input key model backend
cache LazyFrame (plan) serialize (content-addressed) in-memory
cache_parquet builder or plan (+ schema) cache_path (explicit) Parquet (disk)
cache_memory builder or plan (+ schema) none (instance / identity) in-memory

cache derives its key by serializing the plan, so it cannot key a LazyFrame backed by register_io_source whose generator closes over non-picklable state (a connection, a lock, an open iterator). cache_parquet accepts a builder + schema and needs no serialization, but only has a disk backend. cache_memory fills the remaining corner: builder-callable input, no plan serialization, in-memory backend.

Behaviour

It collects the builder at most once into a closure-held buffer and replays it on every reference or collect, so a source read from many branches of a multi-branch evaluation (including pl.collect_all) executes upstream exactly once instead of once per reference. There is no key — reuse happens through the returned frame's identity, and the buffer is reclaimed by garbage collection when that frame is dropped.

  • Construction and collect_schema() (with a callable schema) do no I/O and no collect; the builder runs only on first row demand.
  • The one-time build is thread-safe; concurrent first-collects fanned out across pl.collect_all share a single build.
  • The build outcome is terminal: any failure is recorded and re-raised on subsequent collects (as a neutral error carrying the original type and message), so a failing builder is not re-run per branch. Retry by constructing a fresh cache_memory.
  • The collected frame is reconciled against the declared schema once (missing column or dtype mismatch raises; extras are dropped).

API

  • Top-level: cache_memory(build_or_lf, *, schema)
  • Namespace: lf.piot.cache_memory(*, schema)

Test plan

  • New unit tests in tests/io_sources/test_lazy_cache_memory.py (23 cases): collect-once across references and pl.collect_all fan-out, thread-safe single build, buffer released on GC, non-serializable plugin source, callable-schema resolution without building, terminal/no-amplification failure semantics, no frame/traceback retention on failure, schema reconciliation, and empty-projection cardinality.
  • ruff check and ruff format --check clean.
  • Full io_sources suite passes.

Closes #26.

@ptomecek
ptomecek force-pushed the feat/in-memory-builder-cache branch 2 times, most recently from 605c65f to 26c88fc Compare August 28, 2026 17:09
Adds `cache_memory`, a third caching primitive alongside `cache` and
`cache_parquet`, closing the gap described in issue #26: an in-memory,
builder-driven cache that does not require the plan to be serializable.

`cache` derives its key by serializing the plan, so it cannot key a
LazyFrame backed by `register_io_source` whose generator closes over
non-picklable state (a connection, a lock, an open iterator).
`cache_parquet` accepts a builder + schema and needs no serialization,
but only has a disk backend. `cache_memory` fills the remaining corner:
builder-callable input, no plan serialization, in-memory backend.

It collects the builder at most once into a closure-held buffer and
replays it on every reference or collect, so a source read from many
branches of a multi-branch evaluation (including `pl.collect_all`)
executes upstream exactly once instead of once per reference. There is
no key: reuse happens through the returned frame's identity, and the
buffer is reclaimed by garbage collection when that frame is dropped.

Semantics:
- Construction and `collect_schema()` (with a callable schema) do no
  I/O and no collect; the builder runs only on first row demand.
- The one-time build is thread-safe; concurrent first-collects fanned
  out across `pl.collect_all` share a single build.
- The build outcome is terminal: any failure is recorded and re-raised
  on subsequent collects (as a neutral error carrying the original type
  and message), so a failing builder is not re-run per branch. Retry by
  constructing a fresh `cache_memory`.
- The collected frame is reconciled against the declared schema once
  (missing column or dtype mismatch raises; extras are dropped).

Exposed as a top-level function and as `lf.piot.cache_memory(schema=...)`.
Adds unit tests and documents the primitive in the API reference and
query-optimization guides.

Signed-off-by: Pascal Tomecek <40371786+ptomecek@users.noreply.github.com>
Co-authored-by: hintse <5867507+hintse@users.noreply.github.com>
@ptomecek
ptomecek force-pushed the feat/in-memory-builder-cache branch from 26c88fc to c4462cd Compare August 28, 2026 17:13
@github-actions

Copy link
Copy Markdown
Contributor

Test Results

1 415 tests  +23   1 367 ✅ +23   39s ⏱️ -1s
    2 suites ± 0      48 💤 ± 0 
    2 files   ± 0       0 ❌ ± 0 

Results for commit c4462cd. ± Comparison against base commit 85adfbb.

@ptomecek
ptomecek marked this pull request as ready for review August 31, 2026 14:41
@ptomecek
ptomecek merged commit 605f316 into main Aug 31, 2026
6 checks passed
@ptomecek
ptomecek deleted the feat/in-memory-builder-cache branch August 31, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: an in-memory, builder-driven cache for non-serializable LazyFrames (complement to cache and cache_parquet)

2 participants