feat: add .piot.probe() pass-through telemetry source - #34
Merged
Conversation
Add a result-preserving pass-through IO source, `.piot.probe(description=...)`, that can be inserted at any point in a lazy pipeline to emit one `io_source.execute[probe]` OpenTelemetry span for the data flowing through that point. `next_elapsed_total_ms` is the pull latency of the sub-plan below the probe and `total_rows` is the rows through it; both predicate and projection pushdown are forwarded to the input, so the probe is not an optimization barrier for them. It is modelled on `debug` but omits the per-execution logging and the unconditional `explain()` call, so it is cheap enough to leave in a pipeline. Like every source registered via `register_io_source_with_is_pure`, the span is a no-op unless an OpenTelemetry SDK is configured, and it can be disabled with `OTEL_PYTHON_INSTRUMENTATION_POLARS_IO_TOOLS_ENABLED=false`. Note that the probe inserts a `PythonScan` node that re-enters Polars via `collect_lf_in_io_source`, so it is a measurement point rather than a zero-cost tap. Signed-off-by: Pascal Tomecek <40371786+ptomecek@users.noreply.github.com>
ptomecek
force-pushed
the
feat/probe-source
branch
from
September 1, 2026 13:00
f21418f to
1a1486e
Compare
Contributor
A pass-through cannot know whether its input is pure. Registering it as `is_pure=True` (the default) let Polars deduplicate repeated uses of the same probe and collapse executions, changing results for an impure input. Register `is_pure=False` so every occurrence is measured and results are preserved. Also pass the schema as a callable so it is resolved lazily rather than forced at construction, and note the single-worker full-materialization caveat in the docstring. Adds an impurity regression test plus n_rows-pushdown and exact-order pass-through coverage. Signed-off-by: Pascal Tomecek <40371786+ptomecek@users.noreply.github.com>
The probe's purity is its input's purity, which it cannot infer. A hardcoded `is_pure=False` is safe but hides a pure input's dedup-ability: Polars treats the probe as impure and re-executes the (pure) sub-plan for every repeated use instead of collapsing it. Expose `is_pure` (keyword-only, default `False`) so a caller with a known-pure input can opt into dedup (one execution, one span). Adds a forwarding test. Signed-off-by: Pascal Tomecek <40371786+ptomecek@users.noreply.github.com>
`debug` is side-effecting -- it logs (or prints) on every execution -- so registering it as `is_pure=True` (the previous default) let Polars deduplicate repeated uses and drop log output the caller asked for. Add `is_pure` (keyword-only, default `False`), mirroring `probe`, so repeated uses are not collapsed by default; a caller with a known-pure input can opt into dedup. Adds a test that the log-side-effect runs per occurrence. Signed-off-by: Pascal Tomecek <40371786+ptomecek@users.noreply.github.com>
ptomecek
marked this pull request as ready for review
September 1, 2026 14:10
hintse
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A result-preserving pass-through IO source —
lf.piot.probe(description="...")— that you can drop at any point in a lazy pipeline to get oneio_source.execute[probe]OpenTelemetry span for the data flowing through that point:polars_io_tools.next_elapsed_total_ms— pull latency of the sub-plan below the probe;polars_io_tools.total_rows— rows through it;polars_io_tools.explain_detail— thedescription(e.g. the stage being measured).Both predicate and projection pushdown are forwarded to the input, so the probe is not an optimization barrier for them. It reuses the instrumentation added in #30, so the span is a no-op without an OpenTelemetry SDK and honours
OTEL_PYTHON_INSTRUMENTATION_POLARS_IO_TOOLS_ENABLED=false.Why not
debug?.piot.debug()is already an instrumented pass-through, but it (a) logs/prints on every execution and (b) computesself.explain()unconditionally — too heavy to leave in a pipeline.probeisdebugminus that baggage.Usage
Parquet profiling
Verified locally that pushdown reaches the parquet reader through the probe: a column projection prunes the columns actually read (the span's pull time drops accordingly), and a predicate is pushed into the scan (only matching rows flow through, reflected in
total_rows). So it is genuinely useful for comparative questions — is pushdown working? did column pruning help? is the predicate reaching the reader?Caveats.
next_elapsed_total_msis caller-observed pull latency, so it undercounts absolute read time when the reader decodes on background threads. And the probe inserts aPythonScanre-entry node, so it adds overhead — on the order of milliseconds, growing with the number of rows through it (a per-batch Rust↔Python round trip; the span emission itself is free). That is negligible relative to any non-trivial read but can dominate a small, fully-cached one. Prefer it for comparative profiling rather than absolute scan-time attribution.Tests
tests/io_sources/test_lazy_probe.py: result-preserving pass-through, exactly one span withexplain_name=probe+description, and predicate pushdown reaching the probe. Fullio_sourcessuite: 1319 passed.