Skip to content

Commit 2f63d5f

Browse files
committed
feat: add is_pure flag to debug (default False)
`debug` is side-effecting -- it logs (or prints) on every execution -- so registering it as `is_pure=True` (the previous default) let Polars deduplicate repeated uses and drop log output the caller asked for. Add `is_pure` (keyword-only, default `False`), mirroring `probe`, so repeated uses are not collapsed by default; a caller with a known-pure input can opt into dedup. Adds a test that the log-side-effect runs per occurrence. Signed-off-by: Pascal Tomecek <40371786+ptomecek@users.noreply.github.com>
1 parent fab6101 commit 2f63d5f

2 files changed

Lines changed: 13 additions & 1 deletion

File tree

polars_io_tools/io_sources/lazy_debug.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ def debug(
1515
self: pl.LazyFrame,
1616
log_level: int | None = None,
1717
description: str | None = None,
18+
*,
19+
is_pure: bool = False,
1820
) -> pl.LazyFrame:
1921
"""
2022
A very simple pass-through lazy frame source to help with debugging experimentation of polars io sources and lazy frame behavior.
@@ -23,6 +25,7 @@ def debug(
2325
self: The input data frame to cache columns of.
2426
log_level: If provided, will log at the given level. If None, will print. Defaults to None.
2527
description: Optional free-form description of this source instance, attached to its OpenTelemetry span (``explain_detail``).
28+
is_pure: Whether the wrapped input is pure (deterministic and side-effect-free). Defaults to ``False`` so repeated uses are not deduplicated -- ``debug`` logs on every execution, so collapsing executions would drop output you asked for. Set ``True`` only for a known-pure input where deduplication is wanted.
2629
"""
2730
schema = self.collect_schema()
2831

@@ -55,4 +58,4 @@ def source_generator(
5558
raise RuntimeError(err_msg) from e
5659

5760
# TODO: Turn on validate_schema when this is solved: https://github.com/pola-rs/polars/issues/22110
58-
return register_io_source_with_is_pure(source_generator, schema=schema, validate_schema=False, explain_detail=description)
61+
return register_io_source_with_is_pure(source_generator, schema=schema, is_pure=is_pure, validate_schema=False, explain_detail=description)

polars_io_tools/tests/io_sources/test_lazy_debug.py

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,3 +43,12 @@ def test_debug_filter(caplog):
4343
assert "debug called with" in caplog.text
4444
assert "`with_columns=None`" in caplog.text
4545
assert '`predicate=[(col("a")) > (1)]`' in caplog.text
46+
47+
48+
def test_debug_logs_per_use_by_default(caplog):
49+
# is_pure defaults to False, so reusing the same debug is not deduplicated: its log-side-effect runs per occurrence.
50+
df = pl.DataFrame({"a": [1]}).lazy()
51+
debug = df.piot.debug(log_level=logging.INFO)
52+
caplog.set_level(logging.INFO)
53+
pl.concat([debug, debug]).collect(engine="streaming")
54+
assert caplog.text.count("debug called with") == 2

0 commit comments

Comments
 (0)