Skip to content

Commit a6e1a5e

Browse files
rustyconoverclaude
andcommitted
filter_v2: evaluate on a single-threaded DuckDB
The embedded evaluator opens one DuckDB database per (server thread, evaluation context) and ran it at the default of one scheduler thread per core. An evaluation is one small query over one batch, which DuckDB cannot parallelize, so the pool bought nothing and cost two ways: every evaluation paid to dispatch onto it (~3.3 -> ~2.4 ms per evaluation on a 48-core host), and a waitress worker there carried ~290 threads. Also documents the two PushdownFilters helpers the HTTP replay uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 50953cf commit a6e1a5e

2 files changed

Lines changed: 18 additions & 2 deletions

File tree

‎vgi/filter_v2.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1165,7 +1165,12 @@ def _get_evaluation_connection(context: EvaluationContext) -> Any:
11651165
f"found {engine_name} {engine_version or '<unknown>'}"
11661166
)
11671167

1168-
connection = connect()
1168+
# One thread. Each evaluates one batch against one predicate, which
1169+
# DuckDB cannot parallelize, and there is one of these databases per
1170+
# (server thread, context): at the default of one scheduler thread per
1171+
# core, a waitress worker on a 48-core host carried ~290 threads, and
1172+
# every evaluation paid to dispatch onto that pool.
1173+
connection = connect(config={"threads": 1})
11691174
_apply_context(connection, context)
11701175
connections[key] = connection
11711176
return connection

‎vgi/table_filter_pushdown.py‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1132,7 +1132,12 @@ def apply_delta(self, batch: pa.RecordBatch) -> PushdownFilters:
11321132
return _pushdown_from_v2_state(state)
11331133

11341134
def _revision_map(self) -> dict[str, int]:
1135-
"""Map every predicate ID seen in this scan, tombstones included, to its last applied revision."""
1135+
"""Map every predicate ID seen in this scan, tombstones included, to its last applied revision.
1136+
1137+
Returns:
1138+
``{predicate_id: revision}``; empty for filters not decoded from v2.
1139+
1140+
"""
11361141
return dict(self._v2_state.revisions) if self._v2_state is not None else {}
11371142

11381143
def _with_predicate_order(self, order: list[str]) -> PushdownFilters:
@@ -1143,6 +1148,12 @@ def _with_predicate_order(self, order: list[str]) -> PushdownFilters:
11431148
compacted (an ID removed and later re-added moves to the end, which a
11441149
shorter history does not reproduce by itself).
11451150
1151+
Args:
1152+
order: The live predicate IDs in the order to restore.
1153+
1154+
Returns:
1155+
Filters with the same predicates, arranged in ``order``.
1156+
11461157
Raises:
11471158
FilterDeserializationError: If ``order`` does not name exactly the
11481159
live predicates.

0 commit comments

Comments
 (0)