Skip to content

Commit 1818373

Browse files
committed
perf(cypher): stream query execution chunk by chunk from a paged node scan
On the kernel index (8.5 M nodes, 1.9 M CALLS edges) `MATCH (a)-[:CALLS]->(b) RETURN count(*)` peaked at 18-25 GB RSS (killed in a 12 GB container): the store materialised every start node with all its strings (4.9 GB), every one was deep-copied into a 1.6 KB binding, and every expanded row was materialised again before projection. cypher: execute_single now runs the start nodes through the pipeline in chunks of 1024 (pattern-0 hops, late WHERE, WITH, RETURN). Each hop emits a binding's expansions in input order and every consumer (RETURN rows / star / default projection / aggregation, WITH aggregation / projection) keeps its state between chunks, so the concatenated output is the single-pass output: same rows, row order, first-seen group order, ORDER BY ties, DISTINCT, LIMIT and max_rows truncation, OPTIONAL fallback rows, warnings. Queries with additional MATCH patterns keep one pass (their join strategy depends on the first binding of the whole set). Start nodes and fetched neighbours move their strings into the binding instead of being deep-copied and freed. store: cbm_store_node_scan_open/_next/_close hand out the rows of cbm_store_find_nodes / _by_label a page at a time. Pages are keyset queries along the index the eager query walks (idx_nodes_file: NULL file_path rows by id, then the rest of the last file by id, then the files after it; idx_nodes_label: by id), each reset before it returns, so no read lock is held between pages. open verifies the eager plan with EXPLAIN QUERY PLAN and falls back to the eager rows for any other plan. Kernel, same index, medians of 3 (macOS arm64): count(*) peak RSS 24.9 GB -> 256 MB, 23.0 s -> 13.0 s top-callee peak RSS 12.4 GB -> 1.26 GB, 11.7 s -> 10.9 s Responses byte-identical to the baseline on django, redis and the kernel (19 Cypher + 24 tool cases each); the streamed scan equals the eager scan field by field, in order, for every project and label of all three indexes. A scan that fails part-way fails the whole query ("query aborted: reading the graph failed") instead of answering from the pages read so far: The streamed start-node scan answered from the rows of the pages it had read when a later page failed (a SQLite step error or a page re-prepare failure): a partial result that looked complete. A failed page, a scan that cannot open, or a failed eager fallback now marks the start source failed; execute_chunk returns an error and cbm_cypher_execute reports "query aborted: reading the graph failed (<store error>)" with no rows. The store error is logged as cypher.scan_failed. The test seam cbm_store_node_scan_test_fail_page_after (CBM_ENABLE_TEST_SEAMS) fails one page after a given number of successful pages; the new test fails the second page of a label scan (rows and count(*)) and a later page of the all-nodes scan, and asserts an error result with no rows. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent a372a1f commit 1818373

7 files changed

Lines changed: 1310 additions & 246 deletions

File tree

‎scripts/memory-core-baseline.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ src/cli/config_text_edit.c 72
2424
src/cli/config_toml_edit.c 73
2525
src/cli/config_yaml_edit.c 117
2626
src/cli/hook_augment.c 31
27-
src/cypher/cypher.c 128
27+
src/cypher/cypher.c 125
2828
src/daemon/application.c 107
2929
src/daemon/bootstrap.c 13
3030
src/daemon/daemon.c 31

0 commit comments

Comments
 (0)