Skip to content

Commit eb99608

Browse files
committed
fix(store): cut WAL write amplification on 64 KiB-page indexes (DeusData#1419)
Every index is written with 64 KiB pages (sqlite_writer.c), but read-write connections kept SQLite's default 2000 KiB page cache, which holds about 31 of those pages. A closure-repair delta purges and patches rows across the node and edge tables and all of their indexes, so the transaction kept spilling dirty pages into the WAL and rewriting them in place. A one-line body change on a 294 MB django index wrote 5.2 GiB to the WAL (about 75k in-place page rewrites) and 5.7 GiB of logical writes in total. Read-write on-disk connections now use a 64 MiB cache (1024 pages, the budget bulk mode already used), and end_bulk restores that instead of the 2000 KiB default. Query connections keep the default. cbm_pipeline_publish_staged also forced synchronous=FULL on the private staging database, so every metadata and FTS commit fsynced the WAL. A crash before the atomic rename discards the stage, and cbm_store_seal_for_atomic_publish() still raises the connection to FULL for the TRUNCATE checkpoint and DELETE-mode switch that make the published file durable, so those per-commit fsyncs protected nothing. The commits now run at the store's NORMAL level, which SQLite documents as corruption-safe under WAL. No corruption or atomicity guarantee changes: the live database is still only replaced by a sealed, integrity-checked file via rename. Measured on macOS, main vs this change, same machine and corpora: - django one-line edit (closure repair): WAL writes 5.2 GiB -> 0.52 GiB, logical writes 5.76 GiB -> 1.08 GiB, WAL fsyncs 13 -> 8 - django full index: WAL writes 58 MiB -> 47.5 MiB, WAL fsyncs 9 -> 2 - php (65 MB index) one-line edit: WAL writes 82.5 MiB -> 32.3 MiB, spill rewrites 468 -> 0 - kubernetes (1.7 GB index) full index: WAL writes 669 MiB -> 390 MiB, WAL fsyncs 13 -> 6 The comments that put a healthy WAL at ~4 MiB assumed 4 KiB pages; at 64 KiB the 1000-page autocheckpoint is ~64 MiB, still well under the 256 MiB journal_size_limit. Tests: store_pragmas/store_role_pragmas_issue1419 pins the effective pragmas of each store role (read-write, bulk, seal, query), and pipeline_semantic_manifest_repro/ pipeline_staged_publish_commits_below_full_sync_issue1419 reads the effective synchronous level from a WAL hook on every connection that a full index plus a closure-repair reindex opens, and requires that no commit runs at FULL. Refs DeusData#1419 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 0f52d30 commit eb99608

5 files changed

Lines changed: 229 additions & 13 deletions

File tree

‎src/pipeline/pipeline.c‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2267,10 +2267,15 @@ int cbm_pipeline_publish_staged(char *stage_path, const cbm_pipeline_generation_
22672267
free(stage_path);
22682268
return CBM_PIPELINE_PERSIST_FAILED;
22692269
}
2270-
bool ok = cbm_store_exec(store, "PRAGMA synchronous=FULL;") == CBM_STORE_OK;
2271-
ok = ok && cbm_store_delete_file_hashes(store, generation->project) == CBM_STORE_OK &&
2272-
cbm_store_upsert_file_hash_batch(store, generation->manifest,
2273-
generation->manifest_count) == CBM_STORE_OK;
2270+
/* No synchronous=FULL for these writes (#1419): the stage is private until
2271+
* the atomic rename and a crash discards it, so an fsync per WAL commit
2272+
* protects nothing here. The store's NORMAL level is SQLite's
2273+
* corruption-safe setting under WAL, and cbm_store_seal_for_atomic_publish()
2274+
* raises this connection to FULL for the checkpoint that makes the
2275+
* published file durable. */
2276+
bool ok = cbm_store_delete_file_hashes(store, generation->project) == CBM_STORE_OK &&
2277+
cbm_store_upsert_file_hash_batch(store, generation->manifest,
2278+
generation->manifest_count) == CBM_STORE_OK;
22742279
/* LSP surfaces belong to the generation: written inside the same staging
22752280
* store, before the atomic rename, so graph and surface data can never
22762281
* publish separately. The delete guards the incremental path, whose

‎src/store/store.c‎

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -539,6 +539,16 @@ int64_t cbm_store_resolve_mmap_size(void) {
539539
return (int64_t)parsed;
540540
}
541541

542+
/* #1419: page cache for read-write on-disk connections, in SQLite's negative
543+
* KiB form. sqlite_writer.c gives every index 64 KiB pages, so SQLite's
544+
* default 2000 KiB holds only ~31 of them, while a delta's purge and patch
545+
* dirty pages across the node and edge tables and all of their indexes. The
546+
* transaction then spills dirty pages into the WAL and rewrites them over and
547+
* over (5.2 GiB of WAL writes for a one-line change on a 294 MB django index).
548+
* 64 MiB = 1024 pages, the budget bulk mode already used; the cache only grows
549+
* as pages are touched. Query connections keep the default. */
550+
static const char ST_WRITE_CACHE_SQL[] = "PRAGMA cache_size = -65536;";
551+
542552
/* Configure connection pragmas.
543553
* in_memory — :memory: DB (synchronous OFF, no journal file).
544554
* read_only — query-only connection opened SQLITE_OPEN_READONLY. Runs
@@ -594,8 +604,9 @@ static int configure_pragmas(cbm_store_t *s, bool in_memory, bool read_only) {
594604
* cbm_store_checkpoint's SIGBUS note), so without a size limit the -wal
595605
* file only ever grows; journal_size_limit truncates it back to N bytes
596606
* on the next successful
597-
* reset. N is far above the healthy WAL (~4 MiB under the default
598-
* 1000-page autocheckpoint), so normal indexing never triggers
607+
* reset. N is far above the healthy WAL (~64 MiB: the default
608+
* 1000-page autocheckpoint at the 64 KiB page size sqlite_writer.c
609+
* gives every index), so normal indexing never triggers
599610
* truncate/regrow churn — it only fires after abnormal growth.
600611
* Shared/live paths do NOT use a TRUNCATE checkpoint: truncating the WAL
601612
* to zero can raise SIGBUS in a sibling process that has the DB mmap'd
@@ -604,6 +615,10 @@ static int configure_pragmas(cbm_store_t *s, bool in_memory, bool read_only) {
604615
if (rc != CBM_STORE_OK) {
605616
return rc;
606617
}
618+
rc = exec_sql(s, ST_WRITE_CACHE_SQL);
619+
if (rc != CBM_STORE_OK) {
620+
return rc;
621+
}
607622
char mmap_sql[ST_BUF_64];
608623
snprintf(mmap_sql, sizeof(mmap_sql), "PRAGMA mmap_size = %lld;",
609624
(long long)cbm_store_resolve_mmap_size());
@@ -1811,20 +1826,21 @@ int cbm_store_begin_bulk(cbm_store_t *s) {
18111826
* because the in-memory rollback journal is lost on crash.
18121827
* WAL mode is crash-safe: uncommitted WAL entries are simply discarded
18131828
* on the next open. Performance is preserved via synchronous=OFF and a
1814-
* larger cache, which are safe with WAL. */
1829+
* 64 MiB cache, which are safe with WAL. */
18151830
int rc = exec_sql(s, "PRAGMA synchronous = OFF;");
18161831
if (rc != CBM_STORE_OK) {
18171832
return rc;
18181833
}
1819-
return exec_sql(s, "PRAGMA cache_size = -65536;"); /* CBM_SZ_64 MB */
1834+
return exec_sql(s, ST_WRITE_CACHE_SQL);
18201835
}
18211836

18221837
int cbm_store_end_bulk(cbm_store_t *s) {
18231838
int rc = exec_sql(s, "PRAGMA synchronous = NORMAL;");
18241839
if (rc != CBM_STORE_OK) {
18251840
return rc;
18261841
}
1827-
return exec_sql(s, "PRAGMA cache_size = -2000;"); /* default ~2 MB */
1842+
/* Back to the read-write cache, not SQLite's 2000 KiB default (#1419). */
1843+
return exec_sql(s, ST_WRITE_CACHE_SQL);
18281844
}
18291845

18301846
int cbm_store_drop_indexes(cbm_store_t *s) {

‎src/store/store.h‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -427,11 +427,12 @@ int cbm_store_rollback(cbm_store_t *s);
427427

428428
/* ── Bulk write optimization ────────────────────────────────────── */
429429

430-
/* Tune pragmas for bulk write throughput (synchronous=OFF, large cache).
430+
/* Tune pragmas for bulk write throughput (synchronous=OFF, 64 MiB cache).
431431
* WAL journal mode is preserved throughout for crash safety. */
432432
int cbm_store_begin_bulk(cbm_store_t *s);
433433

434-
/* Restore normal pragmas (synchronous=NORMAL, default cache) after bulk writes. */
434+
/* Restore normal pragmas (synchronous=NORMAL, the 64 MiB read-write cache)
435+
* after bulk writes. */
435436
int cbm_store_end_bulk(cbm_store_t *s);
436437

437438
/* Drop user indexes for faster bulk inserts. */

‎tests/test_pipeline.c‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2916,6 +2916,115 @@ TEST(pipeline_incremental_repoints_call_reference_without_stale_edge) {
29162916
PASS();
29172917
}
29182918

2919+
/* #1419: every index publishes through a private staging DB in WAL mode that
2920+
* cbm_store_seal_for_atomic_publish() checkpoints at synchronous=FULL and
2921+
* switches to DELETE mode before the atomic rename. A crash before that rename
2922+
* discards the stage, so a WAL commit at FULL only adds an fsync per
2923+
* transaction without protecting anything; SQLite documents NORMAL as
2924+
* corruption-safe under WAL. The probe reads the EFFECTIVE synchronous level
2925+
* of the committing handle from a WAL hook on every connection the run opens.
2926+
*
2927+
* SQLite installs its default autocheckpoint WAL hook after auto-extensions
2928+
* run, so the auto-extension installs a statement trace and the trace installs
2929+
* the probe once the connection is fully open. The probe keeps SQLite's
2930+
* default 1000-frame autocheckpoint, so it does not change what the run
2931+
* writes. */
2932+
enum { I1419_SYNC_FULL = 2, I1419_DEFAULT_AUTOCHECKPOINT = 1000 };
2933+
static atomic_int g_i1419_wal_commits;
2934+
static atomic_int g_i1419_full_sync_commits;
2935+
2936+
static int i1419_wal_probe(void *arg, sqlite3 *db, const char *db_name, int wal_frames) {
2937+
(void)arg;
2938+
atomic_fetch_add(&g_i1419_wal_commits, 1);
2939+
sqlite3_stmt *stmt = NULL;
2940+
if (sqlite3_prepare_v2(db, "PRAGMA synchronous;", -1, &stmt, NULL) == SQLITE_OK &&
2941+
sqlite3_step(stmt) == SQLITE_ROW && sqlite3_column_int(stmt, 0) >= I1419_SYNC_FULL) {
2942+
atomic_fetch_add(&g_i1419_full_sync_commits, 1);
2943+
}
2944+
sqlite3_finalize(stmt);
2945+
if (wal_frames >= I1419_DEFAULT_AUTOCHECKPOINT) {
2946+
(void)sqlite3_wal_checkpoint(db, db_name);
2947+
}
2948+
return SQLITE_OK;
2949+
}
2950+
2951+
static int i1419_trace(unsigned type, void *ctx, void *stmt, void *sql) {
2952+
(void)type;
2953+
(void)ctx;
2954+
(void)sql;
2955+
(void)sqlite3_wal_hook(sqlite3_db_handle((sqlite3_stmt *)stmt), i1419_wal_probe, NULL);
2956+
return 0;
2957+
}
2958+
2959+
static int i1419_install_probe(sqlite3 *db, char **errmsg, const sqlite3_api_routines *api) {
2960+
(void)errmsg;
2961+
(void)api;
2962+
return sqlite3_trace_v2(db, SQLITE_TRACE_STMT, i1419_trace, NULL);
2963+
}
2964+
2965+
static int i1419_index(const char *repo, const char *db_path) {
2966+
cbm_pipeline_t *p = cbm_pipeline_new(repo, db_path, CBM_MODE_FULL);
2967+
if (!p) {
2968+
return -1;
2969+
}
2970+
int rc = cbm_pipeline_run(p);
2971+
cbm_pipeline_free(p);
2972+
return rc;
2973+
}
2974+
2975+
TEST(pipeline_staged_publish_commits_below_full_sync_issue1419) {
2976+
char tmp[256];
2977+
snprintf(tmp, sizeof(tmp), "/tmp/cbm_i1419_XXXXXX");
2978+
if (!cbm_mkdtemp(tmp)) {
2979+
FAIL("tmpdir");
2980+
}
2981+
/* Equal lengths + equal mtimes: the second run must route on content
2982+
* bytes to the closure-repair (delta) publish, not a full rebuild. */
2983+
static const char initial_source[] =
2984+
"package syncprobe\n"
2985+
"func alphaSyncTarget() {}\n"
2986+
"func bravoSyncTarget() {}\n"
2987+
"func syncProbeAccept(callback func()) {}\n"
2988+
"func syncProbeSite() { syncProbeAccept(alphaSyncTarget) }\n";
2989+
static const char replacement_source[] =
2990+
"package syncprobe\n"
2991+
"func alphaSyncTarget() {}\n"
2992+
"func bravoSyncTarget() {}\n"
2993+
"func syncProbeAccept(callback func()) {}\n"
2994+
"func syncProbeSite() { syncProbeAccept(bravoSyncTarget) }\n";
2995+
ASSERT_EQ(sizeof(initial_source), sizeof(replacement_source));
2996+
write_temp_file(tmp, "probe.go", initial_source);
2997+
char source_path[512];
2998+
snprintf(source_path, sizeof(source_path), "%s/probe.go", tmp);
2999+
const time_t fixed_mtime = 1700000000;
3000+
ASSERT_EQ(pipeline_test_set_mtime(source_path, fixed_mtime, 123456789L), 0);
3001+
char db_path[512];
3002+
snprintf(db_path, sizeof(db_path), "%s/i1419.db", tmp);
3003+
3004+
atomic_store(&g_i1419_wal_commits, 0);
3005+
atomic_store(&g_i1419_full_sync_commits, 0);
3006+
ASSERT_EQ(sqlite3_auto_extension((void (*)(void))i1419_install_probe), SQLITE_OK);
3007+
/* Both publish routes run before any assertion, so the probe is always
3008+
* unregistered again even when a run fails. */
3009+
int full_rc = i1419_index(tmp, db_path);
3010+
write_temp_file(tmp, "probe.go", replacement_source);
3011+
int mtime_rc = pipeline_test_set_mtime(source_path, fixed_mtime, 123456789L);
3012+
cbm_pipeline_incremental_test_reset_faults();
3013+
int delta_rc = i1419_index(tmp, db_path);
3014+
cbm_incremental_route_t delta_route = cbm_pipeline_incremental_test_last_route();
3015+
(void)sqlite3_cancel_auto_extension((void (*)(void))i1419_install_probe);
3016+
th_rmtree(tmp);
3017+
3018+
ASSERT_EQ(full_rc, 0);
3019+
ASSERT_EQ(mtime_rc, 0);
3020+
ASSERT_EQ(delta_rc, 0);
3021+
ASSERT_EQ(delta_route, CBM_INCREMENTAL_ROUTE_CLOSURE_REPAIR);
3022+
/* Non-vacuous: both publishes commit through the WAL. */
3023+
ASSERT_GT(atomic_load(&g_i1419_wal_commits), 0);
3024+
ASSERT_EQ(atomic_load(&g_i1419_full_sync_commits), 0);
3025+
PASS();
3026+
}
3027+
29193028
/* SQL DDL becomes first-class Table/View nodes wired into FROM/JOIN lineage,
29203029
* while the shared name registry must NOT leak those relations into other
29213030
* languages' textual resolution: a Python call or identifier sharing the
@@ -16556,6 +16665,7 @@ SUITE(pipeline) {
1655616665
SUITE(pipeline_semantic_manifest_repro) {
1655716666
RUN_TEST(incremental_downgrade_preserves_scope_and_artifact_across_change_noop_delete);
1655816667
RUN_TEST(pipeline_incremental_repoints_call_reference_without_stale_edge);
16668+
RUN_TEST(pipeline_staged_publish_commits_below_full_sync_issue1419);
1655916669
RUN_TEST(pipeline_sql_lineage_and_relation_isolation);
1656016670
RUN_TEST(pipeline_incremental_sql_table_rename_drops_stale_lineage);
1656116671
RUN_TEST(pipeline_dbt_jinja_lineage);

‎tests/test_store_pragmas.c‎

Lines changed: 86 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@
1111
#include "test_framework.h"
1212
#include "test_helpers.h"
1313
#include <store/store.h>
14+
#include "sqlite3.h" /* vendored/sqlite3 — read pragmas back on the store's own handle */
15+
#include <stdbool.h>
1416
#include <stdio.h>
1517
#include <stdlib.h>
1618
#include <string.h>
@@ -98,8 +100,9 @@ TEST(journal_size_limit_bounds_wal_issue1083) {
98100

99101
cbm_store_t *s = cbm_store_open_path(tmp_path);
100102
ASSERT(s != NULL);
101-
/* 256 MiB — far above the healthy WAL (~4 MiB), so no truncate/regrow churn
102-
* in normal operation; it only fires after abnormal (starved) growth. */
103+
/* 256 MiB — far above the healthy WAL (~64 MiB: 1000 autocheckpoint pages
104+
* of 64 KiB), so no truncate/regrow churn in normal operation; it only
105+
* fires after abnormal (starved) growth. */
103106
ASSERT(cbm_store_journal_size_limit(s) == (int64_t)268435456);
104107
cbm_store_close(s);
105108

@@ -113,6 +116,86 @@ TEST(journal_size_limit_bounds_wal_issue1083) {
113116
PASS();
114117
}
115118

119+
/* #1419: effective pragmas per store role, read on the handle itself —
120+
* synchronous is per-connection and never persisted, so an external sqlite3
121+
* shell reports its own default (FULL), not what the indexer runs with. */
122+
static int role_pragma_int(cbm_store_t *s, const char *sql) {
123+
sqlite3_stmt *stmt = NULL;
124+
int value = -1;
125+
if (sqlite3_prepare_v2(cbm_store_get_db(s), sql, -1, &stmt, NULL) == SQLITE_OK &&
126+
sqlite3_step(stmt) == SQLITE_ROW) {
127+
value = sqlite3_column_int(stmt, 0);
128+
}
129+
sqlite3_finalize(stmt);
130+
return value;
131+
}
132+
133+
static bool role_journal_mode_is(cbm_store_t *s, const char *want) {
134+
sqlite3_stmt *stmt = NULL;
135+
bool match = false;
136+
if (sqlite3_prepare_v2(cbm_store_get_db(s), "PRAGMA journal_mode;", -1, &stmt, NULL) ==
137+
SQLITE_OK &&
138+
sqlite3_step(stmt) == SQLITE_ROW) {
139+
const char *mode = (const char *)sqlite3_column_text(stmt, 0);
140+
match = mode && strcmp(mode, want) == 0;
141+
}
142+
sqlite3_finalize(stmt);
143+
return match;
144+
}
145+
146+
TEST(store_role_pragmas_issue1419) {
147+
enum {
148+
SYNC_OFF = 0,
149+
SYNC_NORMAL = 1,
150+
SYNC_FULL = 2,
151+
/* 64 MiB = 1024 of the 64 KiB pages every index is written with. */
152+
WRITE_CACHE_KIB = -65536,
153+
SQLITE_DEFAULT_CACHE_KIB = -2000,
154+
};
155+
char tmp_path[256];
156+
snprintf(tmp_path, sizeof(tmp_path), "%s/cbm_test_roles_%d.db", cbm_tmpdir(), (int)getpid());
157+
unlink(tmp_path);
158+
159+
/* Read-write role (live ADR writes, staging generations): WAL at NORMAL,
160+
* with a page cache that holds more than a few 64 KiB pages. */
161+
cbm_store_t *s = cbm_store_open_path(tmp_path);
162+
ASSERT(s != NULL);
163+
ASSERT_TRUE(role_journal_mode_is(s, "wal"));
164+
ASSERT_EQ(role_pragma_int(s, "PRAGMA synchronous;"), SYNC_NORMAL);
165+
ASSERT_EQ(role_pragma_int(s, "PRAGMA cache_size;"), WRITE_CACHE_KIB);
166+
/* Bulk role: sync off for the write burst, then back to the read-write
167+
* settings rather than SQLite's defaults. */
168+
ASSERT_EQ(cbm_store_begin_bulk(s), CBM_STORE_OK);
169+
ASSERT_EQ(role_pragma_int(s, "PRAGMA synchronous;"), SYNC_OFF);
170+
ASSERT_EQ(cbm_store_end_bulk(s), CBM_STORE_OK);
171+
ASSERT_EQ(role_pragma_int(s, "PRAGMA synchronous;"), SYNC_NORMAL);
172+
ASSERT_EQ(role_pragma_int(s, "PRAGMA cache_size;"), WRITE_CACHE_KIB);
173+
/* Seal role: the durable checkpoint before the atomic rename runs at FULL
174+
* and leaves a self-contained DELETE-mode file. */
175+
ASSERT_EQ(cbm_store_seal_for_atomic_publish(s), CBM_STORE_OK);
176+
ASSERT_EQ(role_pragma_int(s, "PRAGMA synchronous;"), SYNC_FULL);
177+
ASSERT_TRUE(role_journal_mode_is(s, "delete"));
178+
cbm_store_close(s);
179+
180+
/* Query role: read-only, never switches a sealed file back to WAL, and
181+
* keeps SQLite's small default cache (one per request, so it stays cheap). */
182+
cbm_store_t *q = cbm_store_open_path_query(tmp_path);
183+
ASSERT(q != NULL);
184+
ASSERT_EQ(sqlite3_db_readonly(cbm_store_get_db(q), "main"), 1);
185+
ASSERT_TRUE(role_journal_mode_is(q, "delete"));
186+
ASSERT_EQ(role_pragma_int(q, "PRAGMA cache_size;"), SQLITE_DEFAULT_CACHE_KIB);
187+
cbm_store_close(q);
188+
189+
unlink(tmp_path);
190+
char tmp_wal[300];
191+
char tmp_shm[300];
192+
snprintf(tmp_wal, sizeof(tmp_wal), "%s-wal", tmp_path);
193+
snprintf(tmp_shm, sizeof(tmp_shm), "%s-shm", tmp_path);
194+
unlink(tmp_wal);
195+
unlink(tmp_shm);
196+
PASS();
197+
}
198+
116199
/* Pagination-cursor generation: minted per DB file, bumped per index run.
117200
* Same store + reads only -> stable; upsert_project (every index run's choke
118201
* point) -> changes; two distinct DB files can never share a generation even
@@ -264,6 +347,7 @@ TEST(corrupt_page_scan_returns_error_not_truncation) {
264347

265348
SUITE(store_pragmas) {
266349
RUN_TEST(journal_size_limit_bounds_wal_issue1083);
350+
RUN_TEST(store_role_pragmas_issue1419);
267351
RUN_TEST(store_generation_tracks_mutations);
268352
RUN_TEST(store_generation_rejects_malformed_metadata_atomically);
269353
RUN_TEST(corrupt_page_scan_returns_error_not_truncation);

0 commit comments

Comments
 (0)