Skip to content

Commit e2c55ed

Browse files
huangyiireneclaude
andauthored
fix(driver-turso): the remote canonical backfill never writes a julian misreading over a stored value (#19719)
Fixes #6009 **Clause-②: no** The sequel to PR #19688, which fixed the `driver-sql` (Knex) half of this card at squash `40626bdca6`. The REMOTE path is a separate module that builds its own statements and did not inherit the guard. ## The defect `RemoteTransport.mapFieldTypeToSQL` declares every temporal column `TEXT`, so a `Field.datetime` / `Field.time` column can hold a digits-only string such as `'2026'`, `'86400'`, `'12'` — or the keyword `'now'`. SQLite's time-value grammar accepts a bare number as a JULIAN DAY and `now` as the wall clock, so for those two shapes `strftime` answers confidently rather than returning NULL, and the `coalesce(strftime(...), col)` that is supposed to preserve unparseable values never fires. The convergence `UPDATE` in `backfillRemoteCanonicalColumn` wrote that answer **over the stored bytes**, and no later run can get them back. Measured here on better-sqlite3 13.0.3 / SQLite 3.53.4, column declared `TEXT`: ```text '2026' -> -4707-06-11T12:00:00.000Z 'now' -> the wall clock, now '86400' -> -4476-06-15T12:00:00.000Z '12' -> -4713-12-06T12:00:00.000Z ``` In a `Field.time` column the same `'2026'` becomes `'12:00:00'` — a wall clock nobody wrote. Both are pinned. ## What changed All of it inside `packages/drivers/driver-turso/src/`. **No `packages/drivers/driver-sql/` file is touched**, and neither `sqliteCanonicalDatetimeSql` nor `sqliteCanonicalTimeSql` moves — the 2026-08-03 cloud#1005 ruling that refused a heuristic in the shared read expression is untouched, and #19688's pins that those expressions still misread a bare number stay green. - **`TursoDriver` hands its own `sqliteNonTemporalTextSql` across the module boundary**, beside the canonical expression it already hands over, as `{ canonical, nonTemporalText }`. The predicate is `protected` on `SqlDriver` and stays that way; nothing is copied, so remote and local cannot drift about which bytes are safe to overwrite. - **The convergence `UPDATE` gains `and not (guard)`** — the local twin's exclusion, same construction. - **Withheld rows block the canonical mark.** This is the half that keeps every query answer identical, and here it costs no second gate: a withheld row is by construction `col IS NOT canonical`, so it is already inside the probe's `residual`, and the mark already requires `residual === 0`. Withholding alone would not do it — a marked column drops the read-side repair, and the raw `'2026'` would then compare as TEXT instead of as the instant the repair reads it as. - **The probe promises exactly what an apply rewrites.** `buildProbeSql` gains a fourth count, `non_temporal_text` — the migration's own `WHERE` with the guard ANDed on instead of negated — and `residual - nonTemporalText` is what decides whether the convergence phase runs. The remote side has no other preview surface: `SqlDriver.previewDeferredSchemaWork` reads `deferredSchemaObjects`, which only the Knex sync path fills, so in remote mode it answers `[]` and this probe is the whole plan. `residual` itself is deliberately **not** netted down, because that is what holds the mark. - **A new per-column report field, `nonTemporalTextRowsWithheld`**, plus an operator warning naming the consequence and the remedy. It is `null` — NOT MEASURED — when no guard was supplied, never `0`. - **No new gate**, no new `check:*` script, and no recommendation for one. ### The corrected sentence The module's contract said rows outside the epoch-recovery band "do not block the canonical mark, because they are fixpoints of the shared repair". That is true only **above** SQLite's julian-day ceiling, where `strftime` returns NULL and `coalesce` hands the value back. **Below it** — `'12'`, `'2026'`, `'86400'` — the repair answers, the row is not a fixpoint, and dropping the repair changes what it matches. Both halves are now written with the measurement behind them, and the field's own contract says which of the two a given row is in. The count was **not** filed under `unresolvedEpochTextRows`, whose documented meaning is *recorded and harmless to the mark*; these rows are the opposite, so they got their own name. ### Why the entry points are not breaking The third position of `probeRemoteCanonicalColumns`, `backfillRemoteCanonicalColumn` and `backfillRemoteCanonicalColumns` now accepts `RemoteBackfillSqlRules` **or** the pre-existing bare `CanonicalSqlFor`. The bare arm is **fail-closed, not a fallback**: with no guard to withhold by, the convergence phase is refused, the column reports `error` and stays unmarked, and reads stay correct on the repair. So an external caller compiled against 17.4.0 keeps compiling and still cannot write a misreading. A required parameter was considered and rejected: this repo forbids a `major` changeset during the launch window, and a `minor` carrying a BREAKING banner would owe an ADR-0087 disposition for which no honest category exists here (`runtime-interface-only` needs zero in-code references, `type-surface-only` needs an erased base-side type, `no-migration-prescription` is closed by the FROM/TO the changeset owes). The union keeps the fix out of that dead end. ## The three questions **1. Does the julian misread reach the epoch-text `UPDATE`?** No, and for two independent reasons, either sufficient. *Structural:* its SET expression is `canonicalSqlFor(kind, 'cast(col as real)')`. `typeof(cast(X as real))` is always `'real'`, so the canonical expression takes its integer/real limb — `strftime('%Y-%m-%dT%H:%M:%fZ', v/1000.0, 'unixepoch')`. The julian limb lives only in the `else` arm's `strftime(..., col)`, which that statement can never reach, whatever the row holds. *Set-theoretic:* its `WHERE` selects digits-only TEXT in `1e12 <= v < 4.1024448e12`, and a bare number is read as a julian day only below `5373484.5`. Disjoint (see Q3). ANDing the guard on there would withhold nothing and would falsely imply the two sets meet, so the statement is left alone and the reading is recorded beside it, with a pin. **2. After this change, can a column still be marked canonical while holding a row the shared repair reads as a BC date?** No — and the path is named rather than assumed absent. `canonical` is `after.residual === 0 && after.epochInBand === 0`. A julian-readable digits-only cell is not equal to what the repair yields for it (`'2026'` vs `'-4707-06-11T12:00:00.000Z'`), so `col IS NOT canonical` holds and the row is counted in `residual`; it is withheld from the `UPDATE`, so it stays counted; therefore `residual >= 1` and the mark cannot be granted. The early fast-path return is gated on the same `probe.residual === 0`, so it is not a second door. *Control:* the same instrument reports `residual = 0` for a digits-only value **above** the julian ceiling (`'1753660800'`, `'999999999999'`, `REMOTE_BACKFILL_EPOCH_MS_MAX + 1`) — so the non-zero residual is about the value, not a stuck counter. The `'huge'` test asserts that column is still marked, and asserts `julianday()` is NULL for it, taken from SQLite rather than claimed. **3. Where does the epoch band sit relative to the julian range?** There is a **gap of nearly six orders of magnitude**; the bands neither overlap nor abut. | | lower | upper | |---|---|---| | julian-day reading of a bare number | `0` | `5373484.5` (exclusive) | | `REMOTE_BACKFILL_EPOCH_MS_MIN` .. `_MAX` | `1000000000000` | `4102444800000` (exclusive) | Measured, not derived: `julianday('5373484.4')` answers `5373484.4`, `julianday('5373484.5')` is NULL, and `julianday()` is NULL at both ends of the epoch band. The gap `5373484.5 <= v < 1e12` is digits-only text that neither limb touches — genuine `coalesce` fixpoints, and that is where `'1753660800'` (epoch **seconds**) lands. Consequence: the guard cannot withhold a single row the 后果 B recovery would convert, which is pinned end to end by a column holding both an in-band epoch-text row and a withheld julian row. ## Verification Run on the merged tree (`origin/main` merged, then the branch squashed onto it with the byte-identical tree; final head as pushed). - `pnpm --filter @objectstack/driver-turso test` — **55 files, 1291 tests, 0 failed** (1289 before; the 9 new tests, minus the rewritten one, account for it). - `pnpm --filter @objectstack/driver-turso typecheck` — exit 0. - `pnpm --filter '...@objectstack/driver-turso' typecheck` — downstream consumers, **scope 21 of 82 workspace projects**, exit 0. No in-repo consumer of the three entry points exists outside this package (grepped); the sweep proves the widened third position breaks none of them. - `pnpm exec turbo run build --filter='./packages/*' --filter='./packages/*/*'` — 72/72, exit 0. - `pnpm lint` (repo-wide `eslint . --no-inline-config`) — exit 0. Not narrowed; the whole scan ran. - **Gate families**: `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived **61** commands. All 61 run on the merged, built tree, each exit code captured before any pipe; **61 run, 0 NOT-MEASURED, 0 UNRUN**, reconciled with `--ran` carrying every exit code (a derived zero, not a claimed one). Three families first answered **exit 3 PREREQUISITE NOT MET** (`check:dual-build-cjs-loads`, `check:lean-entry-closure`, `check:type-check-debt`) and were discharged by building the closure they named, never by declaring them inapplicable. - `check:doc-authoring` caught a real finding on the first pass — the refusal string carried a tracker id an operator cannot resolve. The id moved to an adjacent comment and the gate is green. ### Ablation One anchor, `resolveBackfillSql`'s `nonTemporalText: sql.nonTemporalText`, replaced with a predicate that matches nothing — the single funnel both the probe count and the `UPDATE`'s `WHERE` read, so the mutated module behaves exactly as the pre-fix one. Driven through `scripts/ablation-replace.mjs`, which proves the write against the disk: ```text anchor "nonTemporalText: sql.nonTemporalText" x1 -> x0 replace "nonTemporalText: () => '(0)'" x0 -> x1 blob dcbb875 -> 1dd0ca3cfab510d38ace81d4de30bea0975e1995 ``` Result: **5 failed | 24 passed (29)**. The subject is resolved from source by a relative in-package import, so no build sits between the mutation and the run. Red, each for the right reason: the datetime row became `-4707-06-11T12:00:00.000Z`; the time row became `12:00:00`; `'now'` became the run's own wall clock; the plan's `residual` read 3 instead of 4; and the julian row in the mixed column was rewritten. Deliberate positive controls that stayed **green**, so the suite cannot pass vacuously: the above-the-ceiling fixpoint row is still marked canonical; the not-over-broad case (naive, offset, canonical and junk rows) still converges and marks; the band-disjointness measurement is unmoved; both fail-closed cases are unmoved (they take the bare arm, which the mutation does not reach); and every #5770 batching, resumability, failure and identifier-safety test is unmoved. Restore proven, not assumed: blob back to `dcbb875a6bfb`, `git diff HEAD` empty, and whole-tree `git status --porcelain` empty. ## Changeset `.changeset/6009-turso-remote-julian-guard.md`, `@objectstack/driver-turso: minor`. Measured rather than assumed: the package is not private, its `files[]` ships `dist`, and the changed bytes reach it — `nonTemporalTextRowsWithheld` appears 7x in `dist/index.js`, 7x in `dist/index.mjs` and 2x in `dist/index.d.ts`, `RemoteBackfillSqlRules` 4x in the `.d.ts`, and the refusal string once in `dist/index.js`, against a negative control that matches nothing. `minor` rather than `patch` because new exported types and a new report field land; no BREAKING banner, because nothing breaks. ## Acceptance notes - `unresolvedEpochTextRows` and `nonTemporalTextRowsWithheld` deliberately **overlap** and neither is a subset of the other: the guard also matches shapes the digits-only count never sees (a leading `+` or space, `'2026e0'`, a digits-only BLOB — all measured). Subtracting one from the other to make the older field's sentence uniformly true would have produced a count that can go negative; the fields' contracts say which rows are in which half instead. - The convergence phase still issues one statement that matches zero rows when every residual row is withheld, exactly as the local twin does. Noted, not filed: one wasted statement on a column that cannot converge, on a path that already costs a probe. --- _Generated by [Claude Code](https://claude.ai/code/session_01NcPSwnmJHczmTu6FG7NMjE)_ Co-authored-by: Claude <noreply@anthropic.com>
1 parent 99c19b3 commit e2c55ed

6 files changed

Lines changed: 745 additions & 112 deletions

File tree

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
---
2+
'@objectstack/driver-turso': minor
3+
---
4+
5+
`driver-turso`: the REMOTE canonical temporal backfill no longer overwrites a bare-number cell with the date SQLite reads it as (#6009).
6+
7+
`RemoteTransport.mapFieldTypeToSQL` declares every temporal column `TEXT`, so a `Field.datetime` / `Field.time` column can hold a digits-only string such as `'2026'`, `'86400'` or the keyword `'now'`. SQLite's time-value grammar accepts a bare number as a JULIAN DAY and `now` as the wall clock, so for those two shapes `strftime` answers confidently instead of returning NULL and the `coalesce(strftime(…), col)` that is supposed to preserve unparseable values never fires. The convergence `UPDATE` in `backfillRemoteCanonicalColumn` therefore wrote that answer over the stored bytes, and no later run could get them back — measured on better-sqlite3 13.0.3 / SQLite 3.53.4 with the column declared `TEXT`:
8+
9+
```text
10+
'2026' -> -4707-06-11T12:00:00.000Z 'now' -> the wall clock, now
11+
'86400' -> -4476-06-15T12:00:00.000Z '12' -> -4713-12-06T12:00:00.000Z
12+
```
13+
14+
The local (Knex) half of this was fixed for `SqlDriver` in the same tracker; the remote path is a separate module that builds its own statements and did not inherit it.
15+
16+
- **The rows are withheld from the `UPDATE` and they also BLOCK the canonical mark.** Withholding alone would not be enough: a marked column drops the read-side repair, and the raw `'2026'` would then compare as TEXT instead of as the instant the repair reads it as. A withheld row is by construction `col IS NOT canonical`, so it stays inside the probe's `residual`, which the mark already requires to be zero. The column keeps its (unindexed) repair and every query answer is bit-for-bit what it was.
17+
- **The predicate is the driver's own, handed across the module boundary — never copied.** `TursoDriver` now passes `{ canonical, nonTemporalText }` where it passed a bare canonical expression, the second arm being `SqlDriver.sqliteNonTemporalTextSql`. Nothing in the shared READ expression changes: `sqliteCanonicalDatetimeSql` / `sqliteCanonicalTimeSql` still misread a bare number exactly as before, deliberately, per the 2026-08-03 cloud#1005 ruling that refused a heuristic in a public expression that runs on every read.
18+
- **The third position of `probeRemoteCanonicalColumns`, `backfillRemoteCanonicalColumn` and `backfillRemoteCanonicalColumns` now accepts either shape**, so a caller compiled against the earlier release keeps compiling: `RemoteBackfillSqlRules` (`{ canonical, nonTemporalText }`) is the shape to pass, and a bare `CanonicalSqlFor` is FAIL-CLOSED rather than a fallback — with no guard to withhold by, the convergence phase is refused, the column reports `error` and stays unmarked, and reads stay correct on the repair. The 后果 B epoch recovery still runs on that arm. To move off it: pass `{ canonical: <what you passed before>, nonTemporalText: <the driver's sqliteNonTemporalTextSql> }`.
19+
- **New on the per-column report: `nonTemporalTextRowsWithheld`** — what the guard declined to write, or `null` when no guard was supplied and the count was therefore never measured. It is deliberately NOT folded into `unresolvedEpochTextRows`, whose documented meaning is *recorded and harmless to the mark*; these rows are the opposite.
20+
- **A documented sentence is corrected in the same change.** The module said rows outside the epoch-recovery band "do not block the canonical mark, because they are fixpoints of the shared repair". That is true only ABOVE SQLite's julian-day ceiling, where `strftime` returns NULL. Below it — `'12'`, `'2026'`, `'86400'` — the repair answers, the row is not a fixpoint, and dropping the repair changes what it matches. Both halves are now stated with the measurement behind them.
21+
- **The two bands do not meet, so the guard costs the epoch recovery nothing.** A bare number is read as a julian day only for `0 <= v < 5373484.5` (`'5373484.4'` parses, `'5373484.5'` returns NULL); `REMOTE_BACKFILL_EPOCH_MS_MIN` is `1e12`. The epoch-text `UPDATE` is also structurally out of reach for a second, independent reason: its SET wraps `cast(col as real)`, whose `typeof()` is always `'real'`, so the canonical expression takes its `'unixepoch'` limb and never the `coalesce`/julian one.
22+
23+
No read path, no filter compilation and no stored value changes for any column that holds none of this shape: a table with nothing but ordinary legacy rows converges and is marked exactly as before.

‎packages/drivers/driver-turso/src/index.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,9 @@ export {
5555
type RemoteBackfillKind,
5656
type RemoteBackfillLogger,
5757
type CanonicalSqlFor,
58+
type NonTemporalTextSqlFor,
59+
type RemoteBackfillSql,
60+
type RemoteBackfillSqlRules,
5861
type RemoteCanonicalBackfillOptions,
5962
type RemoteCanonicalBackfillReport,
6063
} from './remote-canonical-backfill.js';

0 commit comments

Comments
 (0)