Skip to content

sqlfmt: omit generated columns from diff -r sql - #11476

Open
vishnujayvel wants to merge 1 commit into
dolthub:mainfrom
vishnujayvel:fix/11445-diff-sql-generated-columns
Open

sqlfmt: omit generated columns from diff -r sql#11476
vishnujayvel wants to merge 1 commit into
dolthub:mainfrom
vishnujayvel:fix/11445-diff-sql-generated-columns

Conversation

@vishnujayvel

@vishnujayvel vishnujayvel commented Aug 9, 2026

Copy link
Copy Markdown

dolt diff -r sql no longer assigns generated columns in the INSERT and UPDATE statements it emits, so its output can be replayed. Previously the engine rejected those assignments; where the diff is emitted as DELETE+INSERT — keyless tables, and tables whose primary key is itself generated — replay deleted the old row and then failed to reinsert it, silently losing the row. dolt_patch()'s statement output is fixed by the same change.

  • schemaFromCreateTableStmt now populates Generated, Virtual, and OnUpdate on schema.Column, so IsGenerated() is correct for schemas built from SHOW CREATE TABLE.
  • SqlRowAsUpdateStmt skips generated columns, and GenerateDataDiffStatement drops them from colsToUpdate.
  • STORED and VIRTUAL generated columns are both covered, on primary-keyed and keyless tables.

Fix #11445

dolt diff -r sql was emitting STORED generated columns in INSERT and
UPDATE statements, which the engine rejects on replay.

Populate Column.Generated and Virtual in schemaFromCreateTableStmt so
IsGenerated works on the CLI SQL-diff path, and skip generated columns
in SqlRowAsUpdateStmt / GenerateDataDiffStatement (mirroring the
existing INSERT filters).

Fixes dolthub#11445
@elianddb

Copy link
Copy Markdown
Contributor

@vishnujayvel Could you shorten the description to be similar to our teams' release notes? They come up as release notes and this change is relatively small.

@vishnujayvel

Copy link
Copy Markdown
Author

Done — shortened the description to match the team's release-note style.

Thanks again for the repro in #11445 — it named both failure points, which is why this landed as a small diff.

I've parked the long-form root cause and test plan below so it stays on the PR for review without ending up in the release notes.

Original description (root cause + test plan)

dolt diff -r sql was writing generated columns into the INSERT and
UPDATE statements it emits. The engine rejects any assignment into a generated
column, so the SQL patch could not be replayed. Where the diff is emitted as
DELETE+INSERT — keyless tables, and tables whose primary key is itself a
generated column — replaying a patch deleted the old row and then failed the
INSERT, silently losing the row.

The same emission path backs the dolt_patch() table function, so its
statement output is fixed by the same change.

Fixes #11445

Thanks to @elianddb (Elian) for the clear repro and for tracing both failure
points in #11445.

Got

UPDATE `t` SET `a`=99,`g`=100 WHERE `id`=1;
INSERT INTO `t` (`id`,`a`,`g`) VALUES (2,20,21);

Want / after this change

UPDATE `t` SET `a`=99 WHERE `id`=1;
INSERT INTO `t` (`id`,`a`) VALUES (2,20);

Root cause (two parts)

  1. Schema construction on the CLI SQL-diff path.
    schemaFromCreateTableStmt (go/cmd/dolt/commands/diff.go) already
    parsed col.Type.GeneratedExpr into a buffer, but wrote
    Generated: "" and Virtual: false into schema.Column. That made
    Column.IsGenerated() always false for schemas built from
    SHOW CREATE TABLE, so the existing INSERT filters in
    InsertStatementPrefix / SqlRowAsTupleString never fired on this path.

  2. UPDATE emission never filtered generated columns.
    SqlRowAsUpdateStmt (go/libraries/doltcore/sqle/sqlfmt/row_fmt.go
    not diff.go) only gated on colsToUpdate, unlike the INSERT helpers
    which already skip col.IsGenerated(). Even with an honest schema,
    UPDATE would still emit generated columns when their values changed
    alongside base columns.

Fix

  • Populate Generated from the already-formatted expression buffer and
    set Virtual from GeneratedExpr != nil && !Stored (and fill
    OnUpdate from its parallel buffer while touching the same struct).
  • Skip col.IsGenerated() columns in SqlRowAsUpdateStmt, and drop them
    from colsToUpdate in GenerateDataDiffStatement so the UPDATE column
    set never includes them.

Test plan

  • Unit: TestSqlRowAsUpdateStmtSkipsGeneratedCols in
    go/libraries/doltcore/sqle/sqlfmt/row_fmt_test.go, mirroring
    TestInsertStatementPrefixSkipsGeneratedCols.
    • Red on base: expected
      UPDATE ... SET a='x' WHERE id=1 vs actual
      UPDATE ... SET a='x',c='x!' WHERE id=1.
    • Green at fix: pass.
  • Full package: go test ./libraries/doltcore/sqle/sqlfmt/ -count=1 → ok.
  • Integration: bats case
    sql-diff: stored generated columns omitted from INSERT and UPDATE in
    integration-tests/bats/sql-diff.bats — INSERT+UPDATE, assert g absent,
    replay after dolt reset --hard, assert final rows. Passed with the
    fixed binary on PATH, and fails against an unpatched binary. The full
    sql-diff.bats file was not run to completion locally (slow host); its
    first three cases (INSERT/UPDATE/DELETE reconcile) pass with the fixed
    binary, and CI covers the rest.
  • Manual end-to-end repro (same as dolt diff -r sql emits statements that assign generated columns #11445 Got/Want): fixed binary omits
    g from both statements; replay succeeds; rows (1,99,100), (2,20,21).
    Also verified by direct comparison against an unpatched build: VIRTUAL
    column tables, keyless tables, generated-column primary keys, and
    dolt_patch() output are corrected; DELETE statements and non-generated
    tables are byte-identical.

Notes

  • Coverage: STORED and VIRTUAL generated columns are both fixed (VIRTUAL
    previously leaked into INSERT column lists as NULL values and broke
    replay the same way), on primary-keyed and keyless tables — including
    when the generated column is itself the primary key. Filters use
    IsGenerated() (any non-empty Generated string), consistent with the
    existing INSERT path.
  • Scope of the newly populated fields: nothing on any diff render path
    reads Generated/OnUpdate/Virtual except the generated-column
    filters this PR relies on. ALTER-DDL statements come from
    tds.AlterStmts and the raw SHOW CREATE TABLE text, not from this
    parsed schema, so schema-diff output is unchanged (verified empirically
    at base and at this branch: ALTER/tabular/JSON outputs byte-identical).

@elianddb elianddb self-assigned this Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dolt diff -r sql emits statements that assign generated columns

3 participants