fix(table): stop Sch-M locks spanning the load in table builds

[Benjamin-Knight]

Resolve #819

Problem

A table_refresh_method: dml model held a Sch-M lock on its scratch table for the
whole load. Sch-M is the one lock mode incompatible with the Sch-S lock every
metadata reader takes, so while a slow model built, it blocked metadata readers in every
other session on the database — database-wide sys / INFORMATION_SCHEMA scans, a
concurrent dbt run's catalog and column lookups, SSMS's object explorer. None of those
asked for the scratch table by name.

Cause

Two independent causes; fixing either alone would have changed nothing.

1. The scratch table was built by one fused SELECT * INTO, which holds Sch-M on the
   new object from the start of the statement to the end, not for the instant of creation.
2. That build ran inside the materialization's ambient transaction. Locks are held to
   commit, so a split create inside the transaction holds Sch-M just as long as the
   fused statement did.

Fix

• Create the table empty, then load it with a separate INSERT ... WITH (TABLOCK) — still
  minimally logged, as SELECT INTO was. Hoisted into
  sqlserver__get_create_table_empty_sql / sqlserver__get_tablock_insert_sql and applied
  to the dml refresh path, the default table build, and every other create_table_as
  caller (incremental full refresh and temp builds, snapshots).
• Every statement in the build passes auto_begin=False, so each autocommits and drops its
  catalog locks as it finishes — the same treatment the incremental temp build already got.
  This also stops the clustered columnstore index built after the load from holding its
  locks to the end of the materialization.
• The dml refresh's DELETE+INSERT swap commits as soon as it completes, instead of
  running on through index reconciliation, masks, grants and persist_docs while holding X
  locks on the target. The swap stays atomic — it is now the whole of its own transaction.

Worth knowing

• A pre-hook with inside_transaction: true (dbt's default) still re-couples the build.
  auto_begin=False only declines to open a transaction; a statement still joins one
  already open. Documented in the macro and the changelog. Same trade-off as the
  incremental path.
• A crashed run can now leave a __dbt_tmp intermediate behind, since the build commits
  standalone. The existing OBJECT_ID guard for adapter-generated throwaways drops it on
  the next run; a fresh create of a real target still surfaces Msg 2714 rather than
  destroying an object dbt doesn't know about.
• Index and mask reconciliation reconverge on the next run if the swap fails — both
  reconcile against the config rather than applying a delta.
• Snapshots get the split, not the lock change; they keep their own transaction semantics.

Tests

• New tests/unit/adapters/mssql/test_table_build_sql.py: renders the batch that actually
  ships (non-contract, contract, temp build) end to end, so a missing statement terminator
  or an unescaped EXEC literal fails without a database. Plus a repo-wide check that no
  macro fuses a create with its load again.
• Re-anchored the query-options hint assertions on the split build. The old swap pattern
  matched the scratch load as readily as the swap, so it would have passed on a log where
  only the swap lost its hint — silently testing nothing. Each pattern is now anchored on
  what its statement selects FROM, and verified mutually exclusive against the emitted SQL.

[axellpadilla]

Can we move creation of the empty intermediate table before run_hooks(pre_hooks, inside_transaction=True) and remove the early commit_if_open()?

That keeps the table-creation Sch-M outside the model transaction while allowing the long INSERT to run without Sch-M, preserving the transaction boundary for transactional hooks. The rename/swap can acquire Sch-M only at the final cutover.

Proposed lock/transaction flow

OUTSIDE TRANSACTION
│
├─ outside-tx pre-hooks
│
├─ CREATE empty intermediate table
│      └─ Sch-M
│         short, released immediately
│
▼
BEGIN TRANSACTION
│
├─ inside-tx pre-hooks
│
├─ INSERT INTO intermediate
│      └─ long-running load
│         no Sch-M
│
├─ rename / swap
│      └─ Sch-M acquired here
│
├─ transactional tail
│
└─ COMMIT
       └─ Sch-M released

The key property is:

Current problematic shape
begin├──────────────────── long load with Sch-M ────────────────────┤ commit

Proposed
Sch-M ├─ create ─┤
                 begin├──────── long load, no Sch-M ────────┤
                                                       ├─ Sch-M ─┤ commit
                                                          swap

One nuance: the Sch-M acquired by the rename/swap is held until COMMIT, so keeping the post-swap transactional tail short still matters.

[Benjamin-Knight]

Not the create — it needs the model SQL to be bindable (the tmp view, or the describe probe on the contract path), so it can't run before a pre-hook that stages what the model reads.

[Benjamin-Knight]

Ok, there is a lot of complex changes in the transaction flow here that definitely need another pair of eyes on them @axellpadilla

I've also added a fix for what I think was a race condition in the usage of the linked server in the open query tests that would intermittently fail my local and the GitHub CI flows.

Default table path (rename swap)

BEGIN  ← first in-tx pre-hook statement, if any
├─ in-tx pre-hooks
│     ▸ scope gate sampled here: transaction_is_open() and scope == 'build'
├─ CREATE VIEW model__dbt_tmp_vw
├─ SELECT TOP 0 * INTO model__dbt_tmp       ← Sch-M, but TOP 0 moves no rows
COMMIT                                       ← Sch-M released, effectively instant
   │
   ├─ INSERT … WITH (TABLOCK)                ← autocommitted; X table lock, never Sch-M
   ├─ CREATE CLUSTERED COLUMNSTORE INDEX     ← autocommitted, on a private name
   │
BEGIN
├─ sp_rename target → backup                 ← Sch-M on the live name
├─ sp_rename intermediate → target
├─ apply_masks                               ← inside: a failure rolls the swap back
├─ in-tx post-hooks
COMMIT                                       ← cutover atomic; Sch-M released
   │
   ├─ create_indexes                         ← the long one, now outside
   ├─ grants / denies / persist_docs
   └─ drop backup, outside-tx post-hooks

INSERT ... WITH (TABLOCK) takes an exclusive table lock, which is compatible with Sch-S
— so the long load never blocks a metadata reader. The hint is also what keeps it minimally
logged; it is not there to be removed for "less blocking".

table_refresh_method: dml

   ├─ scratch build (auto_begin=False)        ← create empty, then TABLOCK load
BEGIN
├─ DELETE + INSERT swap                       ← X locks on the target
├─ in-tx post-hooks                           ← now atomic with the swap; previously not
COMMIT
   │
   ├─ reconcile_indexes → apply_masks         ← reconcile first: index drops before re-masking
   ├─ DROP scratch table
   └─ grants / denies / persist_docs

The macro no longer commits its own swap. It leaves the transaction open and returns
schema_match plus the scratch relation, so the shared tail owns the boundary and picks the
index strategy. On the schema-changed fallback it renames the scratch table into place, masks
inside the transaction, and returns refresh_relation: none — that name is already vacated.

full_refresh_build: prebuilt

   ├─ drop existing, rebuild in place         ← marker committed onto its own transaction
├─ apply_masks                                ← inside the load's transaction
├─ in-tx post-hooks
COMMIT
   ├─ create_indexes / grants / denies / persist_docs

No swap, so a mask failure leaves an empty target carrying dbt_full_refresh_incomplete
rather than rolling back to the old table. That is prebuilt's existing trade, now documented.

incremental

Same gate and same split build on the create_table_as branches. Fresh creates now stage
through __dbt_tmp and swap — they previously built straight into the real target, so a
failed first load committed an empty table and the next run merged into it. Silent data
loss. The append/merge path is untouched: the strategy DML is the cutover, so it keeps its
transaction.

---

What is atomic with what

The transaction spans in-tx pre-hooks → the cutover → in-tx post-hooks. That is what a hook
declaring transaction: true is asking for: atomicity with the model. Index reconciliation,
grants, denies and persist_docs are the adapter's own housekeeping and were never part of
that promise.

Masks are the exception. On a path that builds a new table it carries no masks until
apply_masks runs, so running that after the cutover committed would leave the new table live
with its columns exposed. Masks stay inside the transaction there.

Two behaviour changes

1. In-tx post-hooks now run before index creation (masks are unaffected — they still run
   first). A post-hook that needs indexes present should declare transaction: false, which
   runs it after the whole tail.
2. If you create indexes from post-hooks — the idiom predating the indexes config —
   drop_unmanaged_indexes: true now drops them in the same run, and such an index on a masked
   column trips the index-key check on SQL Server before 2022.

New config

	Transaction covers	Pre-hook rolls back with a failed load	#819 fixed
pre_hook_transaction_scope: schema	pre-hooks + CREATE VIEW + empty CREATE	no	yes
pre_hook_transaction_scope: build	pre-hooks + the whole build	yes, except below	no

dbt_sqlserver_pre_hook_schema_scope supplies the default. It ships False (meaning build)
so current behaviour is preserved, and is expected to flip in a later release.

Inert where it cannot help: a model with no transactional pre-hook always takes the fixed
path. On dml and prebuilt the build joins a pre-hook's open transaction with nothing
committing in between, so schema changes nothing there. And build cannot deliver rollback
on prebuilt or an incremental --full-refresh — both commit an in-progress marker before the
load precisely so it survives a failure. transaction: false on the hook remains the remedy.

Full detail, including the caveats, in docs/transaction_scope.md.

Also in here

• auto_begin=False on seven read-only catalog probes. They were reopening the ambient
  transaction and dragging the tail's mask and index DDL back inside it. A unit test enforces
  the rule so a new probe cannot reintroduce it.
• sqlserver__create_table_as split into stage and load halves; create_table_as is now their
  concatenation, so snapshots and the incremental temp build emit the same statements in the
  same order. A test pins whole == stage + load.
• adapter.transaction_is_open() — the config-derived proxy is wrong in both directions
  (run_hooks skips hooks whose SQL renders empty; mark_full_refresh_incomplete leaves a
  transaction open with no hook involved).
• Podman make targets for the local test server, for anyone who would rather not run Docker.
• An unrelated test fix riding along: the openquery tests hard-coded an instance-wide linked
  server name, so concurrent pytest-xdist workers dropped and recreated each other's server
  mid-run. Each worker now gets its own.

Verification

Functional suites against SQL Server 2022 (constraints, hooks, snapshots, grants, masks,
denies, indexes, dml refresh, prebuilt, incremental, concurrency) plus 601 unit tests. The two
pre_hook_transaction_scope tests assert opposite rollback outcomes and both pass, so the
config demonstrably changes behaviour rather than merely compiling.

[axellpadilla]

@Benjamin-Knight Amazing will set time aside to check this in deep

[Benjamin-Knight]

pre_hook_transaction_scope is probably one of the larger changes, the intent is that we swap behaviour from current to schema only at some point and users opt in to pre hooks holding the transaction open for the entire build. In most cases we do not want the pre hook to do so and schema is the right call, but there are use cases where the pre hook does need to roll back properly but because of the schema lock this causes users should opt in when they need it rather than cause schema locks for the entire project.

[axellpadilla]

I will try some ideas too when checking this, maybe we could use a different connection to create the table at the right time?, I'm currently using latest rc3 with transaction off but I think this PR is the only missing blocker

[Feckinotter]

I'm using rc3 with transaction on, other than the schema lock no issues so far. But the schema lock can cause issues, I've also seen an increase in other locks but I'm not sure these are rc3 related as they are parralellism inside queries deadlocking, my only concern is I never had these before transactions in DBT. (Ben on personal account)

[axellpadilla]

@Feckinotter yes there should be some limits on parallelism but I'm also not sure that kind of in-process locking have something to do with transactions, here are other best practices that help both with and without transactions https://github.com/dbt-msft/dbt-sqlserver/blob/master/docs/sqlserver-best-practices.md