Add experimental ADBC backend - #783
Merged
Merged
Conversation
Adds an experimental `adbc` backend (`backend: adbc`) as a third connection path alongside pyodbc and mssql-python, talking to SQL Server via go-mssqldb instead of an ODBC/DB-API bridge. - New dbt-sqlserver[adbc] extra; driver binary installed separately via the `dbc` CLI (not on PyPI). Full guide in docs/adbc_backend.md. - SQL Server (user/password) authentication only for now. - ? -> @pn placeholder rewriting, quote/bracket-aware so a literal ? in an identifier or string literal never misaligns a binding. - AdapterResponse.rows_affected recovered via a scoped @@rowcount query (ADBC's own cursor.rowcount is always -1), gated to DML statements with no pending result set so it never races an unfetched SELECT (no MARS support) and isn't paid for on DDL/SELECT/transaction-control statements. - Arrow type-code/DataType column metadata mapped to the same SQL Server type names as pyodbc/mssql-python; unrecognized codes raise instead of silently mistyping a column. - cursor.nextset() (unsupported by the driver) drained safely by checking the driver's exception type. CI: new adbc Docker image stage + publish-docker.yml matrix entry, new backend: adbc row in integration-tests-sqlserver.yml (Python 3.13 / SQL Server 2025, continue-on-error while experimental), and devcontainer setup now installs the driver binary automatically. Closes #771
This was referenced Jul 29, 2026
Merged
axellpadilla
added a commit
that referenced
this pull request
Aug 6, 2026
sqlserver__get_empty_subquery_sql neuters a probe query as `select * from (...) where 1 = 0`, but a query opening with a CTE cannot be wrapped that way and is passed through untouched, so it ran in full purely to have its column names read. A snapshot executed its whole staging query once for the probe and again to build the staging table; a contract-enforced model ran twice per build. Draining that result set (previous commit) makes it safe but not cheap, and the cost tracks data volume. sp_describe_first_result_set compiles the query and reports its shape without scanning, which is all this method ever wanted. It is already how sqlserver__get_columns_in_query handles CTEs (#698). The macro cannot be the fix point: its output is also embedded in a CREATE VIEW body by dbt-core's unit-test materialization, where only a bare SELECT is legal, and reading cursor.description of an `exec sp_describe...` batch would describe the procedure's own result shape rather than the query's. Reported names and types are unchanged. Reading cursor.description reports Python classes, which collapse whole type families -- every integer width arrives as int, every string type as varchar -- so the describe path is mapped back onto exactly those names and fed through Column.create as before, leaving the dbt_sqlserver_use_native_string_types flag working without the mapping knowing it exists. Rather than guess, the describe path declines and lets the caller execute whenever it cannot guarantee agreement: a type outside the mapping, a query sp_describe_first_result_set refuses to describe such as one reading a #temp table, or a describe that returns nothing. Every gap degrades to slower, never to silently different. The name a type maps back to is also backend-specific, not just SQL Server's: mssql-python decodes uniqueidentifier as uuid.UUID, datetimeoffset as datetime and sql_variant as str, where pyodbc hands back str, bytearray and bytearray for the same three. A fixed table can't serve both, so _executed_name_for_system_type follows the backend in use (folded in from master's 3354428, minus its ADBC Arrow-map portion -- no ADBC backend on this branch). datetimeoffset on pyodbc depends on whether add_query has registered the -155 output converter yet, so it's left unmapped there and executes, agreeing with itself by construction; uniqueidentifier on pyodbc depends on the process-global pyodbc.native_uuid flag, read at probe time. TestCteProbeAvoidsExecution pins both halves: that a CTE-headed probe no longer executes its query, and that the two branches agree across every probed type on the backend under test. Note: dropped the ADBC-backend guard clause from master's version (checking is_adbc_backend before trusting the describe path), since v1.11 has no ADBC backend (added later in #783) to disagree with it. Reapply that guard if ADBC is ever backported here too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 7cee257, with the backend-aware type resolution folded in from 3354428)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an experimental
adbcbackend (backend: adbcin a profile target) as a third connection path alongsidepyodbcandmssql-python. ADBC (Arrow Database Connectivity) talks to SQL Server viago-mssqldbinstead of an ODBC/DB-API bridge, avoiding row-based marshalling overhead.dbt-sqlserver[adbc]extra (adbc-driver-manager,pyarrow); the driver binary itself ships separately via thedbcCLI (not on PyPI). Full setup/config/troubleshooting guide indocs/adbc_backend.md;README.mdgets the new backend row and a matching setup section.?to T-SQL@pN(go-mssqldb doesn't supportqmarkbinding), skipping?characters that appear inside bracket-quoted identifiers or string literals so a literal?never misaligns a binding.AdapterResponse.rows_affectedis recovered via a scoped@@ROWCOUNTquery, since ADBC's owncursor.rowcountis always-1: only run when the statement is DML and has no pending result set to fetch, so it never races an unfetched SELECT on the same connection (no MARS support) and isn't paid for on DDL/SELECT/transaction-control statements. Other backends' rowcount values are untouched.DataType/type-code column metadata is mapped to the same SQL Server type names pyodbc/mssql-python report; an unrecognized type code raises instead of silently mistyping a column.cursor.nextset()(unsupported by the ADBC driver) is drained safely by checking the driver's exception type rather than pyodbc-style return values.CI / tooling
adbcDocker CI image stage (installs thedbcdriver-manager CLI + mssql driver binary) and matrix entry inpublish-docker.yml.backend: adbcrow inintegration-tests-sqlserver.ymlon Python 3.13 / SQL Server 2025 (latest), markedcontinue-on-errorsince the backend is still experimental.setup_env.sh) installs the ADBC driver binary and setsADBC_DRIVER_PATHautomatically on container start.Closes #771
Test plan
pytest tests/unit— 400 passedpytest tests/functionalagainst a live SQL Server withbackend: adbc— 320 passed, 48 skipped, 2 xfailed, 0 failedpre-commit run --all-files— clean (ruff check/format, mypy, etc.)publish-docker.ymlonly rebuilds CI images on push tomaster, so the newCI-3.13-adbcimage won't exist until that runs once after merge — the new integration-test row will fail-but-not-block until then (or trigger it manually viaworkflow_dispatch).