Skip to content

Commit f4b8f37

Browse files
authored
FEAT: add standalone conda build pipeline + build-validation tooling (#744)
### Work Item / Issue Reference > [AB#47315](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/47315) ### Summary This pull request introduces a new standalone OneBranch pipeline for building, validating, and consolidating self-contained `mssql-python` conda packages across all supported platforms and Python versions. The pipeline is carefully structured to ensure per-platform correctness, best-effort artifact gathering, and robust isolation between build legs. It also includes a template job for consolidating conda artifacts and a `.gitattributes` update for shell scripts. **New Standalone Conda Build Pipeline** *Major features and structure:* - **Adds `conda-build-pipeline.yml`**: Implements a full multi-stage pipeline that builds and validates self-contained `mssql-python` conda packages for Windows, macOS, and Linux, including cross-platform builds (e.g., win-arm64, osx-arm64, linux-aarch64) using platform-appropriate agents and emulation where needed. Each leg is carefully isolated to prevent failures in one subdir from impacting others, and best-effort logic is used for cross-arch builds. - **Artifact Consolidation**: - **Adds `consolidate-conda-artifacts-job.yml`**: Provides a job template that collects all per-platform conda packages into a single artifact, preserving the subdir structure. The consolidation is best-effort in the build pipeline, with hard gating deferred to the release pipeline to avoid blocking primary wheel deliverables. **Build and Release Process Improvements** - **Best-Effort and Gating Strategy**: The pipeline is designed so that conda packaging failures do not block wheel releases. Completeness and artifact presence are strictly enforced only at release time, not during the build. [[1]](diffhunk://#diff-e258bc0778f9b125cb5214aecba116312126f1a37e3db101606171b05a2b8302R1-R349) [[2]](diffhunk://#diff-6a4150f95dfbb94f92e9ec3f674414deb459b23077ac34a99deb40b0479aee29R1-R129) **Cross-Platform and Cross-Arch Handling** - **Cross-Build Logic**: Implements cross-building for win-arm64, osx-arm64, and linux-aarch64 using CONDA_SUBDIR and QEMU emulation as appropriate, with architecture-specific validation and clear blast-radius isolation to prevent a single platform???????????????s failure from impacting others. **Repository Hygiene** - **Shell Script Line Endings**: - Adds `OneBranchPipelines/scripts/.gitattributes` enforcing LF line endings for `.sh` files under that folder (the conda recipe scripts are covered separately by `conda/.gitattributes`). **Build-Validation Tooling + Tests** *The "build-validation tooling" named in the title -- the exact set of files in this slice:* - **`OneBranchPipelines/scripts/build_conda_packages.py`**: the single cross-platform orchestrator (gather wheels -> Miniforge/conda-build -> build the self-contained package -> masking-immune audit -> solve a fresh env + import + driver-load + optional reachability gate -> stage). Replaces the former per-OS PowerShell/bash build scripts. - **`OneBranchPipelines/steps/conda-build-validate-step.yml` + `-posix.yml`**: the Windows / POSIX step templates that invoke the orchestrator. - **`eng/scripts/audit_bundled_binaries.py`, `assert_pe_machine.py`, `assert_macho_arch.py`, `_conda_pkg.py`**: masking-immune RUNPATH, ELF/PE/Mach-O architecture, required Linux driver-inventory, and package-reader validation. - **`requirements.txt`**: declares `zstandard` so `.conda` payload tests execute on Python versions without the stdlib `compression.zstd` backend. - **`conda/driver_load_probe.py`**: the DB-less driver-load proof. - **`tests/test_029_bundled_binary_audit.py`, `test_030_pe_machine_assert.py`, `test_033_driver_load_probe.py`, `test_034_conda_verify_cwd.py`, `test_035_conda_macho_assert.py`**: unit coverage for Linux payload completeness/RUNPATH, PE and Mach-O architecture checks, the driver-load probe, neutral-cwd verification, and blocking win-arm64 environment installation. **Not in this slice (deliberately):** the live `Encrypt=yes` TLS gate (`conda/tls_connect_probe.py` and its `test_028`/`test_031`) is a *release*-pipeline concern -- the build pipeline never enables it (no leg sets `CONDA_TLS_PROBE_CONN`), so it ships with the release slice, not here. The masking-immune static RUNPATH audit already guards the OpenSSL layout at build time.
1 parent 4d029f8 commit f4b8f37

17 files changed

Lines changed: 4972 additions & 0 deletions

‎OneBranchPipelines/conda-build-pipeline.yml‎

Lines changed: 405 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Consolidate Conda Artifacts Job Template
2+
# Collects the per-platform self-contained mssql-python conda packages (which vendor
3+
# the ODBC payload) that each build leg staged under conda/<subdir>/ and gathers
4+
# them into a single conda/ tree for the release pipeline to publish.
5+
#
6+
# BEST-EFFORT (build pipeline): conda is a downstream repackage of the ESRP-signed
7+
# wheels and must NEVER block the primary wheel deliverable. An empty set WARNS here;
8+
# a partial set is reported by its package/subdir counts but does not fail consolidation.
9+
# The HARD count gate lives in the release pipeline (which refuses to publish an
10+
# incomplete conda set), symmetric with how wheel/odbc drops are best-effort collected.
11+
#
12+
# Expected packages (validated conda subdirs). The self-contained mssql-python
13+
# package (which vendors the ODBC payload) is emitted per-Python by each build leg;
14+
# there is NO separate companion package:
15+
# win-64 : 5 py x mssql-python = 5
16+
# win-arm64 : 3 py x mssql-python (py3.12-3.14, cross-built on x64) = 3
17+
# osx-64 : 5 py x mssql-python (Intel Mac, native) = 5
18+
# osx-arm64 : 5 py x mssql-python (Intel Mac, cross-built) = 5
19+
# linux-64 : 5 py x mssql-python (glibc x86_64 host, native) = 5
20+
# linux-aarch64 : 5 py x mssql-python (x86_64 host + QEMU, best-effort) = 5
21+
# ------------------------------------------------------------------------------
22+
# TOTAL (PyPI parity minus musllinux) = 28
23+
# win-arm64 is cross-built for py3.12-3.14 only (py3.10/3.11 have no win-arm64 deps
24+
# on Anaconda defaults); its runtime import is skipped on the x64 host, so its arch
25+
# is enforced by the PE-machine assert (assert_pe_machine.py). musllinux (no conda
26+
# musl subdir) is intentionally NOT conda-built. This job is BEST-EFFORT and never
27+
# hard-fails on a short set; the release pipeline's conda-release-step enforces the
28+
# hard gate (required subdirs present + complete Python matrix) before publish.
29+
parameters:
30+
- name: oneBranchType
31+
type: string
32+
default: 'Official'
33+
# Artifact item pattern the consolidate job downloads. Defaults to the integrated
34+
# wheel-pipeline leg artifacts; the standalone conda-build pipeline overrides it with
35+
# its drop_Conda* leg artifacts.
36+
- name: downloadItemPattern
37+
type: string
38+
default: |
39+
drop_Win_*/**
40+
drop_MacOS_*/**
41+
drop_Linux_*/**
42+
drop_ODBC_BuildAll_*/**
43+
44+
jobs:
45+
- job: ConsolidateArtifacts
46+
displayName: 'Consolidate All Conda Packages'
47+
# Preserve partial artifacts after failed dependencies; do not run after cancellation.
48+
condition: succeededOrFailed()
49+
50+
pool:
51+
type: linux
52+
isCustom: true
53+
name: Azure Pipelines
54+
vmImage: 'ubuntu-latest'
55+
56+
variables:
57+
# Consolidation only moves files; no binaries to scan.
58+
- name: ob_sdl_binskim_enabled
59+
value: false
60+
- name: ob_outputDirectory
61+
value: '$(Build.ArtifactStagingDirectory)'
62+
63+
steps:
64+
- checkout: self
65+
fetchDepth: 1
66+
67+
# The conda packages are staged INSIDE the mssql-python build-leg artifacts
68+
# (drop_Win_*, drop_MacOS_*, drop_Linux_*) under conda/<subdir>/. Scope the
69+
# download to those stages so every leg's self-contained mssql-python conda is
70+
# gathered in one place. (drop_ODBC_BuildAll_* is included only for its wheels,
71+
# which ride along and are ignored below -- we pick only *.conda / *.tar.bz2.)
72+
- task: DownloadPipelineArtifact@2
73+
displayName: 'Download All Platform Artifacts'
74+
inputs:
75+
buildType: 'current'
76+
itemPattern: ${{ parameters.downloadItemPattern }}
77+
targetPath: '$(Pipeline.Workspace)/all-artifacts'
78+
79+
- bash: |
80+
set -euo pipefail
81+
echo "Collecting conda packages (preserving <subdir>/ layout)..."
82+
mkdir -p "$(ob_outputDirectory)/conda"
83+
84+
# Copy every mssql-python* conda package into conda/<subdir>/. Each build
85+
# leg wrote the self-contained mssql-python package under a conda/<subdir>/
86+
# folder, so the parent dir name IS the target subdir.
87+
found=0
88+
while IFS= read -r p; do
89+
subdir=$(basename "$(dirname "$p")")
90+
mkdir -p "$(ob_outputDirectory)/conda/$subdir"
91+
cp -v "$p" "$(ob_outputDirectory)/conda/$subdir/"
92+
found=1
93+
done < <(find "$(Pipeline.Workspace)/all-artifacts" -type f \( -name 'mssql-python*.conda' -o -name 'mssql-python*.tar.bz2' \))
94+
95+
echo ""
96+
echo "Consolidated conda tree:"
97+
find "$(ob_outputDirectory)/conda" -type f | sort
98+
99+
PKG_COUNT=$(find "$(ob_outputDirectory)/conda" -type f \( -name '*.conda' -o -name '*.tar.bz2' \) | wc -l)
100+
echo ""
101+
echo "Per-subdir conda package counts:"
102+
for d in "$(ob_outputDirectory)"/conda/*/; do
103+
[ -d "$d" ] || continue
104+
sub=$(basename "$d")
105+
n=$(find "$d" -type f \( -name '*.conda' -o -name '*.tar.bz2' \) | wc -l)
106+
printf ' %-14s %s\n' "$sub" "$n"
107+
done
108+
echo "Total conda package count: $PKG_COUNT (full PyPI-parity set = 28)"
109+
110+
# BEST-EFFORT: never exit non-zero — an empty set emits a warning; a partial set
111+
# remains visible in the counts above. A conda hiccup must not block the wheel
112+
# release. The release pipeline's conda-release-step enforces the hard gate
113+
# (required subdirs + full Python matrix) before anything is published.
114+
if [ "$found" != "1" ]; then
115+
echo "##vso[task.logissue type=warning]No conda packages found in the build-leg artifacts."
116+
else
117+
echo "Collected $PKG_COUNT conda package(s) (best-effort; release-time gate enforces completeness)."
118+
fi
119+
displayName: 'Consolidate conda packages'
120+
121+
- task: PublishPipelineArtifact@1
122+
displayName: 'Publish Consolidated Conda Artifacts'
123+
inputs:
124+
targetPath: '$(ob_outputDirectory)'
125+
# Distinct name so it does not collide with the wheel consolidate artifact
126+
# (drop_Consolidate_ConsolidateArtifacts) or the odbc one
127+
# (drop_ConsolidateOdbc_ConsolidateArtifacts) in the same run. Matches the
128+
# OneBranch auto-name for a stage named `ConsolidateConda`.
129+
artifact: 'drop_ConsolidateConda_ConsolidateArtifacts'
130+
publishLocation: 'pipeline'
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
*.sh text eol=lf

0 commit comments

Comments
 (0)