Recon Core

Open-source Reconciliation as Code framework for proving source-target data equivalence.

Recon Core helps data teams define equivalence in versioned contracts, compile explicit execution plans, run repeatable checks, and generate evidence that shows what matched, what failed, and why.

Current Status

Recon Core is pre-alpha.

Implemented today:

• Python package skeleton,
• recon --version,
• recon init <project_name>,
• recon parse,
• recon compile,
• duplicate-key-safe YAML loading for authored resources,
• structural equivalence contract parsing,
• local non-contract resource source-file indexing in target/manifest.json,
• target/manifest.json generation,
• compiled contract and compiled checks YAML artifacts,
• built-in recon_core.basic_equivalence expansion,
• explicit sum metric compilation,
• limited contract-level sampling.default_policy metadata on compiled checks,
• adapter-aware recon compile --render-sql,
• selected profile/target loading for adapter-aware compile,
• adapter API, registry, and support-state capability validation foundation,
• in-core DuckDB local development adapter behind recon-core[duckdb],
• DuckDB SQL rendering for currently emitted typed check plans,
• compiled SQL artifacts under target/compiled_sql/,
• structured service results and diagnostics,
• first recon run check-engine boundary for already compiled checks,
• relation-backed same-context DuckDB row_count_diff execution through recon run,
• relation-backed same-context DuckDB grain-key safety execution for compiled null-key, duplicate-key, missing-key, and extra-key checks through recon run when the internal local/dev scan guard verifies a project-local DuckDB file under the size cap and both relation endpoints resolve to local base tables.

Not implemented yet:

• explicit authored checks beyond supported check-pack and metric compilation,
• full sampling, tolerance, schema, and CDC policy engines,
• adapter metadata access,
• adapter execution beyond the current same-context DuckDB row-count and grain-key safety paths,
• query endpoint execution,
• aggregate and row-level value check execution,
• generated run-result artifacts,
• evidence writers.

The documentation in this repository defines the intended framework behavior. The current implementation is being built milestone by milestone.

What Recon Is For

Recon is for proving that one data output matches another according to an explicit contract:

source database -> warehouse replica
old warehouse output -> new warehouse output
Bronze layer -> Silver layer
old business metric -> new business metric

Good use cases include:

• CDC validation,
• source-target reconciliation,
• warehouse migration validation,
• pipeline refactor validation,
• medallion layer reconciliation,
• business logic equivalence,
• sign-off evidence before production cutover.

What Recon Is Not

Recon is not:

• a generic data quality platform,
• an ingestion or CDC movement tool,
• a warehouse transformation framework replacement,
• an MDM or fuzzy matching platform,
• an automatic data repair tool.

Recon complements transformation, ingestion, and data quality tools by focusing on source-target equivalence.

Quick Start

For local development from this repository:

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,duckdb]"
recon --version
recon init ecommerce_recon
cd ecommerce_recon
recon parse

The generated starter project includes:

ecommerce_recon/
  recon_project.yml
  .gitignore
  connections/
    profiles.yml.example
  contracts/
  check_packs/
  macros/
  sample_policies/
  tolerances/
  schema_policies/
  target/
  reports/
  state/

check_packs/, sample_policies/, tolerances/, schema_policies/, and macros/ are indexed by recon parse as source-file metadata in target/manifest.json. Recon still parses contract YAML only; local check-pack, policy, and macro semantics remain future work.

recon parse performs structural project parsing and writes target/manifest.json. recon compile expands the currently supported check packs and explicit metrics into compiled YAML artifacts under target/. recon compile --render-sql also loads connections/profiles.yml, resolves the selected profile target, and writes DuckDB-rendered SQL under target/compiled_sql/ when referenced connections use the in-core duckdb adapter and resolve to the same adapter connection config. If you use the generated profile example, set RECON_DUCKDB_PATH or edit connections/profiles.yml before running recon compile --render-sql. recon run loads already compiled check artifacts and routes them through the first check-engine boundary. It can execute relation-backed same-context DuckDB row_count_diff checks when matching compiled-contract artifacts and runtime profiles are available. Grain-key safety checks execute only when the runtime context passes the internal local/dev bounded scan guard. That guard requires a project-local DuckDB file under the size cap and relation endpoints that resolve to local base tables, not views or externally backed relations; otherwise their scan-heavy paths remain not executable. It reports missing, invalid, empty, unsupported, blocked, or not-executable compiled checks with structured runtime diagnostics. It does not execute query endpoints, aggregate checks, row-level value checks, or write run-result, evidence, report, failure-detail, state, or sink artifacts yet.

Core Idea

The main object in Recon is an equivalence contract.

An equivalence contract defines:

• source output,
• target output,
• grain and keys,
• columns and metrics,
• checks and check packs,
• sampling,
• tolerances,
• schema behavior,
• CDC behavior,
• evidence.

grain.keys define source-target comparison identity. CDC checks that validate change propagation should declare CDC identity separately with cdc.keys.

Example authored contract:

version: 1

name: customer_revenue

source:
  connection: legacy
  relation: qa.v_customer_revenue_compare

target:
  connection: warehouse
  relation: qa.v_customer_revenue_compare

grain:
  keys:
    - customer_id
    - month

columns:
  numeric:
    - name: revenue
      tolerance: 0.01

metrics:
  - name: revenue_by_month
    type: sum
    column: revenue
    group_by:
      - month
    tolerance: 0.01

checks:
  use:
    - recon_core.basic_equivalence

sampling:
  default_policy: full

evidence:
  level: detailed
  store_failures: true

Intended Workflow

Recon is designed around a parse, compile, run workflow:

authored project files
  -> recon parse
  -> recon compile
  -> recon run
  -> results and evidence

The intended command responsibilities are:

• recon parse validates contract files, indexes local resource source files, and writes target/manifest.json.
• recon compile expands contracts, defaults, check packs, metrics, sampling, tolerances, schema policies, and CDC settings into explicit artifacts.
• recon run executes compiled checks and returns or writes results and evidence only as the assigned runner/evidence phases implement those outputs.

Current recon parse validation is intentionally structural. It validates YAML syntax, contract file discovery, required contract fields, endpoint shape, unknown top-level contract fields, simple multi-contract files, and duplicate contract names. Current recon compile expands the supported built-in check pack, compiles explicit sum metrics, and carries limited contract-level sampling.default_policy metadata into compiled checks. Broader compile-time behavior such as authored checks, full sampling policy resolution, tolerance precedence, schema policy resolution, CDC validation, adapter checks, and row-level value check expansion is still future work.

Current recon run consumes target/compiled_checks/ plus matching target/compiled_contracts/ metadata. It does not parse authored YAML or recompile contracts. It loads runtime profiles and opens the DuckDB adapter only for supported relation-backed same-context row_count_diff checks and for grain-key safety checks only when the internal local/dev scan guard classifies the input as bounded. The guard requires a project-local DuckDB file under the size cap and compiled source/target relations that resolve to local base tables; views and externally backed relations fail closed. Unsupported execution surfaces remain blocked or not executable. It does not write generated run/evidence outputs.

Current generated artifacts:

target/manifest.json
target/compiled_contracts/
target/compiled_checks/
target/compiled_sql/       # only when recon compile --render-sql succeeds

Planned future run and evidence artifacts:

target/run_results.json
target/failures/
reports/
state/

Authored contracts are versioned. Generated artifacts are not.

Repository Map

src/recon_core/        Python package source
tests/                 Unit and CLI tests
docs/framework/        Framework concepts and public behavior
docs/architecture/     System boundaries and package layout
docs/implementation/   Implementation guidance and build order
docs/decisions/        Architecture decision records
docs/compatibility/    Compatibility surfaces and version support
examples/              Authored example Recon projects

Documentation

Start here:

• Quickstart
• Installation
• CLI guide
• Equivalence contracts
• Framework concepts
• MVP build order
• Architecture decisions
• Compatibility
• Public contract inventory

Contributing

Set up local development:

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest
ruff check .
ruff format --check .
mypy src

Before opening a pull request, read CONTRIBUTING.md and the relevant docs under docs/framework/, docs/architecture/, and docs/implementation/.

Changes to public contract syntax, artifact formats, validation defaults, adapter interfaces, or evidence behavior may need an ADR under docs/decisions/.

Security and Generated Artifacts

Do not commit credentials, connection profiles, secrets, customer data, or real production evidence.

Keep these paths local or generated:

connections/profiles.yml
.env
target/
reports/
state/
recon_packages/

Generated evidence can contain sensitive values. Use fake data in examples and keep generated outputs out of Git.