d-migrate

Database-agnostic tool for schema migration and data management.

English | Deutsch

📖 Documentation (German): Anwenderhandbuch (task-oriented user guide) · Administrationshandbuch (deployment/ops) · CLI spec · MCP server spec — full index: docs/user/README.md. See "Documentation" below for the complete list.

d-migrate is a database-agnostic tool for schema migration and data management, usable as a CLI and as an MCP server (mcp serve --transport stdio|http, MCP 2025-11-25). You define your schema once in a neutral YAML format and then validate, compare, generate DDL, and execute live diff-based migrations against PostgreSQL, MySQL, and SQLite. d-migrate also covers reverse engineering of existing databases, streaming-based data export/import/transfer between databases, and export to existing migration toolchains (Flyway, Liquibase, Django, Knex).

Who is it for?

d-migrate targets database administrators, platform engineers, data teams, and integrators who:

• need a dialect-agnostic schema artefact (PostgreSQL / MySQL / SQLite from the same YAML source)
• want reproducible, signed migration plans with explicit rollback contracts, drift checks, and per-statement metadata
• run schema and data operations against existing databases — including reverse engineering, comparison, transfer, and incremental export — without locking into a single vendor's tooling
• need an AI-agent-callable MCP server for read-only schema discovery (validate / compare / generate / reverse) plus policy-gated job workers for data import / transfer / profile

It is not (yet) an ETL platform, a streaming-CDC pipeline, or a replacement for hand-tuned dialect-specific migrations — but it captures the schema and data work that's common across these stacks.

What can I run today?

d-migrate is a working production tool at version 1.2.0 (stable, released 2026-09-05).

New in 1.2.0: data seed generates deterministic, FK-safe test data straight from the neutral schema (an optional --rules file steers values per column); mcp serve gains a configurable --policy-file for *_start job policy and a connections/list method with an optional live-status check; --server-state now persists reverse-engineered schemas and generated artifacts too, not just jobs/quota. See CHANGELOG.md for the full list.

New in 1.1.0: MS SQL Server as the fourth dialect (reverse, generate, migrate, data path, profiling), schema migrate can now run statements a database refuses inside an open transaction (SQL Server full-text indexes), and partition-set changes (add/drop a rolling partition) are applied instead of only reported. See CHANGELOG.md for the full list.

New in 1.0.3: restores the native binaries (the 1.0.1 tag could not build them) and makes Parquet import work in the native binary for the first time — it had failed in every native binary released so far; both native build legs now verify a full Parquet round trip. Also carries all of 1.0.1: the shipped artefact contains no known-vulnerable dependency version (down from one critical and 43 high findings), the PostgreSQL driver moves to 42.7.12, and single members of --split-files Parquet bundles import correctly. The CLI contract is unchanged. See CHANGELOG.md.

The current capabilities:

• Schema model: neutral YAML schema with 19 types + Spatial Geometry; validator with 35+ error codes.
• Schema operations: validate, generate, compare, reverse, migrate, rollback for PostgreSQL, MySQL, SQLite — file/file, file/db, db/db.
• Diff migrations: tables, columns, indexes, constraints incl. CHECK/EXCLUDE with live-data preflight, foreign keys, sequences, views, materialized views (PG), triggers, functions/procedures; signed migration-plan.v1 artefact via --plan-artefact.
• Renames for tables, columns, views, triggers, functions, procedures, sequences — native RENAME DDL or Drop+Create fallback per dialect; CLI shortcuts (--rename-table, --rename-column) or file overlay (--migration-overlay).
• Sequence pipeline: MySQL helper-table emulation (dmg_sequences) with live drift check; opt-in preserveCurrentValue for PG / MySQL / SQLite — probe + restore folded into a single transaction under per-dialect lock since 0.9.7 (pg_advisory_xact_lock / SELECT FOR UPDATE / BEGIN IMMEDIATE); SQLite sequence emulation via --sqlite-named-sequences helper_table.
• Spatial DDL: PostGIS, MySQL native, SpatiaLite (--spatial-profile); view-query transformation across dialects.
• Data operations: streaming data export / import / transfer (JSON / YAML / CSV / Parquet) with named connections, UPSERT, truncate, trigger handling, reseeding, incremental export (--since-column / --since); data profile for data statistics.
• Parquet & object storage (0.9.8): data export / import --format parquet for bundle (multi-table + manifest.yaml) and single-file (footer-KV) layouts with checkpoint/resume and --table-order; S3-compatible ArtifactStore (artifacts.store: s3, AWS SDK v2) for server-mode artefacts.
• Integrations: d-migrate export flyway|liquibase|django|knex.
• MCP server (mcp serve --transport stdio|http, MCP 2025-11-25): read-only tool surface (schema_validate, schema_compare, schema_generate, schema_reverse_start) plus policy-gated job workers (data_import_start, data_transfer_start, data_profile_start, procedure_transform_*, testdata_*) with idempotency and JDBC-backed state; auth via JWT-JWKS / RFC-7662 introspection / stdio token registry.
• CLI UX: i18n EN/DE with ICU4J, explicit time-zone / temporal policy, CSV / BOM encoding contract, phased DDL via --split pre-post.
• OCI image at ghcr.io/pt9912/d-migrate:<version> and :latest, mirrored to Docker Hub as pt9912/d-migrate. Each release also ships a <version>-native variant built from the GraalVM native binary — no JVM inside, starts in milliseconds.

The simplest way to try the tool is the published OCI image:

docker run --rm -v $(pwd):/work ghcr.io/pt9912/d-migrate:latest \
  schema validate --source /work/schema.yaml

See Quick start below for more concrete recipes.

What makes it trustworthy?

• ≥ 90 % line coverage per module, enforced by Kover (minBound(90) in every module's build.gradle.kts). The CI build fails if any module drops below.
• Doc-check gate: Markdown link targets in docs/, spec/, and root Markdown files (including both READMEs and CHANGELOG.md) are validated against the file system on every CI run via d-check (digest-pinned container image, configured in .d-check.yml); broken internal links and anchors break the build.
• Static-analysis gate: Detekt plus a SOLID-suppression-gate (scripts/solid-suppression-gate.sh) — @Suppress("LargeClass") and friends are tracked in a ledger and require structural fixes, not inline waivers.
• Cross-dialect test matrix: test/cross-dialect-matrix sweeps every workstream × dialect × test-kind cell, with a carve-out registry (carve-outs.yaml) that requires every non-pinned cell to declare reason + plan-doc reference; silent carve-outs are rejected at load time.
• Live-DB integration tests against Testcontainers PostgreSQL 16, MySQL 8, and file-backed SQLite — every diff, rename, sequence, and atomic-preserve pipeline runs against real engines via scripts/test-integration-docker.sh.
• Reproducible builds: --deterministic plus SOURCE_DATE_EPOCH emit byte-identical DDL across timestamps and OS environments.
• Signed migration plans: --plan-artefact writes a canonical, signed migration-plan.v1 JSON with stable fingerprints, statement IDs, and rollback metadata; tampered artefacts are rejected by the MigrationPlanArtifactValidator.
• Hexagonal architecture: pure-domain hexagon:core plus hexagon:ports-{common,read,write,execute} with driving adapters (CLI, MCP) and driven adapters (drivers, formats, persistence) isolated through explicit interfaces. Architectural decisions live as ADRs in docs/adr/.
• CI mirrors local: every gate make ci runs locally also runs in GitHub Actions on every push (.github/workflows/build.yml).

Status

The full release history lives in CHANGELOG.md.

• Current stable · 1.2.0 (2026-09-05) — what :latest, Homebrew and an unpinned docker pull give you. data seed generates deterministic test data, mcp serve gains a configurable --policy-file and a connections/list method, and --server-state now persists reverse-engineered schemas and artefacts too. MS SQL Server is the fourth dialect (reverse, generate, migrate, data path, profiling); the container image runs as non-root (uid 10001), so writing into a bind mount needs --user "$(id -u):$(id -g)". Native binaries ship for linux-x64 and windows-x64; on macOS use Homebrew, the JVM artefacts or the container image.

For per-milestone task tables and ADR pointers see the canonical roadmap at docs/planning/in-progress/roadmap.md. ADRs live under docs/adr/; the canonical index is docs/adr/README.md.

All releases and details: CHANGELOG.md | GitHub Releases.

Build, Test, Lint

Individual gates for fast feedback loops:

make help              # list all available targets
make ci                # Docker CI build + docs-check (full local gate)
make gates             # Docker check, coverage, docs, and semgrep gates
make docker-build      # build the runtime image
make docker-check      # Gradle check inside the Dockerfile build stage
make docker-test       # Gradle test inside the Dockerfile build stage
make docker-detekt     # Detekt static analysis
make docker-coverage-gate  # Kover ≥ 90 % per module
make docs-check        # validate Markdown link targets + Kover-excludes ledger
make semgrep           # hermetic semgrep scan with pinned rules
make integration       # Testcontainers integration suite
make docker-full-gates # docker-gates plus Docker-backed integration tests
make docker-oci-build  # build the publishable OCI image (runtime stage)
make release-assets    # build ZIP, TAR, fat JAR, SHA256 release assets

Targeted module runs:

make docker-check MODULES=":hexagon:core :adapters:driven:driver-postgresql"
make docker-test  MODULES=":adapters:driving:mcp"

Quick start

Prerequisites

• Docker
• Optional for local development without containers: JDK 21 or newer

Use the published OCI image

No local JDK required — pull the image and run it:

# Validation
docker run --rm -v $(pwd):/work ghcr.io/pt9912/d-migrate:latest \
  schema validate --source /work/schema.yaml

# Compare (file/file)
docker run --rm -v $(pwd):/work ghcr.io/pt9912/d-migrate:latest \
  schema compare --source file:/work/schema.yaml --target file:/work/schema-new.yaml

# Generate DDL
docker run --rm -v $(pwd):/work ghcr.io/pt9912/d-migrate:latest \
  schema generate --source /work/schema.yaml --target postgresql

# Reverse engineering
docker run --rm -v $(pwd):/work ghcr.io/pt9912/d-migrate:latest \
  --config /work/.d-migrate.yaml schema reverse --source mydb --output /work/reverse.yaml

# DB-to-DB data transfer
docker run --rm -v $(pwd):/work ghcr.io/pt9912/d-migrate:latest \
  data transfer --source sourcedb --target targetdb --tables users,orders

Docker / Volumes — running as non-root

The published image runs as a non-root user (uid 10001). Read-only commands (validate, compare) work as shown above. Commands that write into a bind-mounted host directory (reverse --output, generate to a file, data transfer to file targets) need the mount to be writable by the container user — add --user "$(id -u):$(id -g)" so output lands with your host ownership:

docker run --rm --user "$(id -u):$(id -g)" -v $(pwd):/work \
  ghcr.io/pt9912/d-migrate:latest \
  schema reverse --source mydb --output /work/reverse.yaml

GitHub Release assets

Published releases ship ZIP, TAR, a fat JAR, and — from 1.0.0-RC2 on — native binaries that need no Java, on the Releases page.

# Launcher-based distribution
tar -xf d-migrate-<version>.tar
./d-migrate-<version>/bin/d-migrate --help

# Or run the fat JAR directly
java -jar d-migrate-<version>-all.jar --help

# Or the native binary — no JVM required, starts in ~15 ms
chmod +x d-migrate-<version>-linux-x64
./d-migrate-<version>-linux-x64 --help

Native binaries are published for linux-x64 and windows-x64 (each with a .sha256); linux-x64 is guaranteed per release, windows-x64 is best-effort. There is no native macOS binary — on macOS use Homebrew, the JVM artefacts or the container image (ADR 0044). They are dynamically linked against glibc — on Alpine/musl use the JVM artefacts or the container image.

Homebrew (macOS and Linux)

d-migrate lives in its own tap, not in homebrew-core, so brew install d-migrate alone will not find it. Recent Homebrew versions additionally refuse to load formulae from untrusted third-party taps — you need all three steps:

brew tap pt9912/d-migrate
brew trust pt9912/d-migrate
brew install d-migrate

openjdk@21 comes along as a dependency. The tap follows stable releases only — release candidates never move it.

On macOS this is the recommended path, because there is no native macOS binary (ADR 0044).

The formula is maintained in this repository from 0.5.0 on and verified per release via .github/workflows/verify-homebrew-formula.yml.

Build from source

make ci-build

Minimal schema example

Create a file called schema.yaml:

schema_format: "1.0"
name: "My App"
version: "1.0.0"

tables:
  users:
    columns:
      id:
        type: identifier
        auto_increment: true
      email:
        type: text
        max_length: 254
        required: true
        unique: true
      created_at:
        type: datetime
        default: current_timestamp
    primary_key: [id]

Validate it like this:

make docker-build
docker run --rm -v $(pwd):/work d-migrate:dev schema validate --source /work/schema.yaml

And compare two versions like this:

docker run --rm -v $(pwd):/work d-migrate:dev \
  schema compare --source /work/schema.yaml --target /work/schema-v2.yaml

Build and test locally with the Dockerfile

The repository ships a multi-stage Dockerfile that builds and tests the project inside the container and then packages the CLI distribution into a slim JRE runtime image. This is the simplest way to run the full build without installing a local JDK.

Dockerfile stages — overview

• deps: Gradle dependency pre-warm.
• build: build, tests, coverage gate, distribution. Use --target build --build-arg GRADLE_TASKS="..." to scope to a specific task list.
• detekt: Detekt static analysis.
• coverage: aggregated Kover HTML report on port 8080. docker build --target coverage -t d-migrate:coverage . + docker run --rm -p 8080:8080 d-migrate:coverage.
• coverage-json: Kover JSON to stdout via ENTRYPOINT.
• coverage-verify: hard koverVerify (≥ 90 % per module).
• release-assets: ZIP / TAR / fat JAR / SHA256 (target of make release-assets).
• runtime (default): the image that gets published — slim eclipse-temurin:21-jre-noble, non-root (uid 10001), mod_spatialite included. Target of make docker-oci-build (ADR 0041).

Common Dockerfile recipes

# Full build incl. tests and coverage validation
docker build -t d-migrate:dev .

# Force a full test/coverage run (bypasses both the Docker layer cache and the Gradle cache)
docker build --no-cache --progress=plain \
  --build-arg GRADLE_TASKS="build :adapters:driving:cli:installDist --rerun-tasks" \
  -t d-migrate:dev .

# Skip tests — build only the CLI distribution
docker build --build-arg GRADLE_TASKS="assemble :adapters:driving:cli:installDist" \
  -t d-migrate:dev .

# Run only part of the build stage without producing the final runtime image
docker build --target build \
  --build-arg GRADLE_TASKS=":hexagon:core:test :adapters:driven:driver-common:test" \
  -t d-migrate:phase-a .

# Run the locally built CLI
docker run --rm -v $(pwd):/work d-migrate:dev schema validate --source /work/schema.yaml

# Run the testcontainers integration suite
./scripts/test-integration-docker.sh

# Or a subset of integration tests
./scripts/test-integration-docker.sh :adapters:driven:driver-postgresql:test

Supported databases

Database	Status
PostgreSQL	DDL generation, reverse engineering, data export/import/transfer
MySQL	DDL generation, reverse engineering, data export/import/transfer
SQLite	DDL generation, reverse engineering, data export/import/transfer
Oracle	Reverse engineering, DDL generation, schema migration, tool export, data export/import/transfer (rollout, ADR 0052)
MSSQL	Reverse engineering, DDL generation, schema migration, tool export, data export/import/transfer (rollout, ADR 0047)

Project Structure

.
├── .github/workflows/             ← GitHub Actions: build, integration, demo/sample DB, release
├── CHANGELOG.md
├── Dockerfile                     ← multi-stage (deps, build, detekt, coverage, runtime, release-assets)
├── Makefile                       ← build/test gates per Dockerfile stage
├── README.md                      ← English main version (this document)
├── README.de.md                   ← German version
├── build.gradle.kts               ← root build config + module aggregation
├── settings.gradle.kts            ← Gradle multi-module declaration
├── gradle.properties              ← pinned dependency versions
├── config/                        ← detekt and semgrep configuration
├── hexagon/                       ← pure domain + ports (no driver dependencies)
│   ├── core/                      ← neutral schema model, diff core, validators
│   ├── ports-common/              ← cross-cutting port contracts
│   ├── ports-read/                ← read-side ports (DDL generation, reverse, capabilities)
│   ├── ports-write/               ← write-side ports (data import / transfer)
│   ├── ports-execute/             ← atomic-execution ports (preserve, lock contracts)
│   ├── ports/                     ← driver-registry port (DatabaseDriver, DatabaseDriverRegistry, PreGenerationValidator) + facade re-export of ports-{common,read,write,execute}
│   ├── application/               ← use-case orchestration + stage pipelines
│   └── profiling/                 ← perf measurement infrastructure
├── adapters/
│   ├── driven/                    ← outbound: driver-postgresql/-mysql/-sqlite (+ -profiling),
│   │                                formats + formats-parquet, integrations
│   │                                (Flyway/Liquibase/Django/Knex), persistence-jdbc,
│   │                                storage-file/-s3, streaming, text-icu,
│   │                                audit-logging, connection-config
│   └── driving/                   ← inbound: cli, mcp
├── examples/
│   ├── bi-demo/                   ← Compose demo for Parquet/S3/BI flows
│   └── sample-db/                 ← on-demand sample database harness
├── test/
│   ├── consumer-read-probe/       ← read-only consumer surface verification
│   ├── cross-dialect-matrix/      ← workstream × dialect × kind sweep + carve-out registry
│   ├── integration-postgresql/    ← Testcontainers PG live-DB tests
│   ├── integration-mysql/         ← Testcontainers MySQL live-DB tests
│   ├── integration-sqlite/        ← file-backed SQLite live-DB tests
│   ├── integration-concurrency/   ← race-condition reproducers (sequence preserve, atomic locks)
│   ├── integration-integrations/  ← export integration contract tests
│   ├── integration-persistence-jdbc/ ← JDBC store + migration runner ITs
│   ├── integration-server-state/  ← MCP server state machine ITs
│   ├── integration-storage-s3/    ← S3-compatible artifact store ITs
│   ├── e2e-cli/                   ← end-to-end CLI + MCP harness scenarios
│   └── perf-large-schema/         ← N = 100 / 1000 / 10000 perf scales
├── scripts/                       ← verify-doc-refs.sh, solid-suppression-gate.sh,
│                                    test-integration-docker.sh, kover utilities
├── ledger/                        ← suppression and quality ledgers
├── packaging/homebrew/            ← Homebrew formula (d-migrate.rb)
├── spec/                          ← normative specs (German): lastenheft, architecture,
│                                    design, cli-spec, neutral-model-spec,
│                                    ddl-generation-rules, mcp-server, schema-reference,
│                                    connection-config-spec
└── docs/
    ├── adr/                       ← Architecture Decision Records + index
    ├── planning/
    │   ├── open/                  ← trigger watch + open follow-ups
    │   ├── next/                  ← planned but not yet active
    │   ├── in-progress/           ← active roadmap + slice plans
    │   └── done/                  ← completed slices + closure notes
    └── user/                      ← user / operator facing (guide.md, releasing.md)

Note: the linked ADRs, slice plans, and planning documents under docs/ and spec/ are written in German. The English README mirrors the structure and key facts; for deep-dive content, consult README.de.md or the German source documents.

Documentation

Detailed documentation lives in docs/ and spec/:

• Documentation overview (German)
  • Anwenderhandbuch
  • Administrationshandbuch
  • Migrations-Leitfaden
  • API-Referenz (CLI + MCP)
  • Best-Practices-Leitfaden
  • Troubleshooting-Leitfaden
• Quick Start Guide (German)
• Architecture
• Schema YAML reference
• Neutral model specification
• CLI specification
• MCP server
• DDL generation rules
• Connection and configuration specification
• Roadmap
• Release guide
• Requirements document (German)

Contributing

Contributions are welcome! Please open an issue or a pull request on GitHub.

1. Fork the repository
2. Create a feature branch off main
3. Write tests for your changes (≥ 90 % per-module Kover gate applies)
4. Make sure the Docker CI gates pass (make ci)
5. Submit a pull request against main

License

This project is licensed under the MIT License.