CLI tool for idempotent data migrations with validation, deduplication, and dry-run mode. Built from production patterns that handled zero-downtime migrations of 600+ records across multiple data sources.
Data migrations in production are risky. One malformed row can corrupt an entire collection. Manual CSV-to-database imports lack validation, duplicate detection, and rollback capability.
This framework solves it with:
- Schema validation (Zod) that catches ALL errors before writing — not just the first one
- Dry-run mode to preview exactly what would be written
- Deduplication by any field (last occurrence wins)
- Idempotent MongoDB upserts — safe to re-run without duplicating data
- Batch processing with progress logging
flowchart LR
Source[CSV / XLSX / JSON] -->|Parse| Parser[Source Parser]
Parser -->|Normalize headers| Transform[Transformer]
Transform -->|Chain transforms| Validate[Zod Validator]
Validate -->|Collect ALL errors| Dedup[Deduplicator]
Dedup -->|Last wins| Writer[Output Writer]
Writer --> MongoDB[(MongoDB)]
Writer --> JSON[JSON File]
Writer --> CSV[CSV File]
subgraph Report
Validate -->|Invalid rows| Errors[Error Report]
Dedup -->|Duplicates count| Summary[Migration Summary]
end
# Install
npm install
# Dry-run with sample data (no data written)
npm run migrate -- -s samples/users.csv -f csv -t json --dry-run
# Run migration to JSON output
npm run migrate -- -s samples/users.csv -f csv -t json -o output/users.json
# Run with deduplication
npm run migrate -- -s samples/users.csv -f csv -t json --dedup email -o output/users.json
# XLSX to MongoDB
npm run migrate -- -s data/clients.xlsx -f xlsx -t mongodb \
--target-uri mongodb://localhost:27017/mydb \
--target-collection users \
--batch-size 50 \
--dedup id_numberUsage: etl-migrate run [options]
Options:
-s, --source <path> Source file path (CSV, XLSX, or JSON)
-f, --format <type> Source format: csv | xlsx | json
-t, --target <type> Target: json | csv | mongodb (default: json)
--sheet <name> Sheet name for XLSX files
--target-uri <uri> MongoDB connection URI
--target-collection <name> MongoDB collection name
-o, --output <path> Output file path
--batch-size <n> Batch size for MongoDB writes (default: 100)
--dedup <field> Deduplicate by field name
--limit <n> Limit records to process
--dry-run Preview without writing data
| Step | Description |
|---|---|
| Parse | Reads CSV/XLSX/JSON with auto header detection and normalization |
| Transform | Trims strings, adds _migratedAt metadata |
| Validate | Zod schema validation — collects ALL errors, not fail-fast |
| Deduplicate | Removes duplicates by key field (keeps last occurrence) |
| Write | Outputs to JSON, CSV, or MongoDB (batch upsert) |
[1/5] Parsing samples/users.csv (csv)...
Found 6 records
[2/5] Transforming...
Transformed 6 records
[3/5] Validating...
Valid: 5 | Invalid: 1
[4/5] Deduplicating by "email"...
Removed 1 duplicates (4 unique)
[5/5] Writing to json...
--- Migration Report ---
Read: 6
Valid: 5
Invalid: 1
Duplicates: 1
Written: 4
Duration: 12ms
Errors (first 5):
Row 3: name: Name too short, email: Invalid email
------------------------
| Technology | Purpose |
|---|---|
| Node.js 20 | Runtime |
| TypeScript | Type safety |
| Commander.js | CLI framework |
| Zod | Schema validation + transformation |
| xlsx | Excel file parsing |
| csv-parse/csv-stringify | CSV read/write |
| Mongoose | MongoDB batch upserts |
src/
├── cli.ts # CLI entry point (commander.js)
├── pipeline.ts # Orchestrates the full ETL flow
├── types.ts # Shared TypeScript interfaces
├── validator.ts # Zod validation + built-in schemas
├── transformer.ts # Transform chain + deduplication
├── parsers/
│ ├── index.ts # Parser router (csv/xlsx/json)
│ ├── csv-parser.ts # Stream-based CSV parsing
│ ├── xlsx-parser.ts # Excel with auto header detection
│ └── json-parser.ts # JSON array or wrapped format
└── writers/
├── index.ts # Writer router
├── mongo-writer.ts # Batch upsert with progress
└── file-writer.ts # JSON/CSV file output
MIT