Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
124 lines (96 loc) · 4.26 KB

ARCHITECTURE.md

File metadata and controls

124 lines (96 loc) · 4.26 KB
Raw

Architecture

SomNLP-Translate is an independent Cargo workspace for acquiring, validating, and releasing English–Somali parallel data. It mirrors SomNLP-Corpus conventions without a code dependency between repositories.

Workspace

SomNLP-Translate/
├── Cargo.toml
├── configs/
│   └── pipeline.toml          # introduced with executable pipeline stages
├── crates/
│   ├── common/                # implemented in M1
│   ├── translate-tools/       # introduced in M2
│   └── translate-pipeline/    # introduced in M3
├── docs/
├── reports/                   # generated; not versioned
└── data/                      # generated; not versioned

common

Owns stable contracts with no network or pipeline dependencies:

  • RawParallelRecord and ParallelRecord;
  • language-tagged text, provenance, license, hashes, quality, and split types;
  • schema version and record invariant validation.

Hashing and normalization implementations join this crate in M2/M3 after their exact semantics are tested. The schema is deliberately independent of any one source.

translate-tools (M2)

Owns source access and raw-data interoperability:

  • Hugging Face/parquet and direct-download clients;
  • OPUS and MT560 source adapters;
  • source registry and licensing metadata;
  • deterministic merge and exact-pair duplicate handling.

Source adapters must preserve both sides and upstream identifiers. They may repair source-specific serialization defects, but may not perform general quality filtering.

translate-pipeline (M3)

Owns source-independent transformations:

  • pair-safe normalization;
  • language verification for both sides;
  • alignment and quality checks;
  • exact-side and near-duplicate clustering;
  • leakage-resistant split assignment;
  • stage reports, reject sidecars, and run_pipeline.

Data flow

flowchart LR
  Sources[ParallelSources] --> Download[SourceAdapters]
  Download --> Raw[data/raw]
  Raw --> Merge[MergeAndExactPairDedup]
  Merge --> Merged[data/merged]
  Merged --> Clean[NormalizeBothSides]
  Clean --> Lid[DualLanguageVerification]
  Lid --> Filter[AlignmentQualityFilter]
  Filter --> Dedup[NearDedupAndLeakageGroups]
  Dedup --> Split[DeterministicSplit]
  Split --> Final[data/final]
  Merge --> Reports[ReportsAndRejects]
  Clean --> Reports
  Lid --> Reports
  Filter --> Reports
  Dedup --> Reports
  Split --> Reports
Loading

All transformations treat a pair atomically. A stage cannot keep one side after rejecting the other.

Canonical versus directional data

Canonical records preserve upstream source and target positions and carry explicit language tags. An exporter selects the requested input and output by language:

canonical pair (en, so) ─┬─ export en→so
                         └─ export so→en

The exporter swaps sides when needed; the pipeline does not duplicate canonical rows.

Artifact boundaries

Stage Planned artifact
Per-source download data/raw/<source>/<source>_en-so.jsonl
Deterministic merge data/merged/merged_en-so.jsonl
Clean data/cleaned/cleaned_en-so.jsonl
Dual LID data/lid/lid_en-so.jsonl
Quality filter data/filtered/filtered_en-so.jsonl
Deduplicate/group data/dedup/dedup_en-so.jsonl
Split data/final/{train,dev,test}_en-so.jsonl
Directional export data/release/{en-so,so-en}/

Each processing artifact uses ParallelRecord. Raw adapters may initially emit RawParallelRecord. Stage statistics live in reports/; rejects sit beside the stage artifact with a .rejected.jsonl suffix.

Configuration

configs/pipeline.toml becomes the single source of truth when M2/M3 introduce executable knobs. Every report records the config version and effective values. Thresholds are not frozen in M1 because they require labeled benchmarks.

Operational requirements

  • Stream records and bound memory except where a documented index is required.
  • Write outputs atomically through temporary files.
  • Use deterministic source order, hashing, clustering, and split assignment.
  • Do not log dataset text by default.
  • Preserve source license and provenance through every stage.
  • Treat schema changes as versioned, potentially breaking changes.