Merge pull request #34 from ncj-dneg/ncj-dneg-establish-coding-standards

Separate engine and interface packages

Author: drunkenbot-ai

.github/workflows/quality.yml

@@ -0,0 +1,24 @@
+name: Code quality
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  quality:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install quality tools
+        run: python -m pip install -r requirements-dev.txt
+      - name: Check formatting and lint
+        run: ruff check .
+      - name: Check type hints
+        run: mypy .
+      - name: Check Python file sizes
+        run: python tools/check_code_standards.py
+      - name: Check package dependency boundaries
+        run: python tools/check_dependency_boundaries.py

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "engine"]
+	path = engine
+	url = https://github.com/drunkenbot-ai/engine.git

README.md

@@ -16,6 +16,12 @@ creating tokenizers, training small GPT-style language models, benchmarking
 checkpoints, exporting model artifacts, and testing local GGUF models in a
 streamed Markdown chat interface.
 
+The source is split into the non-Qt `engine/` package and the Qt desktop
+`interface/` package. Historical `llm_trainer/` import paths remain available
+for compatibility; new application and test code should use the canonical
+packages. The dependency direction is one-way: `engine/` never imports
+`interface/`, and `interface/` never imports `llm_trainer/`.
+
 
 For development, use Python 3.12 or newer. Distribution builds include a
 private Python runtime and do not require Python on the user's machine.
@@ -74,22 +80,29 @@ license data, logs, and machine identifier under the user's
 
 ## First backend commands
 
-Prepare text/PDF/JSONL files:
+The non-Qt command-line engine is available directly:
+
+```powershell
+python -m engine.cli --help
+```
+
+`python -m llm_trainer.cli` remains a compatibility alias for existing
+automation. Prepare text/PDF/JSONL files:
 
 ```powershell
-python -m llm_trainer.cli prepare --input_dir .\examples\tiny_corpus --output_dir .\runs\tiny_data --context_length 16
+python -m engine.cli prepare --input_dir .\examples\tiny_corpus --output_dir .\runs\tiny_data --context_length 16
 ```
 
 Prepare programming PDFs plus source files in code-aware mode:
 
 ```powershell
-python -m llm_trainer.cli prepare --input_dir .\examples\tiny_corpus --output_dir .\runs\code_data --context_length 128 --code_training_mode
+python -m engine.cli prepare --input_dir .\examples\tiny_corpus --output_dir .\runs\code_data --context_length 128 --code_training_mode
 ```
 
 For large corpora, enable faster preview/build scanning:
 
 ```powershell
-python -m llm_trainer.cli prepare --input_dir .\my_big_data --output_dir .\runs\big_data --fast_scan_mode
+python -m engine.cli prepare --input_dir .\my_big_data --output_dir .\runs\big_data --fast_scan_mode
 ```
 
 `--fast_scan_mode` uses cheaper file fingerprints and cached preview statistics,
@@ -98,7 +111,7 @@ which is significantly faster on very large datasets.
 To reduce false duplicate matches in fast mode, add strict verification:
 
 ```powershell
-python -m llm_trainer.cli prepare --input_dir .\my_big_data --output_dir .\runs\big_data --fast_scan_mode --strict_duplicate_verification --fast_scan_sample_bytes 65536
+python -m engine.cli prepare --input_dir .\my_big_data --output_dir .\runs\big_data --fast_scan_mode --strict_duplicate_verification --fast_scan_sample_bytes 65536
 ```
 
 `--strict_duplicate_verification` only runs full SHA-256 hashing on suspected
@@ -111,7 +124,7 @@ tries to extract code-like blocks from PDFs/text.
 Train a very small smoke-test model:
 
 ```powershell
-python -m llm_trainer.cli train --data_dir .\runs\tiny_data --output_dir .\runs\tiny_model --epochs 1 --batch_size 2 --context_length 16 --embedding_size 32 --head_count 4 --layer_count 2 --device cpu --no_resume
+python -m engine.cli train --data_dir .\runs\tiny_data --output_dir .\runs\tiny_model --epochs 1 --batch_size 2 --context_length 16 --embedding_size 32 --head_count 4 --layer_count 2 --device cpu --no_resume
 ```
 
 Training saves checkpoints in the model folder and can resume from the latest
@@ -163,13 +176,19 @@ contains a real Hugging Face-compatible `hf_model` directory.
 For MicroGPT checkpoints, use the HF-style package export first:
 
 ```bash
-python -m llm_trainer.cli export-hf --model_dir runs/model
+python -m engine.cli export-hf --model_dir runs/model
 ```
 
 That creates `runs/model/hf_model` with config, weights, tokenizer metadata,
 lineage, and a README. It is portable MicroGPT packaging, not a claim that the
 checkpoint is already a llama.cpp-supported Llama/Mistral/Gemma model.
 
+Before submitting changes, run the package-boundary check:
+
+```powershell
+python tools/check_dependency_boundaries.py
+```
+
 ## Current IDE Features
 
 - Dataset Blueprint with dynamic bundled corpus discovery.

UserGuide.md

@@ -37,9 +37,14 @@ their own private Python runtime and do not require Python to be installed
 system-wide.
 
 For installer builds, see [build.md](build.md). The installer excludes the
-bundled `llm_trainer/default_data` training corpus; projects receive only the
+bundled `engine/default_data` training corpus; projects receive only the
 training data that you explicitly copy or add.
 
+The application code is organized into `engine/` (non-Qt data, training, and
+worker services) and `interface/` (the Qt desktop UI). Existing
+`llm_trainer.*` imports and `python -m llm_trainer.cli` remain compatibility
+aliases, but new code should import from the canonical packages.
+
 The app has five main work areas:
 
 - `IN`: prepare datasets.

build.md

@@ -39,8 +39,10 @@ python packager.py --gpu
 ```
 
 The packager creates a private runtime, installs the pinned dependencies,
-builds the launcher bundle, copies application assets and fonts, runs Inno
-Setup, and writes the installer to `packaging/artifacts/`.
+builds the launcher bundle, copies `engine/` and `interface/` directly along
+with application assets and fonts, runs Inno Setup, and writes the installer to
+`packaging/artifacts/`. The historical `llm_trainer/` compatibility package is
+kept in the source tree but is not the primary packaged application path.
 
 The CUDA build detects the NVIDIA driver using `nvidia-smi`. It selects the
 supported PyTorch wheel index and falls back to CPU when no compatible driver
@@ -62,6 +64,10 @@ The installer includes the private Python runtime, application code, third
 party packages, fonts, logo images, and other application assets. Training corpus data is not bundled; users download or select it through the
 Dataset Sources page after installation.
 
+Run `python tools/check_dependency_boundaries.py` before packaging to verify
+that the non-Qt engine does not import the desktop interface and that new
+interface code does not depend on the legacy package.
+
 Build intermediates and installers are written under `packaging/` and are
 ignored by Git.
 

engine

@@ -0,0 +1,2 @@
+"""PySide6 desktop interface for Micro Trainer."""
+