MASTERd — Machine-Assisted Sorting, Tagging, and Extraction of Records and Documents

File disarray is a solvable defect. Organize or be organized.

MASTERd is a fully Rust-first document intelligence platform with a Tauri desktop UI. It ingests, classifies, deduplicates, semantically indexes, and retrieves documents using a multi-stage pipeline backed by embedded LFM2.5 GGUF models — with Ollama as a transparent fallback. Zero cloud dependencies. AMD ROCm–first. Runs entirely on your machine.

---

Features

• Tauri desktop app — Next.js frontend wired to a native Rust backend via typed Tauri commands
• Multi-stage ingestion pipeline — hash → hot-cache → dedup → canonical SQLite write → LanceDB vector snapshot → Meilisearch lexical index → ColBERT rerank queue → Falkor graph mirror
• Embedded GGUF inference — LFM2.5-1.2B-Thinking and LFM2.5-350M-Instruct bundled as include_bytes! assets, loaded via llama.cpp; Ollama fallback when models are unavailable
• ColBERT MaxSim reranker — L2-normalized token-matrix reranking (correct cosine similarity, not raw dot product)
• Embedding services — FastAPI ColBERT/Jina v3 servers plus a Rust model2vec-service wrapper; setup script enforces AMD ROCm PyTorch index
• Rust static embeddings — vendored model2vec-rs provides a fast local embedding path and structural fallback
• Supervised sidecar processes — Meilisearch and Valkey managed by SidecarSupervisor, with optional FalkorDB graph module
• AMD ROCm–first — all Python installs routed through ROCm PyTorch index; CUDA wheels blocked by config/rocm-constraints.txt
• Boot MIDI player — ambient music on app launch, pure Rust

Architecture

apps/
  masterd-desktop-tauri/   ← Tauri 2.x native host
  masterd-shell/           ← Next.js 14 UI
  masterd-ingest/          ← CLI document ingestion tool
  masterd-bootstrap/       ← Sidecar validation & first-launch setup
  masterd-engine-check/    ← Inference + retrieval integration tests
  masterd-tune/            ← AMD kernel auto-tuner
  masterd-midi-player/     ← Boot music player

crates/
  masterd-chat-engine/     ← Embedded GGUF chat + Ollama fallback
  masterd-embed-engine/    ← Local embedding stack (ColBERT / Jina / model2vec-rs)
  masterd-index/           ← ColBERT reranker, atomic hash-index dedup
  masterd-pipeline/        ← Typed stage-graph, retrieval pipeline, naming rules
  masterd-core/            ← Shared types, cancellation contract
  masterd-prompt-core/     ← MASTERd personality + avatar prompt registry
  masterd-runtime-tune/    ← Startup-safe AMD kernel profiler
  masterd-sidecars/        ← Sidecar topology validation
  masterd-ui-contract/     ← Typed Tauri event contract

models/
  lfm2.5-1.2b-thinking/   ← LFM2.5-1.2B-Thinking Q8_0 GGUF + tokenizer
  lfm2.5-350m-instruct/   ← LFM2.5-350M-Instruct Q8_0 GGUF + tokenizer
  lfm2-colbert-350m/      ← LFM2-ColBERT-350M Q8_0 GGUF (reranker)
  jina-v5-omni-nano-gguf/ ← Jina v5 Omni Nano retrieval/text-matching Q4_K_M GGUF
  jina-v5-omni-small-gguf/← Jina v5 Omni Small retrieval/text-matching Q4_K_M GGUF
  masterd-identity/       ← MASTERd system personality prompt

services/
  colbert-service/         ← FastAPI ColBERT HTTP server (port 11450)
  jina-service/            ← FastAPI Jina v3 HTTP server (port 11447)
  model2vec-service/       ← Rust HTTP wrapper around model2vec-rs (port 11448)

vendor/
  candle/                  ← Hugging Face Candle ML framework (vendored)
  tauri/                   ← Tauri framework source (vendored)
  tokenizers/              ← HF Tokenizers (vendored)

Requirements

Build tools

• Rust ≥ 1.85 (nightly, see rust-toolchain.toml)
• Node.js >= 20 + pnpm
• Tauri CLI — cargo install tauri-cli
• curl or Python huggingface-hub — used by scripts/download-models.sh

Runtime sidecars (downloaded by build script)

• Meilisearch v1.8.3 — lexical search engine
• Valkey v7.2.5 — hot-cache and dedup store
• FalkorDB (optional) — graph relationship queries

Python embedding services (AMD ROCm)

• Python 3.12
• uv — curl -LsSf https://astral.sh/uv/install.sh | sh
• AMD ROCm 6.x or 7.x runtime

Linux desktop libraries

The launcher now bootstraps missing native GUI/build dependencies on Linux automatically the first time you run it. That includes the Tauri/WebKitGTK stack, pkg-config, patchelf, and the usual GTK/WebKit support libraries.

Quick start

# 1. Clone the repo
git clone https://github.com/carlosfundora/masterd-rs
cd masterd-rs

# 2. Provide a Hugging Face token for gated Liquid AI model repos
#    (GGUF files are not stored in git — too large)
export HF_TOKEN=hf_your_token_here  # required for gated Liquid AI repos

# Optional: pre-download model weights, tokenizers, and chat templates now
./scripts/download-models.sh

# Optional: verify existing local model files without downloading
./scripts/download-models.sh --verify-only

# 3. Bootstrap: validates sidecar config and creates first-launch directories
cargo run -p masterd-bootstrap

# 4. Run the desktop app
#    This starts the Tauri app, launches Meilisearch/Valkey/FalkorDB + the
#    ColBERT/Jina embedding services, the model2vec-rs local fallback, and preloads both LFM2.5 models.
pnpm dev

# — OR — run the desktop app directly from the Tauri crate:
cd apps/masterd-desktop-tauri
cargo tauri dev

Ingest documents

cargo run -p masterd-ingest -- --root /path/to/your/documents

Pipeline stages (configurable in config/pipeline.toml):

1. Rapid SHA-256 hash
2. Valkey hot-cache write (offline fallback if Valkey unavailable)
3. Rigorous dedup gate
4. Canonical SQLite write (data/masterd.db)
5. LanceDB vector snapshot queue
6. ColBERT rerank queue
7. Meilisearch lexical queue
8. Jina omni multimodal queue (optional)
9. Falkor graph mirror queue

Embedding stack

The live embedding stack uses Jina as the primary service path and vendored model2vec-rs as a fast local parallel source. The main installer sets up the Jina service by default, and all Python installs are routed through the AMD ROCm PyTorch index — no CUDA wheels are permitted.

Note

Whenever the desktop app is launched, it automatically starts the embedding services (ColBERT, Jina, model2vec-service) as supervised processes, keeps the local model2vec-rs path available, and preloads the embedded LFM2.5 thinking and instruct models. This happens from both pnpm dev at the repo root and cargo tauri dev inside apps/masterd-desktop-tauri.

You can also start the services manually for CLI tools or development:

# Set up the embedding service venvs (Python 3.12 + ROCm torch)
./scripts/setup-embedding-services.sh all

# Skip embedding-service setup during installer builds only when needed:
MASTERD_SKIP_EMBEDDING_SERVICES=1 ./scripts/build-installer-bundles.sh

# Start a service manually
services/colbert-service/.venv/bin/python services/colbert-service/server.py
services/jina-service/.venv/bin/python    services/jina-service/server.py
bash services/model2vec-service/start.sh

Service endpoints (when running):

Service	Port	Role
ColBERT	11450	Token-matrix reranking
Jina v3	11447	Dense code/text embeddings
model2vec-service	11448	Fast static embeddings / structural fallback

Switch backend in config/embedding_engine.toml or env vars:

export MASTERD_INFERENCE_BACKEND=http   # use HTTP service endpoints
export MASTERD_INFERENCE_BACKEND=direct # self-contained Rust (default)

Engine validation

cargo run -p masterd-engine-check -- --chat-url http://127.0.0.1:3000
# Report written to: data/engine_validation.json

AMD kernel auto-tuner

cargo run -p masterd-tune -- --auto     # startup-safe tune
cargo run -p masterd-tune -- --retune   # full retune

AMD profiles live in config/amd_profiles/. Kernel manifest at config/kernel_manifest.toml.

Ollama fallback

MASTERd automatically falls back to Ollama when embedded models fail to load:

1. Tries to load embedded GGUF model from assets/models/
2. On any failure, calls http://127.0.0.1:11434 (configurable in Settings)
3. Uses resolve_model() — picks the configured model name or the first available Ollama model
4. Same ChatToken streaming interface; model badge shows ollama/<model>

Configure in the desktop app → Settings → Ollama Fallback Engine.

Boot music

cargo run -q -p masterd-midi-player -- --seconds 8
# Disable: export MASTERD_NO_MUSIC=1

Configuration

File	Purpose
config/pipeline.toml	Stage order, vector authority, cache engine
config/embedding_engine.toml	Model URLs, batch size, backend mode
config/sidecars.toml	Sidecar process topology
config/kernel_manifest.toml	AMD kernel pack registry
config/rocm-constraints.txt	Blocks CUDA wheels in all Python installs
uv.toml	ROCm PyTorch index configuration for uv

License and author

MASTERd is licensed under the MIT License.

Author: Carlos Fundora sentseven@gmail.com

Third-party credits and attribution notes are in THIRD_PARTY_NOTICES.md.

What was set up

• Rust workspace with foundation crates:
  • crates/masterd-core (shared capability model)
  • crates/masterd-prompt-core (MASTERd personality + avatar prompt registry)
  • crates/masterd-pipeline (hash→cache→dedup→index pipeline interfaces)
  • crates/masterd-sidecars (sidecar topology + validation)
  • apps/masterd-bootstrap (validates sidecar config and bootstrap assumptions)
  • apps/masterd-desktop-tauri (desktop shell stub for upcoming Tauri UI wiring)
• Sidecar topology config at config/sidecars.toml
• Pipeline architecture config at config/pipeline.toml
• Vendor helper scripts in scripts/

Vendor repos cloned

App-local source dependencies live under vendor/:

• candle
• tokenizers
• tauri
• lopdf
• tesseract-rs
• iced

Embedding model (critical)

You can ship a single installer that includes everything, but not all of these should be one in-process binary:

• Meilisearch: run as supervised sidecar process
• Valkey: run as supervised sidecar process
• Falkor module: load into Valkey/Redis sidecar (--loadmodule)
• LanceDB: in-process Rust crate integration (not a daemon)
• model2vec-rs: vendored Rust crate for local static embeddings and structural fallback

This repo enforces that model via masterd-sidecars::validate_foundation().

Personality + prompt port scope

• MASTERd personality source is consolidated in:
  • models/masterd-identity/masterd_personality_prompt.txt
• Rust prompt registry loader:
  • crates/masterd-prompt-core

Run-only shortcuts

cd /home/local/ai/projects/MASTERd
cargo run -p masterd-bootstrap

cd /home/local/ai/projects/MASTERd
cargo run -p masterd-ingest -- --root /path/to/files

Embedded local inference setup (ported)

Copied from your atom-rs/gfxatom runtime pattern:

• ColBERT wrapper: http://127.0.0.1:11450 (colbert-lfm2-305m)
• Jina embeddings: http://127.0.0.1:11447 (jina-code-embed)
• model2vec-rs fallback: vendored local crate in crates/model2vec-rs

Config file: config/embedding_engine.toml
Env overrides supported: MEMORYBANK_COLBERT_WRAPPER_URL, MEMORYBANK_JINA_URL, MEMORYBANK_EMBED_CONCURRENCY.

Backend mode:

• MASTERD_INFERENCE_BACKEND=direct (default): self-contained Rust direct calls (no local model HTTP servers required)
• MASTERD_INFERENCE_BACKEND=http: use the local endpoint wrappers above

To run ingest + engine verification/benchmark:

cd /home/local/ai/projects/MASTERd
cargo run -p masterd-ingest -- --root /path/to/files --verify-engine true --benchmark-engine true

To validate inference + retrieval + optional thinking chat and write a report:

cd /home/local/ai/projects/MASTERd
cargo run -p masterd-engine-check -- --chat-url http://127.0.0.1:3000

Report output path (default): data/engine_validation.json.

AMD-first installer + auto-tuning

Profiles and kernel manifest:

• config/amd_profiles/*.toml
• config/kernel_manifest.toml

Run startup-safe tune:

cd /home/local/ai/projects/MASTERd
cargo run -p masterd-tune -- --auto

Run full retune:

cd /home/local/ai/projects/MASTERd
cargo run -p masterd-tune -- --retune

Build installer bundle:

cd /home/local/ai/projects/MASTERd
./scripts/build-installer-bundles.sh

Installer sequence:

1. Launch boot MIDI unless MASTERD_NO_MUSIC=1.
2. Run scripts/download-models.sh to install model weights, tokenizers, and chat templates unless MASTERD_SKIP_MODEL_DOWNLOAD=1.
3. Run scripts/setup-embedding-services.sh all unless MASTERD_SKIP_EMBEDDING_SERVICES=1.
4. Download/build sidecars, build the Next shell, and package Tauri.

Boot screen + Rust music

• Installer launch shows the ANSI boot logo and waits for Enter.
• Boot music is played by bundled Rust app: apps/masterd-midi-player.
• Disable installer music with MASTERD_NO_MUSIC=1.

Run music player directly:

cd /home/local/ai/projects/MASTERd
cargo run -q -p masterd-midi-player -- --seconds 8

---

Aggressive Rust pipeline surfaces (superior-by-default)

All critical pipeline capabilities are now fully implemented in Rust.

Stage graph runtime (crates/masterd-pipeline)

Typed, deterministic stage-graph with cooperative cancellation and rollback hooks. Configure stage order in config/pipeline.toml under [runtime].stage_order.

Telemetry taxonomy (masterd_pipeline::telemetry)

Machine-actionable failure classes, per-stage counters, and wall-clock timing:

FailureClass: TransientIo | CorruptInput | DependencyUnavailable | ResourceExhausted | PolicyRejected | Cancelled | InternalError

Each class carries is_retryable() and is_expected() predicates for automated triage.

Naming and routing (masterd_pipeline::naming)

Rule-pack loader + deterministic priority resolver. Rule packs live in config/naming/*.json.

# Resolve a file path to route + canonical name (example usage in ingest):
cargo run -p masterd-ingest -- --root /path/to/files

Retrieval pipeline (masterd_pipeline::retrieval)

Typed query parser + multi-stage retrieval + dedup-merge + rerank hooks as the default search path. Query syntax: terms... key:value top:N mode:(lexical|semantic|hybrid).

UI workflow contract (crates/masterd-ui-contract)

Typed Tauri/Iced event contract for the review queue, operator commands, and correction loop. All events are namespaced under masterd:// for Tauri routing.

Cancellation contract (crates/masterd-core)

CancellationSource / CancellationToken — cooperative, reason-carrying cancellation across all long-running pipeline stages.

Extraction fallback policy (crates/masterd-embed-engine)

Centralized multi-provider fallback with bounded retries, quality scoring, and full audit trail via ExtractionExecutionReport.

Atomic hash-index dedup (apps/masterd-ingest)

AtomicHashIndexService with advisory lock-file, atomic tmp→rename write, and Drop-based lock cleanup. Concurrency-safe across threads.

Language integrity and authority policy

• Preserve established propositions exactly: scope, quantifiers, comparison class, implication direction, authority relationship, and stated conditions.
• Do not add unsourced caveats, counterclaims, balancing clauses, exceptions, or competing interpretations to settled statements.
• Project-defined semantics and authority relationships override generic terminology, model priors, and analogies.
• Never invert authority because a precursor, raw serialization, cache, residual, reconstruction, or historical form existed first.
• Keep Established, Derived, Measured, Proposed, Speculative, Reject, and Conflict statements distinct.
• Do not use “within this architecture”, “this does not mean”, “does not necessarily imply”, “but/however/although”, “merely”, “only”, or similar language to weaken an established proposition without explicit governing evidence.
• Preserve comparison classes and units; name exactly what is exact, approximate, represented, measured, routed, reconstructed, or executed.
• If evidence conflicts, record an explicit Conflict; do not silently hedge or rewrite the canonical statement.
• Do not manufacture uncertainty and do not invent undocumented limitations.
• ZetaQuant: A successfully ZetaQuantized datum is the canonical enriched form of its source. It preserves exact recoverability of the source representation and adds further calculated structure; therefore the ZetaQuantized datum is richer, more informative, and more exact as a represented state than the original source serialization.