fix: remediate July-22 technical review findings

.gitignore

@@ -14,3 +14,8 @@ runs/
 forge_output/
 .DS_Store
 *.ipynb_checkpoints/
+
+# coverage artifacts
+.coverage
+coverage.xml
+htmlcov/

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help install test lint run clean
+.PHONY: help install test lint clean cli ui serve
 
 # Default command when just running 'make'
 help:
@@ -32,7 +32,12 @@ clean:
 
 cli:
 	@echo "Running CLI test query..."
-	epistemic-forge --query "Is RAG strictly better than Long-Context LLMs?"
+	epistemic-forge --title "RAG vs Long-Context" --question "Is RAG strictly better than Long-Context LLMs?"
+
+ui:
+	@echo "Launching Streamlit dashboard (requires the 'ui' extra)..."
+	pip install -e ".[ui]"
+	streamlit run epistemic_forge/ui/app.py
 
 serve:
 	@echo "Booting Enterprise API Server..."

docs/REVIEW_REMEDIATION.md

@@ -0,0 +1,82 @@
+# Review Remediation Plan — Epistemic Forge
+
+This document maps every finding from the July 22, 2026 technical review to a
+concrete fix that has been (or will be) applied. The goal is to address the
+**legitimate engineering issues** raised, while being honest about scope:
+formal epistemic-logic research (Kripke models, AGM belief revision, trained
+PRMs, true MCTS-UCB) is a multi-month research program and is documented here
+as a roadmap item rather than overclaimed as "done".
+
+## Priority 1 — CI/CD is broken & non-enforcing (CRITICAL)
+- [x] Rewrite `.github/workflows/ci.yml`: remove `|| echo`, make `pytest` blocking.
+- [x] Add coverage reporting (`pytest-cov`) with a floor.
+- [x] Make `ruff` a **blocking** lint gate (was `--exit-zero` warning only).
+- [x] Add `mypy` (type checking) and `bandit` (security scan) to CI.
+- [x] Fix `tests.yml` matrix + blocking tests + coverage.
+
+## Priority 2 — Critical correctness bugs
+- [x] Fix `test_l2_conductor_routing`: it passed `spec.domain.value` (str) where
+      `SemanticConductor._route_experts(spec: ProjectSpec)` is expected.
+- [x] `arsenal_run.py`: move `route_project` import to module top (was imported
+      inside `run()`).
+- [x] `arsenal_run.py`: remove unreachable dead code after `return`.
+- [x] `arsenal_run.py`: `run_pipeline` used `raise SystemExit(1)` in library code →
+      replaced with a re-raised `PipelineError` (CLI still does `sys.exit`).
+
+## Priority 3 — Code integrity (RED FLAG)
+- [x] Delete `patch_run.py` (string-replacement hack). Its *intended* logging was
+      folded into `arsenal_run.py` properly via a normal edit.
+
+## Priority 4 — Error handling hygiene
+- [x] `memory/economy.py`: bare `except: pass` → `except Exception`.
+- [x] `memory/skill_library.py`: bare `except:` in `get_all_skills` → `except Exception`.
+- [x] General: no new bare excepts introduced; ruff `E722` enforced in CI.
+
+## Priority 5 — Security
+- [x] `llm.py`: stop writing API keys into `os.environ` globally (redundant — the
+      key is already passed per-call via `call_params["api_key"]`).
+- [x] `llm.py`: add input validation (`validate_messages`) — reject empty/oversized
+      prompts before dispatch, capping token blow-up.
+- [x] `l1_5_adas.py`: harden dynamic schema generation — bound field count, validate
+      identifiers, reject empty/unsafe blueprints (no arbitrary code execution; only
+      Pydantic `create_model` from sanitized names).
+
+## Priority 6 — Dependency hygiene
+- [x] Add `ruff`, `pytest-cov`, `mypy`, `bandit`, `types-requests` to `dev`.
+- [x] Move `streamlit` out of core deps into an optional `ui` extra (non-UI users
+      no longer pull a heavy web framework).
+- [ ] Lock file (`uv.lock` / `pip-compile`) — tracked as follow-up; see CHANGELOG.
+
+## Priority 7 — Testing (was <10% coverage)
+- [x] Fix existing test + big expansion: `llm` offline fallback, `router`,
+      `l1_optimizer`, `l2_conductor`, `l1_5_adas`, `l3_search`, `l4_refine`,
+      `l6_stages`, `skill_library` (with injected fake client), `economy`,
+      `models` (Pydantic constraints), `cli` arg parsing, and a **full offline
+      pipeline integration test**.
+- [x] Coverage floor enforced in CI (start modest, raise as suite grows).
+
+## Priority 8 — Architecture honesty
+- [x] `pipeline/machine.py`: explicit, data-driven **stage registry** (`L0→L6`)
+      with per-stage enable predicates, plus a typed `PipelineContext`
+      (replaces ad-hoc `Dict[str, Any]`), so routing is inspectable, not hidden
+      in `if` statements.
+- [x] Async: `llm.agenerate_structured` (real `litellm.acompletion` + instructor),
+      parallel `conduct_async` (experts run concurrently via `asyncio.gather`),
+      and `arun_pipeline` entrypoint — addresses "no async / sequential experts".
+
+## Priority 9 — Documentation honesty
+- [x] README: mark the "85% cost / 80% fewer tokens" figures as **illustrative
+      targets, not independently benchmarked**; add a "What 'Epistemic' Means
+      Here" section clarifying it is structured Toulmin prompting, not formal
+      logic; add architecture diagram.
+- [x] `docs/ARCHITECTURE.md` with Mermaid diagram + data contracts.
+- [x] `docs/BENCHMARKS.md` describing methodology + current caveats.
+- [x] `CHANGELOG.md`.
+
+## Out of scope / roadmap (documented honestly, NOT overclaimed)
+- Formal Dynamic Epistemic Logic / Kripke structures / AGM belief revision.
+- Trained Process Reward Model (current "PRM" is LLM-as-Judge prompting).
+- True MCTS with UCB / value backpropagation (current L3 is bounded beam enumeration).
+- Full multi-agent message-passing communication layer (current "experts" are
+  sequential strategy-pattern nodes).
+- These remain research roadmap items; the naming is now qualified in docs.

epistemic_forge/__init__.py

@@ -1,7 +1,7 @@
 """Epistemic Forge — ARSENAL-powered research & writing kit."""
 
-from .pipeline.arsenal_run import ArsenalRun, run_pipeline
 from .models import Claim, ForgeResult, ProjectSpec
+from .pipeline.arsenal_run import ArsenalRun, run_pipeline
 
 __version__ = "0.1.0"
 __all__ = [

epistemic_forge/benchmark/baseline.py

@@ -2,10 +2,8 @@
 
 from __future__ import annotations
 
-from typing import List
 
-
-def baseline_answer(title: str, question: str, domain: str, keywords: List[str]) -> str:
+def baseline_answer(title: str, question: str, domain: str, keywords: list[str]) -> str:
     """Produce a short unstructured answer — typical one-shot Q&A quality."""
     kw = ", ".join(keywords) if keywords else "the main themes"
     # Deliberately thin: claim-ish sentence, weak support, no rebuttal structure

epistemic_forge/benchmark/llm_judge.py

@@ -1,35 +1,47 @@
 """LLM-as-a-Judge for Automated, Scientific Epistemic Evaluation (Toulmin Model)."""
-from typing import Dict, Any
-from epistemic_forge.models import JudgeEvaluation
+
+from typing import Any
+
+from pydantic import BaseModel, Field
+
 from epistemic_forge.llm import generate_structured
 from loguru import logger
 
-def evaluate_artifact_quality(question: str, artifact_text: str) -> Dict[str, Any]:
+
+class JudgeEvaluation(BaseModel):
+    """Structured output schema for the LLM-as-Judge benchmark."""
+
+    logical_coherence_score: float = Field(..., ge=0.0, le=1.0)
+    hallucination_detected: bool = False
+    critique: str = ""
+
+
+def evaluate_artifact_quality(question: str, artifact_text: str) -> dict[str, Any]:
     """Uses a stronger model to judge the output based strictly on Toulmin's Model of Argumentation."""
     logger.info("⚖️ Initiating strict Toulmin-based evaluation of the final artifact...")
-    
+
     messages = [
         {
-            "role": "system", 
+            "role": "system",
             "content": (
                 "You are an Elite Academic Peer Reviewer specializing in the Toulmin Model of Argumentation. "
                 "Do NOT judge the artifact based on prose or formatting. You must ONLY evaluate the strength of the 'Warrants' (do they bridge the data to the claim?) "
                 "and the validity of the 'Rebuttals/Falsifiers' (are they real weaknesses or just strawmen?)."
-            )
+            ),
         },
-        {"role": "user", "content": f"Core Inquiry: {question}\n\nSubmitted Artifact:\n{artifact_text}\n\nExecute the Toulmin Evaluation."}
+        {"role": "user", "content": f"Core Inquiry: {question}\n\nSubmitted Artifact:\n{artifact_text}\n\nExecute the Toulmin Evaluation."},
     ]
-    
+
     # We use a robust model for judging, maintaining temp 0.0 for deterministic grading
     evaluation: JudgeEvaluation = generate_structured(
         messages=messages,
         response_model=JudgeEvaluation,
-        model="openai/gpt-4o-mini", # Standardizing to openrouter/openai model format
-        api_base="https://openrouter.ai/api/v1" # Enforce OpenRouter for testing consistency
+        model="openai/gpt-4o-mini",  # Standardizing to openrouter/openai model format
+        api_base="https://openrouter.ai/api/v1",  # Enforce OpenRouter for testing consistency
     )
-    
+
     return {
         "score": evaluation.logical_coherence_score,
         "hallucination": evaluation.hallucination_detected,
-        "critique": evaluation.critique
+        "critique": evaluation.critique,
     }

epistemic_forge/benchmark/metrics.py

@@ -8,7 +8,7 @@
 
 import re
 from dataclasses import asdict, dataclass
-from typing import Any, Dict, List, Optional
+from typing import Any
 
 
 @dataclass
@@ -44,26 +44,26 @@ def overall(self) -> float:
             total += getattr(self, k) * w
         return round(total, 4)
 
-    def to_dict(self) -> Dict[str, Any]:
+    def to_dict(self) -> dict[str, Any]:
         d = asdict(self)
         d["overall"] = self.overall()
         return d
 
 
-def _hit(text: str, patterns: List[str]) -> float:
+def _hit(text: str, patterns: list[str]) -> float:
     t = text.lower()
     return 1.0 if any(p in t for p in patterns) else 0.0
 
 
-def _count_hits(text: str, patterns: List[str]) -> int:
+def _count_hits(text: str, patterns: list[str]) -> int:
     t = text.lower()
     return sum(1 for p in patterns if p in t)
 
 
 def score_document(
     text: str,
     domain: str = "hybrid",
-    keywords: Optional[List[str]] = None,
+    keywords: list[str] | None = None,
 ) -> QualityScores:
     """Score a free-text answer for Toulmin completeness + packaging quality."""
     keywords = keywords or []

epistemic_forge/benchmark/suite.py

@@ -5,7 +5,7 @@
 import json
 from dataclasses import asdict, dataclass
 from pathlib import Path
-from typing import Any, Dict, List, Optional
+from typing import Any
 
 from epistemic_forge.benchmark.baseline import baseline_answer
 from epistemic_forge.benchmark.metrics import score_document, toulmin_coverage
@@ -18,10 +18,10 @@ class BenchCase:
     title: str
     question: str
     domain: str
-    keywords: List[str]
+    keywords: list[str]
 
 
-BENCHMARK_CASES: List[BenchCase] = [
+BENCHMARK_CASES: list[BenchCase] = [
     BenchCase(
         "p1",
         "Predictive processing and blame",
@@ -106,8 +106,8 @@ class CaseResult:
     forge_toulmin: float
     lift_overall: float
     lift_toulmin: float
-    baseline_scores: Dict[str, Any]
-    forge_scores: Dict[str, Any]
+    baseline_scores: dict[str, Any]
+    forge_scores: dict[str, Any]
 
 
 def _forge_text(case: BenchCase) -> str:
@@ -125,9 +125,9 @@ def _forge_text(case: BenchCase) -> str:
     return "\n\n".join(a.content for a in result.artifacts)
 
 
-def run_benchmark(cases: Optional[List[BenchCase]] = None) -> Dict[str, Any]:
+def run_benchmark(cases: list[BenchCase] | None = None) -> dict[str, Any]:
     cases = cases or BENCHMARK_CASES
-    rows: List[CaseResult] = []
+    rows: list[CaseResult] = []
     for case in cases:
         base_txt = baseline_answer(
             case.title, case.question, case.domain, case.keywords
@@ -187,7 +187,7 @@ def run_benchmark(cases: Optional[List[BenchCase]] = None) -> Dict[str, Any]:
     }
 
 
-def write_benchmark_reports(out_dir: str | Path) -> Dict[str, Any]:
+def write_benchmark_reports(out_dir: str | Path) -> dict[str, Any]:
     out = Path(out_dir)
     out.mkdir(parents=True, exist_ok=True)
     report = run_benchmark()

epistemic_forge/cli.py

@@ -8,14 +8,15 @@
 
 import argparse
 import sys
+
 from rich.console import Console
 from rich.panel import Panel
 from rich.progress import Progress, SpinnerColumn, TextColumn
 from rich.tree import Tree
 
+from epistemic_forge.memory.economy import budget_manager
 from epistemic_forge.models import ProjectSpec
 from epistemic_forge.pipeline.arsenal_run import run_pipeline
-from epistemic_forge.memory.economy import budget_manager
 
 console = Console()
 
@@ -107,12 +108,13 @@ def main():
                 "[cyan]L2 Conductor is dispatching semantic experts...", total=None
             )
 
-            # Execute the core pipeline
+            # Execute the core pipeline (Hermes routing overrides are passed through)
             result = run_pipeline(
                 title=spec.title,
                 question=spec.question,
                 domain=spec.domain.value,
-                # Note: To fully support Hermes, pipeline/arsenal_run.py must pass target_model down.
+                target_model=args.model,
+                api_base=args.api_base,
             )
 
             progress.update(
@@ -123,17 +125,20 @@ def main():
         # Render output
         console.print("[bold green]✔ Pipeline Execution Successful.[/bold green]\n")
 
-        if hasattr(result, "claims"):
+        if hasattr(result, "claims") and result.claims:
             display_claim_lattice(result.claims)
-
-            # SOTA EXPORT
-            from epistemic_forge.io.export import export_result
-            export_dir = f'runs/{spec.title.replace(" ", "_").lower()}'
-            export_result(result, export_dir)
+
         else:
             console.print(
                 "[yellow]Notice: No claims extracted in the final result.[/yellow]"
             )
+
+        # SOTA EXPORT
+        from epistemic_forge.io.export import export_result
+
+        export_dir = f'runs/{spec.title.replace(" ", "_").lower()}'
+        export_result(result, export_dir)
+
         console.print(f"\n[bold yellow]💰 {budget_manager.get_report()}[/bold yellow]")
 
     except Exception:

epistemic_forge/errors.py

@@ -0,0 +1,42 @@
+"""Public error types for Epistemic Forge.
+
+Library code must never call ``sys.exit`` / ``SystemExit``. Callers (CLI,
+notebooks, servers) decide how to surface failures; the library only raises
+typed exceptions that are safe to catch.
+"""
+
+from __future__ import annotations
+
+
+class EpistemicForgeError(Exception):
+    """Base class for all library errors."""
+
+
+class PipelineError(EpistemicForgeError):
+    """Raised when the L0–L6 pipeline cannot complete.
+
+    Wraps the underlying cause so callers can introspect ``__cause__``.
+    """
+
+    def __init__(self, message: str, *, stage: str | None = None) -> None:
+        self.stage = stage
+        super().__init__(message)
+
+
+class BudgetExceededError(EpistemicForgeError):
+    """Raised when the cognitive economy budget is exhausted."""
+
+    def __init__(self, used: int, limit: int) -> None:
+        self.used = used
+        self.limit = limit
+        super().__init__(
+            f"Cognitive budget exceeded: {used} tokens used (limit {limit})."
+        )
+
+
+class LLMDispatchError(EpistemicForgeError):
+    """Raised when the universal LLM router cannot fulfil a request."""
+
+
+class InvalidInputError(EpistemicForgeError):
+    """Raised when user-supplied input fails validation."""