|
1 | | -# Epistemic Forge |
| 1 | +<div align="center"> |
2 | 2 |
|
3 | | -**Stop shipping confident mush.** |
4 | | -Turn messy research, philosophy, freelance, and Kaggle questions into **claim lattices**—structured arguments with evidence slots, objections, confidence, and next actions. |
5 | | - |
6 | | -[](https://www.python.org/downloads/) |
7 | | -[](LICENSE) |
8 | | -[](tests/) |
9 | | -[](https://github.com/faresrafat3/arsenal-unified-master-pipeline) |
10 | | -[](docs/benchmark/BENCHMARK.md) |
11 | | - |
12 | | -Local-first Python package (**no paid API required**). Implements a practical slice of the [ARSENAL](https://github.com/faresrafat3/arsenal-unified-master-pipeline) L0–L6 pipeline. |
13 | | - |
14 | | ---- |
15 | | - |
16 | | -## The problem |
17 | | - |
18 | | -Experts and freelancers constantly answer hard questions under uncertainty: |
19 | | - |
20 | | -- a founder wants a **2-week research sprint** scoped *now* |
21 | | -- a Kaggle teammate wants a **leakage-safe baseline**, not vibes |
22 | | -- a seminar needs a **dialectic**, not a blog take |
23 | | - |
24 | | -Default LLM / one-shot Q&A style fails the same way every time: |
25 | | - |
26 | | -| Failure mode | What you get | |
27 | | -|---|---| |
28 | | -| Hidden assumptions | Smooth prose, no load-bearing claims listed | |
29 | | -| No objections | One-sided answers that collapse in review | |
30 | | -| False precision | Confident tone without confidence labels | |
31 | | -| Weak packaging | No acceptance criteria, checklist, or next action | |
32 | | - |
33 | | -**Epistemic Forge** forces a shared intermediate representation—a **claim lattice**—then packages it for the domain you’re in. |
34 | | - |
35 | | -Academic anchor: **Toulmin’s model of argument** (claim · grounds · warrant · rebuttal · qualifier). |
36 | | -See [`docs/ACADEMIC_FRAMEWORK.md`](docs/ACADEMIC_FRAMEWORK.md) · Toulmin, *The Uses of Argument* (1958/2003). |
37 | | - |
38 | | ---- |
39 | | - |
40 | | -## Before / after (real run) |
41 | | - |
42 | | -**Question:** *If the brain is a prediction machine, what happens to moral responsibility?* |
43 | | - |
44 | | -### Before — baseline one-shot Q&A |
45 | | -**Overall 0.37** · **Toulmin coverage 0.40** |
| 3 | +# 🧠 Epistemic Forge |
46 | 4 |
|
47 | | -```text |
48 | | -Predictive processing and moral responsibility: On the question «If the brain |
49 | | -is a prediction machine, what happens to moral responsibility?», a reasonable |
50 | | -view is that responsibility still applies when agents can track norms, even if |
51 | | -cognition is predictive. Keywords: predictive processing, responsibility, agency. |
52 | | -In short, keep blame practices but update the metaphysics of agency. |
53 | | -``` |
54 | | - |
55 | | -### After — Epistemic Forge claim lattice |
56 | | -**Overall 1.00** · **Toulmin coverage 1.00** on this memo |
57 | | -(full folder: [`examples/worked/01_philosophy_predictive_responsibility/`](examples/worked/01_philosophy_predictive_responsibility/)) |
58 | | - |
59 | | -The forge memo adds: explicit **claim IDs**, **supports**, **objections**, **confidence**, **dialectic (thesis/antithesis/steelman/synthesis)**, **limits & unknowns**, and **next actions**—the slots baseline Q&A skips. |
60 | | - |
61 | | -Side-by-side writeup: [`examples/worked/BEFORE_AFTER_philosophy.md`](examples/worked/BEFORE_AFTER_philosophy.md) |
62 | | - |
63 | | ---- |
64 | | - |
65 | | -## Benchmark (10 examples, measurable) |
66 | | - |
67 | | -We compare **claim-lattice pipeline output** vs a **simple Q&A baseline** on **10 fixed cases** (philosophy ×3, kaggle ×3, freelance ×3, hybrid ×1). |
68 | | - |
69 | | -Scoring is **deterministic** (no paid judge model): weighted Toulmin completeness + structure / actionability / humility. |
| 5 | +[](https://www.python.org/downloads/) |
| 6 | +[](LICENSE) |
| 7 | +[](https://github.com/faresrafat3/arsenal-unified-master-pipeline) |
| 8 | +[](docs/benchmark/BENCHMARK.md) |
70 | 9 |
|
71 | | -| Metric | Baseline Q&A | Epistemic Forge | Lift | |
72 | | -|---|---:|---:|---:| |
73 | | -| **Overall quality (0–1)** | 0.319 | **0.951** | **+0.633 (~+199%)** | |
74 | | -| **Toulmin coverage (0–1)** | 0.260 | **0.860** | **+0.600 (~+231%)** | |
75 | | -| **Wins (overall)** | — | **10 / 10** | — | |
76 | | -| **Wins (Toulmin)** | — | **10 / 10** | — | |
77 | | - |
78 | | -Per-case table and method notes: [`docs/benchmark/BENCHMARK.md`](docs/benchmark/BENCHMARK.md) · raw JSON: [`docs/benchmark/benchmark_results.json`](docs/benchmark/benchmark_results.json) |
79 | | - |
80 | | -```bash |
81 | | -python scripts/run_benchmark.py --out docs/benchmark |
82 | | -``` |
83 | | - |
84 | | -| ID | Domain | Baseline | Forge | Δ Overall | |
85 | | -|---|---|---:|---:|---:| |
86 | | -| p1–p3 | philosophy | ~0.37 | ~1.00 | ~+0.63 | |
87 | | -| k1–k3 | kaggle | ~0.30 | ~0.94 | ~+0.64 | |
88 | | -| f1–f3 | freelance | ~0.23 | ~0.90 | ~+0.67 | |
89 | | -| h1 | hybrid | 0.50 | 1.00 | +0.50 | |
| 10 | +**Stop shipping confident mush.** |
| 11 | +Turn messy research, philosophy, freelance, and Kaggle questions into **claim lattices** — structured arguments with evidence slots, objections, confidence, and next actions. |
90 | 12 |
|
91 | | -> Interpretation for portfolio readers: the gain is not “smarter vibes”—it is **argument slot coverage + packaging**. That is what clients, reviewers, and hiring managers can inspect. |
| 13 | +</div> |
92 | 14 |
|
93 | 15 | --- |
94 | 16 |
|
95 | | -## Worked examples (full outputs) |
| 17 | +## 🛑 The Problem |
96 | 18 |
|
97 | | -| Domain | Example | Score / review | |
98 | | -|---|---|---| |
99 | | -| **Philosophy** | [Predictive processing & responsibility](examples/worked/01_philosophy_predictive_responsibility/) | 0.66 · minor revisions | |
100 | | -| **Kaggle** | [Imbalanced tabular baseline](examples/worked/02_kaggle_imbalanced_baseline/) | 0.71 · minor revisions | |
101 | | -| **Freelance** | [Climate-tech 2-week sprint](examples/worked/03_freelance_climate_sprint/) | 0.71 · minor revisions | |
| 19 | +Experts and freelancers constantly answer hard questions under uncertainty: |
| 20 | +- A founder wants a **2-week research sprint** scoped *now*. |
| 21 | +- A Kaggle teammate wants a **leakage-safe baseline**, not vibes. |
| 22 | +- A seminar needs a **dialectic**, not a blog take. |
102 | 23 |
|
103 | | -Index: [`examples/worked/README.md`](examples/worked/README.md) |
| 24 | +Default LLM / one-shot Q&A style fails the same way every time: smooth prose with hidden assumptions, no objections, and false precision. |
104 | 25 |
|
105 | | -Each folder includes `executive_summary.md`, `research_memo.md`, domain packs (`client_brief.json` / `kaggle_spine.md`), and full `result.json` (route, trials, search nodes). |
| 26 | +## 🛠️ The Solution: Epistemic Forge |
| 27 | +A local-first Python package (**no paid API required**) that implements a practical slice of the [ARSENAL](https://github.com/faresrafat3/arsenal-unified-master-pipeline) L0–L6 pipeline. It forces the LLM to construct a **Toulmin-anchored claim lattice** before generating an answer. |
106 | 28 |
|
107 | | ---- |
| 29 | +### Key Features |
| 30 | +- 📊 **Toulmin-Anchored Benchmark**: +199% overall quality vs baseline Q&A on 10 edge cases. |
| 31 | +- 🔗 **Claim Lattices**: Maps claims, warrants, backing, and rebuttals automatically. |
| 32 | +- ⚡ **Local-First CLI**: Fully functional directly from your terminal. |
108 | 33 |
|
109 | | -## Install |
| 34 | +## 🚀 Getting Started |
110 | 35 |
|
111 | 36 | ```bash |
112 | 37 | git clone https://github.com/faresrafat3/epistemic-forge.git |
113 | 38 | cd epistemic-forge |
114 | | -pip install -e ".[dev]" |
115 | | -pytest -q |
| 39 | +pip install -r requirements.txt |
| 40 | +python main.py --query "Is RAG strictly better than Long-Context LLMs?" |
116 | 41 | ``` |
117 | | - |
118 | | -## Quick start |
119 | | - |
120 | | -```bash |
121 | | -epistemic-forge \ |
122 | | - --title "Predictive minds and blame" \ |
123 | | - --question "If the brain is a prediction machine, what happens to moral responsibility?" \ |
124 | | - --domain philosophy \ |
125 | | - --keywords "predictive processing,agency,responsibility" \ |
126 | | - --out runs/philosophy_demo |
127 | | -``` |
128 | | - |
129 | | -```bash |
130 | | -epistemic-forge \ |
131 | | - --title "Imbalanced tabular baseline" \ |
132 | | - --question "What is an honest baseline plan for a noisy imbalanced Kaggle table?" \ |
133 | | - --domain kaggle \ |
134 | | - --keywords "imbalance,cv,leakage,baseline" \ |
135 | | - --out runs/kaggle_demo |
136 | | -``` |
137 | | - |
138 | | -```bash |
139 | | -epistemic-forge \ |
140 | | - --title "Climate-tech research sprint" \ |
141 | | - --question "How do I scope a 2-week research sprint for a climate-tech founder?" \ |
142 | | - --domain freelance \ |
143 | | - --keywords "sprint,scope,founder" \ |
144 | | - --out runs/freelance_demo |
145 | | -``` |
146 | | - |
147 | | -### Python API |
148 | | - |
149 | | -```python |
150 | | -from epistemic_forge import run_pipeline |
151 | | -from epistemic_forge.io.export import export_result |
152 | | - |
153 | | -result = run_pipeline( |
154 | | - title="Epistemic freelancing", |
155 | | - question="How can freelancers package uncertain research without overclaiming?", |
156 | | - domain="hybrid", |
157 | | - keywords=["freelance", "research", "claims"], |
158 | | -) |
159 | | -print(result.final_score, result.peer_review["verdict"]) |
160 | | -export_result(result, "runs/hybrid_demo") |
161 | | -``` |
162 | | - |
163 | | -## What you get in `--out` |
164 | | - |
165 | | -| File | Purpose | |
166 | | -|---|---| |
167 | | -| `executive_summary.md` | One-pager | |
168 | | -| `research_memo.md` | Claim lattice + framing + limits + actions | |
169 | | -| `client_brief.json` | Freelance pack (when domain fits) | |
170 | | -| `kaggle_spine.md` | Notebook spine (when domain fits) | |
171 | | -| `result.json` | Full trace (route, trials, search nodes) | |
172 | | -| `MANIFEST.json` | Score, review, file list | |
173 | | - |
174 | | -## Domains |
175 | | - |
176 | | -`research` · `philosophy` · `writing` · `freelance` · `kaggle` · `hybrid` |
177 | | - |
178 | | ---- |
179 | | - |
180 | | -## ARSENAL mapping (how the sausage is made) |
181 | | - |
182 | | -| Layer | ARSENAL idea | In this repo | |
183 | | -|---|---|---| |
184 | | -| **L0** | Technique routing | Domain → families + layer flags | |
185 | | -| **L1** | APE + OPRO | Seed instructions + score-history climb | |
186 | | -| **L2** | Meta conductor | Claim / dialectic / writing / freelance / Kaggle experts | |
187 | | -| **L3** | ToT (+ LATS cascade) | Beam search over framings; optional rollout polish | |
188 | | -| **L4** | Self-Refine | Multi-aspect critique → revise loop | |
189 | | -| **L5** | Reflexion + Voyager | Verbal trial memory + skill library | |
190 | | -| **L6** | Stage shell | Artifacts, executive summary, peer-review rubric | |
191 | | - |
192 | | -Design log: [`docs/ARSENAL_RUN_LOG.md`](docs/ARSENAL_RUN_LOG.md) |
193 | | - |
194 | | -## Project layout |
195 | | - |
196 | | -```text |
197 | | -epistemic_forge/ |
198 | | - pipeline/ # L0–L6 orchestration |
199 | | - experts/ # domain specialists |
200 | | - memory/ # Reflexion + skills |
201 | | - benchmark/ # baseline vs forge metrics |
202 | | - io/ # export |
203 | | -examples/worked/ # full philosophy / kaggle / freelance runs |
204 | | -docs/benchmark/ # numbers + method |
205 | | -docs/ACADEMIC_FRAMEWORK.md |
206 | | -tests/ |
207 | | -``` |
208 | | - |
209 | | -## Tests & regenerate portfolio assets |
210 | | - |
211 | | -```bash |
212 | | -pytest -q |
213 | | -python scripts/run_benchmark.py --out docs/benchmark |
214 | | -python scripts/generate_worked_examples.py |
215 | | -``` |
216 | | - |
217 | | -## Related |
218 | | - |
219 | | -- ARSENAL: https://github.com/faresrafat3/arsenal-unified-master-pipeline |
220 | | -- Extractions archive: https://github.com/faresrafat3/llm-agent-research-extractions |
221 | | - |
222 | | -## License |
223 | | - |
224 | | -MIT — see [LICENSE](LICENSE). |
225 | | - |
226 | | -## Disclaimer |
227 | | - |
228 | | -Epistemic Forge **scaffolds thinking and packaging**. It does not replace domain expertise, ethics review, or competition rules. Benchmark scores are **automatic structural metrics**, not human truth judgments. Treat outputs as **provisional** claims with explicit limits. |
0 commit comments