Skip to content

Commit 4021174

Browse files
committed
Updated Guide and Build Fix
1 parent 23fd2c6 commit 4021174

7 files changed

Lines changed: 103 additions & 17 deletions

File tree

Asset/SCR-20260423-oqsn.png

57.7 KB
Loading

Asset/SCR-20260423-ordo.png

221 KB
Loading

Asset/SCR-20260423-orzh.png

40.1 KB
Loading

Asset/l Missionctrl TASK.png

562 KB
Loading

README.md

Lines changed: 48 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,12 +44,41 @@ A real-time visualization dashboard at `/dashboard` shows:
4444
- Action timeline with reward indicators
4545
- **Accumulated run results** that persist across tiers with expandable per-tier reports
4646

47+
![MissionCtrl Dashboard](Asset/l%20Missionctrl%20TASK.png)
48+
49+
#### Run Results Breakdown
50+
The Run Results panel now supports expandable per-tier drilldowns for:
51+
- Score breakdown contributions by signal
52+
- Hallucination stats (injected/caught/TP/FP)
53+
- Action-by-action reward history
54+
55+
![Run Results Panel](Asset/SCR-20260423-ordo.png)
56+
4757
### 🔄 Deterministic Replay
4858
Every episode can be deterministically replayed via seeded randomness, enabling reproducible debugging and benchmarking.
4959

5060
### 🐳 Single-Container Deployment
5161
Server + inference in one Docker image. No orchestration, no external databases — just `docker run` and go.
5262

63+
### 🧾 Verbose LLM Trace View
64+
When `VERBOSE_TRACE=1`, inference prints compact boxed traces for each step:
65+
- Prompt metadata (including char count)
66+
- Prompt preview for fast debugging
67+
- Action normalization and guardrail rewrites
68+
- Step transition outcomes and rewards
69+
70+
![LLM Prompt Trace](Asset/SCR-20260423-oqsn.png)
71+
72+
![LLM Response Trace](Asset/SCR-20260423-orzh.png)
73+
74+
### 🚦 Token-Budget Guardrails
75+
Inference now includes hardening for provider token limits:
76+
- **Stateless per-step LLM requests** (fresh system + current observation only)
77+
- **No retry loop for permanent oversized-request errors**
78+
- Retry/backoff remains enabled for transient provider throttling
79+
80+
This prevents late-step context blowups (for example, step 5 payload growth) from repeatedly failing with the same "request too large" response.
81+
5382
---
5483

5584
## 📈 Baseline Results
@@ -204,13 +233,29 @@ pytest tests/ -v
204233

205234
| Variable | Default | Description |
206235
|---|---|---|
207-
| `API_BASE_URL` | `https://api.groq.com/openai/v1` | LLM API endpoint |
208-
| `MODEL_NAME` | `llama-3.3-70b-versatile` | Model to use |
236+
| `API_BASE_URL` | `https://router.huggingface.co/v1` | OpenAI-compatible LLM API endpoint |
237+
| `MODEL_NAME` | `openai/gpt-oss-120b` | Model to use |
209238
| `HF_TOKEN` || API key |
210-
| `STEP_DELAY_S` | `0.5` | Delay between steps (reduce for speed) |
239+
| `ENV_BASE_URL` | `http://localhost:8000` | MissionCtrl server base URL |
240+
| `STEP_DELAY_S` | `4.0` | Delay between steps (reduce for speed) |
211241
| `VERBOSE_TRACE` | `1` | Show detailed step traces |
242+
| `PROMPT_PREVIEW_CHARS` | `200` | Prompt preview truncation length in trace logs |
243+
| `TRACE_WRAP_WIDTH` | `76` | Text wrap width for trace block content |
244+
| `TRACE_BOX_WIDTH` | `76` | Width of the boxed trace output |
245+
| `SPINNER_ENABLED` | `0` | Enable CLI spinner while waiting for LLM response |
212246
| `MAX_STEPS` | `5` | Steps per episode |
213247

248+
### Troubleshooting: Request Too Large / TPM Errors
249+
250+
If your provider returns errors like:
251+
`Request too large ... tokens per minute ... Requested > Limit`
252+
253+
Use this checklist:
254+
1. Ensure you are running the latest image/code with stateless per-step requests.
255+
2. Reduce verbosity/observation size if needed (fewer long output snippets).
256+
3. Switch to a model/tier with higher TPM limits.
257+
4. Keep retries for transient rate limits; oversized requests are now treated as non-retryable.
258+
214259
---
215260

216261
## 📋 Task Tiers

inference.py

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@
3131
import httpx
3232
from dotenv import load_dotenv
3333
from openai import OpenAI
34-
from tenacity import retry, stop_after_attempt, wait_exponential, before_sleep_log
34+
from tenacity import retry, stop_after_attempt, wait_exponential, before_sleep_log, retry_if_not_exception_type
3535
import logging as _logging
3636

3737
# ---------------------------------------------------------------------------
@@ -96,6 +96,11 @@ def _validate_env() -> None:
9696
_retry_logger = _logging.getLogger("missionctrl.retry")
9797

9898

99+
class PromptTooLargeError(RuntimeError):
100+
"""Raised when provider rejects a request as permanently oversized."""
101+
102+
103+
99104
def _append_bounded_unique(bucket: List[str], value: str, limit: int) -> None:
100105
value = value.strip()
101106
if not value:
@@ -685,6 +690,7 @@ def _spin():
685690
stop=stop_after_attempt(LLM_MAX_RETRIES),
686691
wait=wait_exponential(multiplier=2, min=2, max=30),
687692
before_sleep=before_sleep_log(_retry_logger, _logging.WARNING),
693+
retry=retry_if_not_exception_type(PromptTooLargeError),
688694
reraise=True,
689695
)
690696
def _call_llm(messages: List[Dict[str, str]]) -> str:
@@ -698,6 +704,9 @@ def _call_llm(messages: List[Dict[str, str]]) -> str:
698704
)
699705
except Exception as exc:
700706
msg = str(exc)
707+
lower_msg = msg.lower()
708+
if "request too large" in lower_msg or ("tokens per minute" in lower_msg and "requested" in lower_msg):
709+
raise PromptTooLargeError(f"Prompt too large: {msg.splitlines()[0]}") from exc
701710
if "429" in msg or "rate_limit" in msg.lower():
702711
raise RuntimeError(f"Rate-limited: {msg.splitlines()[0]}") from exc
703712
raise
@@ -834,7 +843,7 @@ def run_task(task_id: str, policy_memory: PolicyMemory) -> float:
834843
done = False
835844

836845
try:
837-
messages: List[Dict[str, str]] = [{"role": "system", "content": SYSTEM_PROMPT}]
846+
system_message = {"role": "system", "content": SYSTEM_PROMPT}
838847
action_history: List[str] = []
839848
episode_memory = EpisodeMemory()
840849

@@ -850,7 +859,10 @@ def run_task(task_id: str, policy_memory: PolicyMemory) -> float:
850859
episode_memory,
851860
policy_memory,
852861
)
853-
messages.append({"role": "user", "content": user_msg})
862+
messages: List[Dict[str, str]] = [
863+
system_message,
864+
{"role": "user", "content": user_msg},
865+
]
854866
before_obs = obs
855867

856868
if VERBOSE_TRACE:
@@ -872,7 +884,6 @@ def run_task(task_id: str, policy_memory: PolicyMemory) -> float:
872884
raw_action = "NOOP"
873885

874886
safe_action = _normalize_action(raw_action, obs, episode_memory)
875-
messages.append({"role": "assistant", "content": safe_action})
876887

877888
extracted = _extract_action_from_response(raw_action)
878889
was_cleaned = raw_action.strip() != extracted

server/dashboard.html

Lines changed: 40 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -276,6 +276,40 @@ <h1>🛡️ MissionCtrl</h1>
276276
document.getElementById('kpi-fp').className = 'kpi-value';
277277
}
278278

279+
function pickMetric(...values) {
280+
for (const v of values) {
281+
const n = Number(v);
282+
if (Number.isFinite(n)) return n;
283+
}
284+
return 0;
285+
}
286+
287+
function updateHallucinationKpis(stats, hasTasks) {
288+
const drEl = document.getElementById('kpi-detection');
289+
const fpEl = document.getElementById('kpi-fp');
290+
291+
if (!hasTasks) {
292+
drEl.textContent = '—';
293+
drEl.className = 'kpi-value';
294+
fpEl.textContent = '—';
295+
fpEl.className = 'kpi-value';
296+
return;
297+
}
298+
299+
const totalInjected = pickMetric(stats.total_injected, stats.injected, stats.num_injected);
300+
const totalCaught = pickMetric(stats.total_caught, stats.true_positives, stats.caught);
301+
const totalFlags = pickMetric(stats.total_flags, stats.flags);
302+
const falsePositives = pickMetric(stats.false_positives, stats.false_positive_count, stats.fp);
303+
304+
const dr = totalInjected > 0 ? (totalCaught / totalInjected) * 100 : 100;
305+
drEl.textContent = dr.toFixed(0) + '%';
306+
drEl.className = 'kpi-value ' + (dr >= 85 ? 'good' : dr >= 50 ? 'warn' : 'bad');
307+
308+
const fpRate = totalFlags > 0 ? (falsePositives / totalFlags) * 100 : 0;
309+
fpEl.textContent = fpRate.toFixed(0) + '%';
310+
fpEl.className = 'kpi-value ' + (fpRate <= 10 ? 'good' : fpRate <= 25 ? 'warn' : 'bad');
311+
}
312+
279313
// ── Dashboard Update ─────────────────────────────────────────────────────
280314

281315
function updateDashboard(state, history) {
@@ -309,6 +343,10 @@ <h1>🛡️ MissionCtrl</h1>
309343
const doneCount = tasks.filter(t => t.status === 'DONE').length;
310344
document.getElementById('kpi-tasks').textContent = `${doneCount} / ${tasks.length}`;
311345

346+
// Keep detection/FP KPIs live during the episode.
347+
const liveStats = state.hallucination_stats || {};
348+
updateHallucinationKpis(liveStats, tasks.length > 0);
349+
312350
// Task table
313351
const tbody = document.getElementById('task-table');
314352
if (tasks.length > 0) {
@@ -368,17 +406,9 @@ <h1>🛡️ MissionCtrl</h1>
368406
}
369407
document.getElementById('score-bars').innerHTML = barsHtml;
370408

371-
// Detection KPIs
409+
// Prefer final stats after episode completion.
372410
const stats = sb.hallucination_stats || {};
373-
const dr = stats.total_injected > 0 ? ((stats.total_caught / stats.total_injected) * 100) : 100;
374-
const drEl = document.getElementById('kpi-detection');
375-
drEl.textContent = dr.toFixed(0) + '%';
376-
drEl.className = 'kpi-value ' + (dr >= 85 ? 'good' : dr >= 50 ? 'warn' : 'bad');
377-
378-
const fpRate = stats.total_flags > 0 ? ((stats.false_positives / stats.total_flags) * 100) : 0;
379-
const fpEl = document.getElementById('kpi-fp');
380-
fpEl.textContent = fpRate.toFixed(0) + '%';
381-
fpEl.className = 'kpi-value ' + (fpRate <= 10 ? 'good' : fpRate <= 25 ? 'warn' : 'bad');
411+
updateHallucinationKpis(stats, tasks.length > 0);
382412

383413
// Incidents
384414
if (stats.total_injected > stats.total_caught || stats.false_positives > 0) {

0 commit comments

Comments
 (0)