v57: audit Synta requirements, recover owned CPU startup and journal real compute decisions

Author: Mantiee

SYNTA_REQUIREMENTS_AUDIT.md

@@ -0,0 +1,57 @@
+# Synta: requirement and verification audit, v57
+
+This audits the visible conversation. "Implemented" means code exists and the
+listed checks pass, not that it has run successfully on the operator's computers.
+Hardware-dependent checks are separate. No real orders, spending, paid APIs,
+account creation or sales are authorized by this implementation.
+
+| Requirement | Code / checks | Result and boundary |
+| --- | --- | --- |
+| Name Synta; distinguish user/master/controller | `mission_chat.py`, `tests/test_v100_chat_backend.py` | Implemented; controller capability/status answers are identified, not claimed as model generation. |
+| Chat answers; finite timeouts and no endless queue retries | `mission_chat.py`, chat/backend tests | Implemented; ordinary questions use one generation, identity probes bounded, connection retries finite, incomplete replies fail explicitly. An offline backend cannot supply a genuine model answer. |
+| Master executes requests instead of only writing plans | `research_tools.py`, `mission_chat.py`, action-object/resource tests | Named validated tools and receipts; plan-only implementation replies do not count as execution. Model tool choice is not guaranteed. |
+| Goal editable by explicit chat request | `chat_goals.py`, `planning.py`, chat-goal tests | Implemented; old experiments preserve their original goal; fetched content cannot change the operator goal. |
+| Dashboard accessible over LAN and updated automatically | `tools/serve-v100-dashboard.py`, `dashboard_editor.py` | Read-only host service with trusted renderer and validated passive guest layout. Host/guest publication hashes must match. |
+| Master edits dashboard without pasting HTML into chat | `dashboard_editor.py`, dashboard tests | Scoped read/write/repair tools; source read is separate from publication validation. Guest scripts remain disallowed; host supplies refresh. |
+| Clear resource, agent, plan, job and evidence sections | Dashboard renderer, activity-view tests | Implemented cards/timelines. No visual browser binary here, so aesthetic acceptance is not certified. |
+| Reading position and expansion survive all refresh paths | Transactional/keyed renderer, dashboard Node tests | Same DOM reader/text nodes and scroll offsets retained across repeated refreshes; live browser on operator machine remains an acceptance check. |
+| Show actions and results of controlled tools | `activity.py`, `research_tools.py`, `activity_browser.py` | Correlated start/result/error events and paged journal. Redaction, bounds and omitted secrets are intentional. This is not an OS-wide syscall recorder. |
+| Show each model's returned text/token measurements | `llamacpp.py`, `remote_helper.py`, `streaming.py`, streaming tests | Returned fragments and actual usage counters, with model/device/request provenance. Missing counters are unknown. Not every backend exposes raw token IDs. |
+| See model reasoning | Opt-in local returned-trace capture | Only reasoning actually returned by owned local models can be recorded. Unavailable internal reasoning is not fabricated or inferred from GPU usage. |
+| GPU/CPU utilization is distinct from actual useful work | Activity/resource cards; job receipts | Implemented. Readiness, loaded VRAM and idle heartbeat do not prove a task was executed. |
+| RTX helper survives Windows logon and recovers | `install-rtx3090-autostart.ps1`, guardian/firewall policies | Current-user logon task, existing guardian/game guard retained. Matching narrowly scoped firewall rules reused, unrelated rules unchanged. Windows runtime test required. |
+| Visible PowerShell monitor | `watch-rtx3090-helper.ps1`, `Synta-Helper-Monitor` task | Separate visible read-only window at logon; closing it does not stop compute. LAN binding corrected. |
+| Owned Windows CPU worker autostart | `install-owned-compute-autostart.ps1` | Added; existing authenticated mailbox/settings recovered; credentials/share unchanged; running worker preserved, new revision used after it exits/logon. CPU child remains 2 threads / 4 GiB. |
+| 50% helper target consistently | RTX autostart and `start-windows-lab.ps1` | Both use 50% active wall-time pacing, not a hard board utilization/power/temperature cap. Exact installed CPU dependencies avoid repeated network resolution. |
+| Use Windows RAM/disk/CPU and RTX | Owned mailbox, distributed compute, remote helper | Worker-local bounded jobs and safe artifact transfer. RAM/VRAM is not pooled into Debian; available hardware need not run dummy work to raise utilization. |
+| Automatically allocate useful owned CPU work | `goal_compute.py`, dispatch tests | A/B tasks queued for fresh idle workers when enough verified active-goal outcomes exist. No valid outcomes means an explicit blocker. |
+| Goal-oriented ML from newly discovered signals | `goal_learning.py`, `goal_compute.py`, `continuous.py` | Precommitted numeric signal/target forecasts, observed later outcomes, source-disjoint splits, verified master examples plus separate tiny CPU baselines. Tools support operator directions and model proposals. Arbitrary goals need an independently verifiable task/label definition. |
+| Social/news/filings evidence linked to outcomes | Public-source/JSON feed registration, `paper_agents.py`, goal observation tools | Available primitives; a measured association does not prove causality or profit. No promise of universal social-feed access or discovery of an income edge. |
+| Compare lawful opportunities, sports, crypto, equities | Research tools, provider registry, paper ledger | Research and bounded paper simulations; execution requires valid fees, feeds and domain settlement rules. Stale/unavailable sources remain blockers, not fabricated substitutes. |
+| Scalping/news/arbitrage and high leverage | Earlier assistant-generated plans | Not falsely certified as implemented profitable algorithms. These names came from model replies; placeholders and promises are not executable evidence. Leveraged live trading remains outside the authorized goal. |
+| Retention experiments including capacity growth | `retention.py`, `capacity_growth.py`, training/distillation, actual CPU PEFT tests | Replay, KL, reference-gradient projection, Fisher/drift constraints and optional adapter growth; residual-column/embedding extension experiments. Activation requires independent finite gates. No universal zero-forgetting guarantee or claim every published method was implemented. |
+| Prior weights/checkpoints and A/B histories survive | Protection, branch-lineage, activation/rollback tests | Implemented; candidates do not overwrite immutable originals. Tiny worker weights are not automatically substituted for master weights. |
+| Honest training progress and optimizer counters | `mission_evidence.py`, training-health tests | Attempted/global steps, completed optimizer updates and accepted versions remain separate; missing means unknown. GUI readiness is not a prerequisite for native inference/training. |
+| End-to-end task execution and acceptance decision | Real child-process goal test and independent local tensor validation | Observations -> frozen job -> actual optimizer -> safe weights -> host reload -> locally validated/rejected decision. Hardware sensors are simulated in sandbox; old goal labels cannot enter a new goal pool. Master activation separately tested through quality/retention/goal gates; no actual V100 training here. |
+| Code size / number of changed lines | Git tracked source/diff history | Reproducible with Git; temporary test directories are excluded. Line count is not a completion or quality measure. |
+
+## Hardware acceptance still required
+
+After installing the pinned Debian and Windows revisions, verify a real model
+answer, stable browser reading position, fresh Windows CPU heartbeat plus actual
+job receipt, RTX inference logs, and a logon recovery. Verify GPU training and
+accepted weights only from new run artifacts and independent gate results.
+Neither screenshots nor passing mocked transport tests establish these facts.
+
+## Operator goal is retained
+
+The financial goal is not replaced by arithmetic, dashboard maintenance or a
+small forecast task. Arithmetic is an auxiliary experiment. Failed trading tests,
+losses and rejected updates remain evidence. No result here establishes income.
+
+Source-size snapshot: 46130 physical lines in 190 tracked/new Python, PowerShell and shell files under `rlm/v100`, `tools` and `tests`. Includes comments/tests, excludes temporary files.
+
+Verification result: full test suite 1028 passed, 65 skipped, no deselections;
+ruff and upgrade shell checks pass. Windows startup policy has static contract
+checks, not a Windows runtime execution. Real CPU child-training/tensor-validation
+integration is included with simulated resource readings.

V100_AUDIT.md

@@ -670,3 +670,30 @@ deselected; one existing PEFT warning. Ruff, diff and upgrade shell checks pass.
 Tests cover the actual question route (one generation, bounded identity check,
 no CPU cold start/tool loop), incomplete output rejection and queue advancement
 after terminal connection failures.
+
+## v57: complete requirement audit, CPU startup and compute action journal
+
+`SYNTA_REQUIREMENTS_AUDIT.md` maps the conversation to concrete modules/checks and
+separates implemented mechanisms, bounded experiments, hardware acceptance and
+unavailable guarantees. The Windows helper installer now includes a limited
+current-user CPU autostart task and a separate visible read-only monitor. Existing
+mailbox configuration/running worker is preserved, no credentials/share are
+created, and the monitor uses the actual LAN binding. Both Windows helper paths
+use the operator's 50 percent active-time target. Installed exact CPU dependencies
+are checked before resolving packages over the network.
+
+Owned-compute state transitions are journaled once with job IDs, artifact paths,
+errors and independent validation decisions. Remote metrics remain untrusted.
+The real subprocess integration formerly deselected for lack of /proc is now
+exercised with explicit simulated sensor readings; training/tensor reload remain
+real. A new integration uses precommitted goal observations all the way through
+worker training and independent host validation, preserving goal provenance.
+Windows Scheduled Task execution and real browser/GPU operation still require
+operator hardware. This audit does not replace those checks with assertions.
+
+Validation: full suite 1028 passed, 65 skipped, no deselected tests; one existing
+PEFT warning. Ruff checks over `rlm` and `tests`, diff whitespace checks and upgrade
+shell syntax pass. Goal/worker integrations perform actual CPU optimizer updates
+and safe-tensor subprocess reload, with explicit simulated resource sensors.
+PowerShell execution is not available in this sandbox; startup policy checks do
+not certify Windows task registration/execution.

pyproject.toml

@@ -3,7 +3,7 @@ name = "rlms"
 authors = [
     {name = "Alex Zhang", email = "altzhang@mit.edu"},
 ]
-version = "0.1.3+v100.56"
+version = "0.1.3+v100.57"
 description = "Recursive Language Models."
 readme = "README.md"
 license = "MIT"

rlm/v100/distributed_compute.py

@@ -10,12 +10,39 @@
 import uuid
 from pathlib import Path
 
+from rlm.v100.activity import ActivityLog
 from rlm.v100.common import atomic_json
 from rlm.v100.compute_kernel import validate
 from rlm.v100.compute_worker import read_json
 from rlm.v100.protection import file_hash
 
 
+def record_state(root: Path, path: Path, state: dict) -> None:
+    """Journal observable job transitions once, retaining the full state artifact."""
+    before = read_json(path) if path.exists() else {}
+    atomic_json(path, state)
+    fields = ("state", "worker", "attempts", "validation_job", "local_report_sha256")
+    if any(before.get(key) != state.get(key) for key in fields):
+        job_path = path.parent / "job.json"
+        experiment_goal = (
+            read_json(job_path).get("provenance", {}).get("goal_id") if job_path.exists() else None
+        )
+        ActivityLog(root, state.get("branch", "controller"), "owned-compute").write(
+            "training",
+            "compute-state-changed",
+            {
+                "job_id": path.parent.name,
+                "experiment_goal_id": experiment_goal,
+                "previous_state": before.get("state"),
+                **{key: state[key] for key in fields if key in state},
+                "state_file": str(path),
+                "detail": state.get("detail"),
+                "weights_promoted": state.get("weights_promoted", False),
+                "scope": "Observed coordinator transition; remote metrics remain untrusted until host validation",
+            },
+        )
+
+
 def configure(root: Path, mailbox: Path) -> dict:
     from rlm.v100.mission import status
 
@@ -160,7 +187,8 @@ def queue_job(
     atomic_json(folder / "job.json", job)
     shutil.copyfile(kernel, folder / "kernel.py")
     (folder / "kernel.py").chmod(0o444)
-    atomic_json(
+    record_state(
+        root,
         folder / "state.json",
         {
             "state": "queued",
@@ -209,7 +237,7 @@ def cancel(root: Path, identity: str) -> dict:
     path = root / "research/compute-jobs" / identity / "state.json"
     state = read_json(path)
     state.update(state="cancelled")
-    atomic_json(path, state)
+    record_state(root, path, state)
     atomic_json(mailbox_path(root) / "closed" / (identity + ".json"), {"state": "cancelled"})
     return state
 
@@ -225,7 +253,7 @@ def tick(root: Path) -> None:
                 try:
                     scheduled = schedule(root, state["branch"], "compute-audit", identity, 0)
                     state["validation_job"] = scheduled["id"]
-                    atomic_json(path, state)
+                    record_state(root, path, state)
                 except ValueError:
                     # A full bounded queue defers validation, never a completed import.
                     pass
@@ -252,7 +280,7 @@ def tick(root: Path) -> None:
             if receipt is None:
                 if time.time() - lease["heartbeat"] < 300:
                     state.update(state="running", worker=lease["worker"])
-                    atomic_json(path, state)
+                    record_state(root, path, state)
                     continue
                 raise ValueError("Compute worker lease expired")
             if receipt.get("nonce") != lease["nonce"] or receipt.get("job_id") != identity:
@@ -286,7 +314,7 @@ def tick(root: Path) -> None:
                 weights_promoted=False,
                 detail="Remote metrics untrusted; queued local safe-tensor validation",
             )
-            atomic_json(path, state)
+            record_state(root, path, state)
             atomic_json(mailbox / "closed" / (identity + ".json"), {"state": "imported"})
         except (ValueError, KeyError, TypeError, OSError) as error:
             attempts = state["attempts"] + 1
@@ -295,7 +323,7 @@ def tick(root: Path) -> None:
                 attempts=attempts,
                 detail=str(error)[:400],
             )
-            atomic_json(path, state)
+            record_state(root, path, state)
             if attempts >= 3:
                 atomic_json(mailbox / "closed" / (identity + ".json"), {"state": "failed"})
             claim = mailbox / "claims" / identity
@@ -383,5 +411,5 @@ def validate_locally(root: Path, identity: str) -> dict:
         weights_promoted=False,
         detail="Independent host loss evaluation of safe tensors; not a master promotion or proof of the user goal",
     )
-    atomic_json(folder / "state.json", state)
+    record_state(root, folder / "state.json", state)
     return state

tests/test_v100_distributed_compute.py

@@ -167,11 +167,24 @@ def test_owned_worker_executes_pinned_child_and_host_validates_weights(tmp_path,
 
     mailbox, identity = configured(tmp_path, monkeypatch)
     monkeypatch.setattr(compute_worker, "available", lambda: (True, "Test CPU host ready"))
-    available_memory = psutil.virtual_memory()
     monkeypatch.setattr(
         psutil,
         "virtual_memory",
-        lambda: available_memory._replace(available=12 * 2**30),
+        lambda: type("Memory", (), {"available": 12 * 2**30})(),
+    )
+    # The sandbox has no /proc. Training and validation are real subprocesses;
+    # resource sensor readings are simulated, not asserted as hardware validation.
+    monkeypatch.setattr(
+        psutil,
+        "Process",
+        lambda pid: type(
+            "Process",
+            (),
+            {
+                "memory_info": lambda self: type("RSS", (), {"rss": 100 * 2**20})(),
+                "children": lambda self, recursive: [],
+            },
+        )(),
     )
     path, lease = compute_worker.claim(mailbox, "actual-cpu")
     compute_worker.execute(mailbox, path, lease, Path(compute_kernel.__file__))
@@ -210,3 +223,22 @@ def full(*args):
     monkeypatch.setattr(drones, "schedule", actual)
     distributed_compute.tick(tmp_path)
     assert drones.inspect(tmp_path)[0]["kind"] == "compute-audit"
+
+
+def test_compute_transition_journal_is_complete_and_does_not_repeat_polling(tmp_path):
+    path = tmp_path / "research/compute-jobs" / ("a" * 24) / "state.json"
+    state = {"state": "queued", "branch": "A", "attempts": 0}
+    distributed_compute.record_state(tmp_path, path, state)
+    distributed_compute.record_state(tmp_path, path, state)
+    state.update(state="running", worker="windows-cpu")
+    distributed_compute.record_state(tmp_path, path, state)
+    state.update(state="failed", attempts=3, detail="Worker failed")
+    distributed_compute.record_state(tmp_path, path, state)
+    events = [
+        json.loads(line)
+        for p in tmp_path.glob("research/logs/activity/*/timeline.jsonl")
+        for line in p.read_text().splitlines()
+    ]
+    assert [e["payload"]["state"] for e in events] == ["queued", "running", "failed"]
+    assert all(e["payload"]["job_id"] == "a" * 24 for e in events)
+    assert events[-1]["payload"]["detail"] == "Worker failed"