Skip to content

Commit e97b304

Browse files
MantieeMantiee
committed
Preflight updater executables before stopping mission and recover preserved stopped profiles
1 parent 9476995 commit e97b304

4 files changed

Lines changed: 219 additions & 1 deletion

File tree

‎SYNTA_V82.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Synta v82: updater recovery after exit 127
2+
3+
Observed host failure: transient systemd updater preserved the profile and stopped the mission, then exited 127. Package remained v78. Exit 127 means a command was unavailable; bare `uv` in the updater and the transient service's PATH are the leading diagnosis, not a proven full journal trace.
4+
5+
Changes in `tools/update-synta-v82.sh`:
6+
- Add `$HOME/.local/bin` to PATH and resolve executable `uv` to an explicit path. Run `uv --version` and check Python/git/curl/systemctl before any service change.
7+
- Permit recovery of a previously stopped updater, using its preserved profile under `research/availability-recovery-update` after validating its path. Do not require the service to be active in this recovery case.
8+
- Save an invocation-specific restore marker before requesting a stop. On subsequent failure, if the mission is stopped, submit the preserved profile to `supervisor.install`; if already running, leave it alone. Preserve original error status.
9+
- A recovery attempt is not a package rollback. If package imports or supervisor restart fail, print an explicit error and require journal inspection.
10+
- Retain training exclusivity guard, prior v79-v81 fixes, model qualification tools and fresh dashboard verification. Correct the installed-version success message.
11+
12+
`tests/test_v100_updater_recovery.py`: real Bash execution with a simulated host proves that missing uv never invokes the mission-stop Python, that ~/.local/bin/uv works without an interactive shell, and that failed package installation calls the recovery path and preserves its error status.
13+
14+
Validation: two new recovery tests passed; Bash syntax and all embedded Python heredoc syntax checked; Ruff/diff checks passed. Prior v81 full Python regression: 1316 passed, 65 skipped, one existing PEFT warning. Live host install/restart still requires receipts. This release does not claim a new model was promoted, a GPU speedup, income, or a repaired Windows LAN path.

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ name = "rlms"
33
authors = [
44
{name = "Alex Zhang", email = "altzhang@mit.edu"},
55
]
6-
version = "0.1.3+v100.81"
6+
version = "0.1.3+v100.82"
77
description = "Recursive Language Models."
88
readme = "README.md"
99
license = "MIT"

‎tests/test_v100_updater_recovery.py‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
import os
2+
import subprocess
3+
from pathlib import Path
4+
5+
SCRIPT = Path(__file__).parents[1] / "tools/update-synta-v82.sh"
6+
7+
8+
def host(tmp_path):
9+
root = tmp_path / "ai-v100"
10+
(root / "research").mkdir(parents=True)
11+
(root / "venvs/v100-continual/bin").mkdir(parents=True)
12+
(root / "env.sh").write_text(f'export AI_V100_ROOT="{root}"\n')
13+
environment = {**os.environ, "HOME": str(tmp_path), "PATH": "/usr/bin:/bin"}
14+
python = root / "venvs/v100-continual/bin/python"
15+
return root, environment, python
16+
17+
18+
def executable(path, code):
19+
path.parent.mkdir(parents=True, exist_ok=True)
20+
path.write_text("#!/bin/bash\n" + code)
21+
path.chmod(0o755)
22+
23+
24+
def test_missing_uv_cannot_stop_mission(tmp_path):
25+
root, environment, python = host(tmp_path)
26+
executable(python, f'touch "{root}/python-called"\n')
27+
result = subprocess.run(
28+
["/bin/bash", str(SCRIPT), "a" * 40], env=environment, capture_output=True, text=True
29+
)
30+
assert result.returncode == 127
31+
assert "before any service change" in result.stderr
32+
assert not (root / "python-called").exists()
33+
34+
35+
def test_noninteractive_local_uv_is_resolved_and_failed_install_recovers(tmp_path):
36+
root, environment, python = host(tmp_path)
37+
uv = tmp_path / ".local/bin/uv"
38+
executable(uv, 'if [[ "$1" == "--version" ]]; then echo uv-test; exit 0; fi\nexit 9\n')
39+
executable(
40+
python,
41+
f'''input="$(/bin/cat)"
42+
if [[ "$input" == *"Preserved recovery profile unavailable"* ]]; then
43+
touch "{root}/recovery-attempted"
44+
exit 0
45+
fi
46+
printf '%s' '{root}/research/profile.json' > "$SYNTA_RESTORE_MARKER"
47+
exit 0
48+
''',
49+
)
50+
result = subprocess.run(
51+
["/bin/bash", str(SCRIPT), "a" * 40], env=environment, capture_output=True, text=True
52+
)
53+
assert result.returncode == 9
54+
assert "uv-test" in result.stdout
55+
assert (root / "recovery-attempted").exists()
56+
assert not list((root / "research").glob(".update-v82-restore.*"))

‎tools/update-synta-v82.sh‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
#!/usr/bin/env bash
2+
# Install continuous goal work; preserve configuration.
3+
set -euo pipefail
4+
REV="${1:?Pass the pinned 40-character commit}"
5+
[[ "$REV" =~ ^[0-9a-f]{40}$ ]] || { echo 'Invalid revision' >&2; exit 1; }
6+
source "$HOME/ai-v100/env.sh"
7+
PY="$AI_V100_ROOT/venvs/v100-continual/bin/python"
8+
export SYNTA_GOAL_WORK_REV="$REV"
9+
export PATH="$HOME/.local/bin:$PATH"
10+
UV="$(command -v uv || true)"
11+
if [[ -z "$UV" || ! -x "$UV" ]]; then
12+
echo 'uv executable unavailable; preflight failed before any service change.' >&2
13+
exit 127
14+
fi
15+
"$UV" --version
16+
for COMMAND in git curl systemctl; do
17+
command -v "$COMMAND" >/dev/null || { echo "Required executable unavailable: $COMMAND; no service changed." >&2; exit 127; }
18+
done
19+
[[ -x "$PY" ]] || { echo 'Existing Synta Python unavailable; no service changed.' >&2; exit 127; }
20+
RESTORE_MARKER="$(mktemp "$AI_V100_ROOT/research/.update-v82-restore.XXXXXX")"
21+
export SYNTA_RESTORE_MARKER="$RESTORE_MARKER"
22+
restore_after_failure() {
23+
local failure=$?
24+
trap - ERR
25+
echo "Update failed with exit $failure. Checking preserved mission recovery." >&2
26+
if [[ -s "$RESTORE_MARKER" ]]; then
27+
if ! "$PY" <<'RECOVER'
28+
import json
29+
import os
30+
from pathlib import Path
31+
from rlm.v100.mission import status
32+
from rlm.v100.supervisor import install
33+
root = Path(os.environ['AI_V100_ROOT']).resolve()
34+
profile = Path(Path(os.environ['SYNTA_RESTORE_MARKER']).read_text().strip()).resolve()
35+
if not profile.is_relative_to(root / 'research') or not profile.is_file():
36+
raise SystemExit('Preserved recovery profile unavailable; inspect update journal')
37+
if status(root).get('running'):
38+
print('Mission is already running; recovery did not restart it', flush=True)
39+
else:
40+
print(json.dumps(install(root, profile), indent=2), flush=True)
41+
print('Preserved profile submitted to supervisor. Verify mission-status.', flush=True)
42+
RECOVER
43+
then
44+
echo 'Automatic mission recovery failed; inspect journal. Package rollback was not claimed.' >&2
45+
fi
46+
fi
47+
rm -f "$RESTORE_MARKER"
48+
exit "$failure"
49+
}
50+
trap restore_after_failure ERR
51+
52+
"$PY" <<'PY'
53+
import json
54+
import os
55+
import subprocess
56+
import time
57+
from importlib.metadata import version
58+
from pathlib import Path
59+
from rlm.v100.common import atomic_json, load_profile
60+
from rlm.v100.mission import status, stop
61+
62+
installed = version('rlms')
63+
if installed not in tuple(f'0.1.3+v100.{number}' for number in range(78, 83)):
64+
raise SystemExit(f'Targeted goal work update requires v78-v82; installed: {installed}')
65+
service = subprocess.run(['systemctl', '--user', 'is-active', '--quiet', 'v100-mission.service'])
66+
root = Path(os.environ['AI_V100_ROOT'])
67+
record = status(root)
68+
if not record.get('run'):
69+
raise SystemExit('No mission profile to preserve; no service changed')
70+
for line in subprocess.check_output(['ps', '-eo', 'pid=,args='], text=True).splitlines():
71+
if 'rlm.v100.cli' in line and any(' '+mode+' ' in line+' ' for mode in
72+
('train', 'mission-prepare', 'optimize-mtp', 'calibrate-training', 'test-mtp')):
73+
raise SystemExit('Exclusive experiment still runs; no service changed. '+line)
74+
if service.returncode == 0 and record.get('running'):
75+
source = Path(record['learning']['live_profile']) if record.get('learning', {}).get('live_profile') else Path(record['run']) / 'input-profile.json'
76+
else:
77+
checkpoint = root / 'research/availability-recovery-update/current.json'
78+
if not checkpoint.is_file():
79+
raise SystemExit('Stopped mission has no preserved updater profile; no service changed')
80+
source = Path(json.loads(checkpoint.read_text())['profile']).resolve()
81+
if not source.is_relative_to(root.resolve() / 'research/availability-recovery-update'):
82+
raise SystemExit('Preserved profile escaped updater folder; no service changed')
83+
print('Recovering previously stopped update from:', source, flush=True)
84+
profile = load_profile(source, root)
85+
folder = root / 'research/availability-recovery-update' / str(time.time_ns())
86+
folder.mkdir(parents=True)
87+
atomic_json(folder / 'before.json', record)
88+
atomic_json(folder / 'profile.json', profile)
89+
atomic_json(root / 'research/availability-recovery-update/current.json', {'profile': str(folder / 'profile.json')})
90+
print('Preserved profile:', folder / 'profile.json', flush=True)
91+
Path(os.environ['SYNTA_RESTORE_MARKER']).write_text(str(folder / 'profile.json'))
92+
print(stop(root), flush=True)
93+
subprocess.run(['systemctl', '--user', 'stop', 'v100-mission.service'], check=True, timeout=90)
94+
if status(root)['running']:
95+
raise SystemExit('Mission still running; package not changed')
96+
PY
97+
98+
export UV_HTTP_TIMEOUT=600 UV_HTTP_RETRIES=5
99+
"$UV" --no-config pip install --python "$PY" --no-deps --reinstall-package rlms \
100+
"rlms @ git+https://github.com/Mantiee/rlm.git@$REV"
101+
"$UV" --no-config pip check --python "$PY"
102+
"$PY" <<'PY'
103+
import json
104+
import os
105+
from importlib.metadata import version
106+
from pathlib import Path
107+
from rlm.v100.supervisor import install
108+
from rlm.v100.mission import status
109+
import time
110+
111+
root = Path(os.environ['AI_V100_ROOT'])
112+
if version('rlms') != '0.1.3+v100.82':
113+
raise SystemExit('Pinned revision is not the v82 updater recovery update; mission remains stopped')
114+
path = Path(json.loads((root / 'research/availability-recovery-update/current.json').read_text())['profile'])
115+
print(json.dumps(install(root, path), indent=2), flush=True)
116+
deadline = time.monotonic() + 30
117+
while time.monotonic() < deadline:
118+
record = status(root)
119+
if record.get('running'):
120+
print(json.dumps(record, indent=2), flush=True)
121+
break
122+
time.sleep(0.5)
123+
else:
124+
raise SystemExit('Supervisor installed, but mission did not start within 30 seconds; inspect v100-mission.service journal')
125+
PY
126+
curl -fL --retry 3 "https://raw.githubusercontent.com/Mantiee/rlm/$REV/tools/serve-v100-dashboard.py" -o "$AI_V100_ROOT/bin/serve-v100-dashboard.py.new"
127+
"$PY" -m py_compile "$AI_V100_ROOT/bin/serve-v100-dashboard.py.new"
128+
mv "$AI_V100_ROOT/bin/serve-v100-dashboard.py.new" "$AI_V100_ROOT/bin/serve-v100-dashboard.py"
129+
systemctl --user restart v100-dashboard.service
130+
systemctl --user try-restart synta-remote.service
131+
if ! "$PY" <<'PY'
132+
import json
133+
from rlm.v100.dashboard_health import wait
134+
import os
135+
from pathlib import Path
136+
print(json.dumps(wait(60, root=Path(os.environ['AI_V100_ROOT'])), indent=2))
137+
PY
138+
then
139+
systemctl --user status v100-dashboard.service --no-pager -l || true
140+
journalctl --user -u v100-dashboard.service -n 30 --no-pager || true
141+
exit 1
142+
fi
143+
echo 'v82 installed: mission restarted with its preserved profile. Goals, weights, policy, balances and archives retained. Reload dashboard once.'
144+
145+
curl -fL --retry 3 "https://raw.githubusercontent.com/Mantiee/rlm/$REV/tools/qualify-synta-models-debian.sh" -o "$AI_V100_ROOT/qualify-synta-models-debian.sh"
146+
147+
trap - ERR
148+
rm -f "$RESTORE_MARKER"

0 commit comments

Comments
 (0)