Repository navigation
Release/v3 - #124
Release/v3#124chandan25809-lyzr wants to merge 79 commits into
Conversation
A single static binary that runs one agent loop over a persistent shell: native tool calling with a text fallback, PTY sessions with end markers, output shaping with spill files, an environment snapshot, batched masking and compaction, time and loop guards, and an evidence-based stop check. Provider adapters for Anthropic, OpenAI, Google and OpenAI-compatible endpoints; per-family profiles in YAML; ATIF trajectories and counters written every turn. Also included: the Harbor shim and benchmark scripts for Terminal-Bench 2.1 (dev/held-out split, rounds, pricing, dashboard), the offline lesson learner with its leakage audit, the .claude/ project-file layers, the gitagent-sdk library API, a Landlock+seccomp sandbox, and the integrity audit script.
…timeout multipliers, task versions, Hub link) and a compliance strip per round; Harbor's per-trial log is no longer shown
…ks, not only the dev ones; badge on such rounds; running jobs listed as unfinished
…provisional while its errored trials are relaunched
….5, no lessons); relaunch logs kept
…itions of the published Terminal-Bench 2.1 run
…TS.md as fallback) ahead of the working instructions; model.preferred, runtime limits, model constraints and the skills list apply; schema problems are warnings; benchmark mode reads none of it
…AGENT_MODEL_BASE_URL outside benchmark mode, and model fallback (fallback_models, model.fallback) that switches on a fatal error or three transient errors in a row
…econd apart (fixes a timing-dependent test)
…acker) are accepted in tool lists, with argument translation for hooks; the model keeps seeing gitagent's own names
…wn in the system prompt, and the memory tool (load, save) appends to them, moves lines beyond max_lines to memory/archive/, and commits each save on its own; on for agent directories or with memory: true, never in benchmark mode
…he model can call; scripts get the arguments as JSON on stdin, http tools send them as body or query with ${VAR} headers, 120 s default timeout, arguments checked against the input schema; tool lists keep the agent's own tool names; never in benchmark mode
…and end, pre and post tool use, pre and post response, on_error) and the 1.x ones (pre_query, post_tool_failure, file_changed); JSON in, allow/block/modify out, 1.x fields and names kept for old scripts; a failing hook blocks unless fail_open, compliance hooks run even with hooks off; the init message lists every offered tool; never in benchmark mode
…y's mcp_servers option are connected once per session (stdio, streamable HTTP, the older SSE transport; ${VAR} filled at connect time; 30 s to connect and list); their tools are offered as <server>__<tool>; a server that fails is left out with a warning; mcp_server tools in tools/*.yaml call their server; never in benchmark mode
…t waits for a person (human_in_the_loop always or conditional with action_type, error_detected and token_count_above triggers; requires_confirmation tools; pre_review_required responses); the terminal asks on /dev/tty, the library takes an approver, and nobody to ask means denied; kill_switch stops the run between steps (.gitagent/KILL, or Query::kill), exit code 130; the OpenGAP compliance lines go into the prompt; never in benchmark mode
… OpenGAP schema (closed objects, types, ranges, patterns), the skills, tools and sub-agents it names, SOUL.md, hooks/hooks.yaml and its scripts, every tools/*.yaml and skills/*/SKILL.md, and with --compliance the regulatory rules (risk tier, FINRA, SR 11-7, segregation of duties, financial governance); findings name the file and line; exit 1 on errors
…th working instructions for talking with a person (prompts/conversation.md); on in the interactive session, with --conversation, and in the library's conversation option; final replies pass the pre_response hooks and review approval like task_complete; benchmark runs never use it
…ent libraries; initialize with options, callbacks and client tools, then user turns in one conversation (conversation mode by default) streamed as delta, assistant, tool_use, tool_result and turn_end messages; steer, abort, kill and end; callback requests for hooks (onSessionStart, preQuery, preToolUse, postToolUse, postResponse, onError), approvals and client tools with timeouts; 1.x tool names on request; sessions stored and resumed with resume: true. The loop gains a steering queue, caller-run tools, a reason on the kill switch, and resuming from stored messages
…he agent directory and the output directory are made absolute, since hooks, tools and the shell run in other directories
…eft out with a warning instead of stopping the run; a scripted model gets no fallbacks
…ogress instead of dropping the conversation; a type mismatch in agent.yaml is a warning (name, version and description still apply) and YAML slips like version: 1.0 are put right; runtime.timeout applies; a tool script outside the agent directory is refused like a hook script; a tool result's error: false is not an error; a malformed client tool in IPC initialize is refused instead of crashing; a steer message arriving as a turn ends becomes the next message; the terminal approver reads on one thread so a timed-out question cannot take the next answer; IPC callback entries are freed when a request is given up; IPC costs use the library's arithmetic; memory saves run git without blocking the runtime
…p (--env, the env option, or GITAGENT_ENV); model_override, temperature_override, timeout_override, env_vars for tool and hook scripts, compliance_mode (the agent must pass validate --compliance to start) and audit_logging apply, other settings are kept; validate checks config/*.yaml; never in benchmark mode
…ys_load go into the system prompt after the skills (as information, not instructions; non-text or very long ones are listed instead), the others are listed for the file tools; examples/*.md follow under an Examples section, scenarios/ stays out; validate checks the index; never in benchmark mode
…d into .gitagent/parent/); the child's agent.yaml is merged over the parent's with skills and tools joined, SOUL.md is the nearest, RULES.md are the parents' then the child's, DUTIES.md, hooks, knowledge, examples and config files are the nearest that has them, skills and tools are joined with the child shadowing by name, memory is never shared; chains are followed with cycle detection; gitagent install fetches a git parent and the dependencies (git clones at their version, local copies) into their mounts; validate checks both
… with its own prompt, skills, tools, hooks and memory (the parent's model, supervision and tool hooks apply when it has none of its own), agents/<name>.md runs on the parent's prompt with its instructions; the agents: map in agent.yaml names them with description and delegation triggers, without one every entry is offered; the prompt lists them and the delegation mode (auto, explicit, router); validate checks sub-agent directories too
…the session-start procedure; hooks/teardown.md runs as one last turn of the session, on its conversation and shell, before on_session_end (one-shot runs, the interactive session, repo mode, IPC and the library); a failed teardown is reported and the run's own result stands
…and -r, --pat and --session for repo mode (GIT_TOKEN read too), .env files in ~/.gitagent and the agent directory loaded before a run (never in benchmark mode), gitagent init to start an agent directory, and the interactive session's /help, /memory, /skills, /agents, /tools, /skill:<name> and /quit with a banner naming the agent; --sandbox and --voice say they are not available
…dings The results table and chart name the highest published Terminal-Bench 2.1 entry, Artificial Analysis's own harness with Claude Fable 5.1 at maximum effort, and link every source: Vals.ai, the official leaderboard at Snorkel, and Artificial Analysis. The two recordings are re-made from a small billing service: the console one shows a real fix (failing unit tests, the file read, an edit, passing tests, the diff) and the task board one scans the service's three source files for TODOs in parallel and checks them. Both play faster.
…pace A scripted model turn may carry `delay_ms`, so a demo shows the model thinking before it answers; the provider releases its lock before the wait, so parallel sub-agents are not held up. The README's recordings are re-made from a pseudo-terminal with keystrokes typed one at a time and real pauses: the prompt appears as it is typed, the thinking indicator runs while the model works, and the tool lines arrive at the speed they did.
bench/RESULTS.md stays on the benchmark machine, ignored by git; the README's results point at the headline run's public job logs on the Harbor hub instead.
The MIT licence from main, a contributing guide for the Rust workspace (build, tests without keys, the four checks, the benchmark guard, profile switches, 1.x compatibility, the crates), Contributing and License sections in the README, and the install command now fetches the script from main, since no v2 branch exists.
…troubleshooting The workspace version is 3.0.0: the TypeScript packages already published 2.x, so the Rust binary follows them as 3, matching the Python and TypeScript clients. The README opens with why gitagent (the agent is a git repository) and what 3 is, then gains an Architecture section (the agent folder and an annotated agent.yaml), SDKs for any language over the IPC protocol (Python, TypeScript, a complete client in bash, the messages both ways, and the Rust crate), a guide for coming from the TypeScript gitagent, and Troubleshooting. The recordings and the results chart say 3.
TASKS.md, BLUEPRINT.md and AGENTS.md say gitagent 3, and no longer say that the TypeScript gitagent's formats are dropped: they are accepted for compatibility, only its code is not reused. The results chart has a wider label column, so the longest label is no longer cut off.
The blueprint now describes gitagent 3 as built: agents as git folders, one loop for every entry point, the requirements for the agent and for the loop, a section on why the benchmark runs through Harbor and how gitagent runs under it, the architecture and one run start to finish, the design decisions with their reasons (compatibility, hooks, IPC, parallel plans, schedules, the terminal as the interface, the benchmark guard), the product surface, the code layout, the configuration surfaces, the interfaces, observability, testing and the open questions. The OpenAI profiles' cache key is "gitagent".
…d in TASKS.md (the milestone checklist, all done) and docs/DEVIATIONS.md (differences from the old blueprint) are removed: the new blueprint describes what was built and docs/HOW-IT-WORKS.md the mechanics. docs/LEARNING.md is folded into HOW-IT-WORKS's learning section. docs/INTEGRITY.md keeps the rules and the audit without the details of how tasks were divided. The README's development list points at the blueprint, how it works and integrity.
Every flag of the binary, grouped by what it is for, and the six subcommands, taken from gitagent --help.
The loop detector, the stop check (task mode) and the time budget are on in the built-in profiles, and Claude clears old tool results server-side. The snapshot stays a benchmark setting. The built-in config sets no time limit: the console runs without one, and -p, schedules and the SDKs fall back to 30 minutes. Docs explain the defaults and how the benchmark passes each task's time limit.
- A workflow step that calls a declared or MCP tool itself now passes the same checks as a model's call: kill switch, hooks, spending and role rules, approval. - Spending: a payment is counted when it is let through and taken off if its call fails, under a lock, so payments running at once cannot pass the monthly cap together. An amount that cannot be read blocks the payment instead of skipping the rules. - Hooks that fail closed (compliance, fail_open: false) block on output that is not an answer; plain hooks still allow, as in 1.x. - Plan tasks and the reconciling step get the time actually left. - A steps: mapping in a workflow runs in file order. - The audit log records spending and role blocks. - Schedules: a job never runs twice for the same time, and the local offset is read each time round, so daylight-saving changes are followed. - Starting an agent in an existing repository commits only the files it created, never the user's uncommitted work. - The console help no longer lists the removed web page.
…only - Sub-agents are told when 80% of their turns are used and again before the last one, so one that runs out still hands back a summary (turn_warning; also on for the lead in the built-in profiles). - The subagent tool and plan tasks take max_turns (up to 100), so the lead can size each task's limit. - A task that runs out of turns is marked partial: on the task board, in the results the reconciler and the lead see, in plan_partial_tasks. - Plan tasks and sub-agent calls run in a rolling window: the next starts as soon as one finishes; task start and end are reported as they happen. - The task board cuts lines to the terminal width, so a long task name no longer makes every redraw add lines; it shows running and done counts. - The welcome box fits narrow terminals. - The /tests write guard applies to benchmark runs only; it refused ordinary commands that mention /tests in a pattern. - Status lines say "model calls" instead of "turns".
- Providers report a refusal for rate or capacity reasons (HTTP 429,
overloaded) as ProviderError::RateLimited, apart from other transient
errors. It is retried with the same backoff.
- One Throttle per session: max_parallel is the ceiling; a rate limit
halves the window (never below one); ten calls in a row that go through
give a slot back. Plan tasks and sub-agent calls start new work only
while the window has room; running work is never stopped.
- Counters provider_retries and rate_limit_retries; a guard event of kind
rate_limited; a plan event window once a rate limit has narrowed it,
which the task board shows ("slowed to 2 at a time after 3 rate limits").
- The score is shown as 93.0% ±0.8, the run-to-run standard error across
the five runs, the measure the official leaderboard uses; the across-tasks
interval stays, labelled as such.
…e repository The dev/held-out split, the round history, the published scores, the lesson store and the rerun logs stay on the benchmark machine. The profiles, the public job links, the prices and the smoke fixture stay in bench/, where the guard test and the reproduction steps use them.
One static binary that runs the same agent folders, hooks, schedules and SDK code as the TypeScript gitagent, scored 93.03% ±0.8 on Terminal-Bench 2.1 with Claude Opus 5.5, and adds a terminal console, parallel sub-agents with a live task board, schedules, and SDKs for Python, TypeScript and any other language over a line protocol.
…ion records - `gitagent install` and `gitagent plugin` keep to the agent's folder: a dependency's `mount` must be a relative path inside it, names must not be paths, and a folder that is not an installed agent is replaced only with --force. Before, `mount: ../x` or an absolute path deleted that folder. - Spending: an amount above a trillion dollars is unreadable and so blocked, the monthly sum uses checked addition, and a state.json that is not a JSON object no longer crashes the run. Before, a huge amount wrapped the sum and passed the cap. - The approval prompt shows every argument in full, with escape codes, carriage returns and other control characters made visible; an argument over 4,000 characters keeps its start and end and says how much is left out. - Each console session writes its records to its own folder (console-<time>-<pid>/task-<n>), so a new session no longer overwrites an earlier one's. - masks_applied counts each cleared tool result once; the provider reports every cleared result again on each call.
- README: the results note keeps the run-to-run ±0.8 and drops the across-tasks range, which moves to docs/INTEGRITY.md with an explanation. - Cache reads are 91% of input (72.0M of 79.3M tokens), not 92%, and on Opus 5.5 a cache read costs a twentieth of normal input, not a tenth. - Clearing old tool results invalidates the provider's cache from the cleared point; README and HOW-IT-WORKS now say so, and why clearing is batched.
shreyas-lyzr
left a comment
There was a problem hiding this comment.
Backward-compatibility review: v3 (Rust) vs 1.x TypeScript main
I diffed this branch against main across five dimensions (CLI, config schema, agent-folder conventions, SDK API, tools + subsystems), checking each claim in the Compatibility section against the actual code in both trees.
Overall: ≈ 83% of the 1.x surface is preserved. An existing agent folder (config + skills + hooks + declared tools + MCP + schedules) loads and runs — the gaps are feature breadth and a few silent behavioral changes, not core loading.
| Dimension | Compat | Notes |
|---|---|---|
| Agent-folder conventions | ~95% | All 9 conventions load from identical paths; explicit 1.x hook/compliance shims |
| Config schema | ~93% | 23/26 manifest fields honored; config/<env>.yaml + .env full parity |
| SDK API | ~80% | query()/tool() preserved by name; regressions below |
| Tools + subsystems | ~74% | Core tools / MCP / schedules / audit / cost carry over cleanly; some drops |
| CLI surface | ~70% | Core run flags preserved; exit-code values changed |
Intentional drops — verified accurate 👍
The Compatibility section checks out for: remote --sandbox (now local Linux-only), plugin entry JavaScript (its files still load), and the 1.x OpenTelemetry-SDK adapter (→ post-run OTLP export).
Undocumented regressions — please document or fix before merge
Not mentioned in the PR description. The
⚠️ Sampling constraints silently ignored.model.constraints.top_p,top_k,stop_sequencesare parsed (crates/loader/src/manifest.rs:99-101) and validated (validate.rs:170-172) but never applied — runtime only honorstemperature+max_tokens(lib.rs:408-419). Same in the SDK. No warning. → apply them, or havevalidatewarn.⚠️ CLI exit codes changed. 1.x returned1for every error; v3 returns2for usage errors (crates/cli/src/main.rs:327-329),3provider,130interrupt. Any CI/wrapper keying on exit1for usage errors will misread. Worth a migration note.⚠️ SDKtool()contract changed. The handler drops theAbortSignalparameter, and the return shape goes from{text, details}→{content, isError}(clients/typescript/src/index.ts:69,answer()~434-437). A 1.x tool returning{text: "…"}is now passed through as-is instead of mapped to content — silently wrong model input. (Plain-string returns remain safe in both versions — worth stating that explicitly.)⚠️ Plugin settings no longer reach plugin script tools. Declared script tools get env only fromspec.env_config.env_vars(crates/core/src/lib.rs:940); nothing plumbsPlugin.configthrough. This is noted as a known gap forentry, but it also affects file-based plugin tools.task_tracker→todois a downgrade, not a rename. 1.x was a searchable/semantic task memory; v3todois a replace-whole-list checklist (crates/tools/src/lib.rs:119-141).- Online skill-learning removed.
skill_learner+ skill-confidence reinforcement (SkillStats/adjustConfidence) are gone (crates/loader/src/v1.rs:12,36); replaced by offline lesson distillation (crates/learn). The README's "learning between runs" should clarify it's offline-only now. - Compliance prompt content dropped. Free-text
regulatory-map.yaml/validation-schedule.yamlare existence-checked only; their content is no longer injected into the system prompt (v3 builds constraints from structured fields,compliance.rs:283-381). - Interpreter default changed. A
.shdeclared tool or hook with no explicitruntimenow runs underbash, notsh(declared.rs:265-296). Hook timeout default also moved 10s → 30s. - macOS has no sandbox. Landlock/seccomp are Linux-only (
crates/sandbox/src/lib.rs:53-58); the old remote sandbox was cross-platform, so macOS users lose sandboxing entirely. gitagent.yamlis strict. It usesdeny_unknown_fieldsand hard-errors on unknown keys (crates/loader/src/config.rs:10,lib.rs:302-303), andvalidatedoesn't check it — inconsistent with the "warn, never error" posture everywhere else.
Bottom line
The Compatibility section is largely honest — every positive claim I checked holds — but it understates the drop list. Folding items 1–10 (especially the three silent ones) into a Migration section would make this a clean, trustworthy cutover for existing 1.x users.
Method note: verified by cross-reading both trees file-by-file; line references are against this PR's head and main.
- model.constraints top_p, top_k and stop_sequences reach every provider whose API takes them; the others are reported by the run and by validate, which also warns that current Claude models reject sampling settings. The SDKs' constraints apply them too. - TypeScript tool handlers get an AbortSignal again, aborted when the turn is stopped or the session ends. - A plugin's settings reach its script tools and hooks as environment variables. - compliance/regulatory-map.yaml and validation-schedule.yaml are summed up in the system prompt again. - .sh tools and hooks run under sh unless their #! line names another shell; hooks get 10 seconds by default. - Unknown gitagent.yaml keys are warnings instead of errors, and validate checks the file. - README: what changed from 1.x; BLUEPRINT D28.
On macOS the sandbox launcher starts the shell through the system's sandbox-exec, with a profile that allows writes only beneath the same directories as on Linux (plus the macOS cache folders); reads stay unrestricted. Directories are also given under their real paths, since /tmp and the temp folders are links into /private. A gitagent started inside the sandbox stays in it instead of failing to enter another. The sandbox crate's apply() becomes prepare(), which returns how to start the program: directly on Linux, where the launcher restricts itself as before, or through sandbox-exec on macOS. No new dependency. CI gains a macOS job that runs the sandbox tests and checks that the launcher refuses a write outside the allowed folders. The docs no longer list macOS as running without a sandbox.
gitagent 3: the Rust rewrite
gitagent becomes one static binary. It runs the same agent folders, hook scripts, schedules and SDK code as the TypeScript gitagent, scores higher than any published harness on Terminal-Bench 2.1, and adds a terminal console, parallel sub-agents with a live task board, schedules, and SDKs for any language over a line protocol.
This branch replaces the TypeScript tree. Merge with squash and tag the result
v3.0.0. The full history stays onfeat/v3-rust.Benchmark
Terminal-Bench 2.1: 93.03% ±0.8 with Claude Opus 5.5, all 89 tasks, five runs each, code frozen at the published commit. The ± is the run-to-run variation, as on the official leaderboard. For comparison: Terminus 2 with the same model 87.6%, Codex CLI with GPT-6 Astra 87.4% ±0.9, and the highest published entry, Artificial Analysis's harness with Claude Fable 5.1 at maximum effort, 91.4%.
Every trial is public on the Harbor hub, one job per time limit; the README's Results section links them. The rules that keep the number honest are in
docs/INTEGRITY.mdand are checked byscripts/integrity-audit.sh: the agent never reads a task's tests or verifier output, benchmark mode uses only the built-in configuration, and a test fails if the prompt or tool list sent to the model in--benchever changes.What is new
curlto install; no Node or npm. Linux x86_64 and arm64 (static musl), macOS.README.mdexplains why these raise the score;docs/HOW-IT-WORKS.mdhas the mechanics.gitagent -d <agent>: tool lines, approvals,/commands with suggestions,/help, a live task board for parallel work.runtime.parallel_tasks: truegives the agent aplantool: phases of tasks run as sub-agents, a reconciling step combines the results. Each task can carry its own turn limit; a task that runs out is warned first and comes back marked partial.max_parallelis a ceiling: a rate-limit refusal halves the number running at once, and it grows back as calls go through.gitagent schedule list|add|remove|enable|disable|run|start, cron and one-off, with per-schedule logs.--repo <url>: work in a fresh clone on a session branch, commit, optionally push.gitagent --ipcspeaks JSON lines on stdin and stdout. Python and TypeScript clients are inclients/; any language can drive it (the README shows a bash client). The TypeScript client keeps the 1.xquery()andtool()API.agent.yaml.sandbox-execon macOS.gitagent learnturns finished runs into general lessons for the system prompt;learning: onlinedoes it per project after each finished task.Compatibility
Agent folders,
agent.yaml(1.x settings included),SOUL.md,RULES.md, memory, skills, declared tools, hook scripts, schedules, workflows, sub-agents,config/<env>.yaml,.env, OpenGAP folders and.claude/project layouts load unchanged.gitagent validatepoints out old spellings as warnings, never errors. The CLI flags-p,-d,-m,-e,--repo,--pat,--sessionand the slash commands work as before.1.x behaviour that this branch keeps, after the compatibility review:
model.constraintstop_p,top_kandstop_sequencesare sent to every provider whose API takes them; where one cannot be sent (OpenAI has notop_kor stop sequences), the run andgitagent validatesay so.validatealso warns that current Claude models rejecttemperature,top_pandtop_k.tool()handler gets anAbortSignalagain, aborted when the turn is stopped or the session ends; returning 1.x's{text, details}works as before.compliance/regulatory-map.yamland the checks incompliance/validation-schedule.yamlare in the system prompt again..shtools and hooks run undershunless their#!line names another shell; hooks get 10 seconds unlesshooks.yamlsetstimeout.gitagent.yamlis a warning, not an error, andgitagent validatechecks the file.Migration from 1.x
The README's "Coming from the TypeScript gitagent" section has the details. In short:
--voice; the remote--sandbox(the binary prints a one-line note); a plugin'sentryJavaScript; the 1.x telemetry adapter (OTLP export replaces it).task_trackerkept a searchable list of tasks; gitagent 3'stodois a checklist the agent rewrites whole. The old name still maps to it..pywithpython3,.jswithnode, …) where 1.x usedshfor all of them.None of this touches a benchmark run:
--benchreads noagent.yaml, plugin, hook, compliance file or usergitagent.yaml, and the guard test and integrity audit still pass.Layout
A Cargo workspace:
cli,core,loader,provider,tools,terminal,shaper,context,guards,hooks,mcp,record,repo,sandbox,learn,sdk;clients/pythonandclients/typescript;gitagent_harbor/(the Harbor shim);bench/(the benchmark profiles, public job links and prices);profiles/(per-model-family behaviour);prompts/;scripts/.Checks
cargo test: 374 tests, including the benchmark guard test.cargo clippy --all-targets(Linux and macOS),cargo fmt --check,scripts/integrity-audit.sh: clean. CI runs the sandbox tests on a macOS runner.npm run checkinclients/typescript.Docs
README.md(why, install, use, CLI options, results, how it works, architecture, SDKs, migration, configuration, Harbor, troubleshooting),BLUEPRINT.md(requirements, design decisions, open questions),docs/HOW-IT-WORKS.md,docs/INTEGRITY.md,CONTRIBUTING.md.Known gaps