|
| 1 | +--- |
| 2 | +title: "Squad Achieves #1 on SWE-bench Lite: 66% with Multi-Agent Orchestration" |
| 3 | +date: 2026-06-23 |
| 4 | +author: "Tamir Dresher" |
| 5 | +wave: null |
| 6 | +tags: [squad, benchmarks, swe-bench, multi-agent, results] |
| 7 | +status: published |
| 8 | +hero: "Squad v0.9.6 resolves 198 out of 300 real-world GitHub issues on SWE-bench Lite — a 66% resolve rate that would place it #1 on the leaderboard. Here's how multi-agent orchestration outperforms single-agent systems." |
| 9 | +--- |
| 10 | + |
| 11 | +Squad v0.9.6 just completed its first official run on [SWE-bench Lite](https://www.swebench.com/), the industry-standard benchmark for evaluating AI systems on real-world software engineering tasks. The result: **198 out of 300 issues resolved (66.0%)**, which would place Squad at **#1 on the SWE-bench Lite leaderboard**. |
| 12 | + |
| 13 | +This post is the technical report for our submission — describing the system architecture, configuration, and methodology behind the result. |
| 14 | + |
| 15 | +## What is SWE-bench? |
| 16 | + |
| 17 | +SWE-bench Lite is a curated subset of 300 real GitHub issues from 12 popular open-source Python repositories (Django, Sympy, Matplotlib, Scikit-learn, etc.). Each task requires the system to: |
| 18 | + |
| 19 | +1. Read a GitHub issue description |
| 20 | +2. Analyze the relevant codebase |
| 21 | +3. Generate a patch that fixes the issue |
| 22 | +4. Pass the repository's existing test suite |
| 23 | + |
| 24 | +This isn't code generation in a vacuum — it's real-world bug fixing against real test suites, in repositories with hundreds of thousands of lines of code. |
| 25 | + |
| 26 | +## Results Summary |
| 27 | + |
| 28 | +| Metric | Value | |
| 29 | +|--------|-------| |
| 30 | +| **Resolved** | 198 / 300 (66.0%) | |
| 31 | +| **Patches generated** | 280 / 300 (93.3%) | |
| 32 | +| **Patch apply errors** | 38 (12.7%) | |
| 33 | +| **Unresolved (tests fail)** | 44 (14.7%) | |
| 34 | +| **No generation** | 20 (6.7%) | |
| 35 | + |
| 36 | +### Leaderboard Context |
| 37 | + |
| 38 | +| Rank | System | Score | |
| 39 | +|------|--------|-------| |
| 40 | +| 🥇 | **Squad v0.9.6** | **66.0%** | |
| 41 | +| 🥈 | Claude Opus 4.6 | 62.7% | |
| 42 | +| 🥉 | MiniMax M2.5 | 56.3% | |
| 43 | +| 4 | OpenAI GPT-5 | 54.3% | |
| 44 | +| 5 | Claude Haiku 4.5 | 54.3% | |
| 45 | + |
| 46 | +### Per-Repository Breakdown |
| 47 | + |
| 48 | +| Repository | Resolved | Total | Rate | |
| 49 | +|-----------|----------|-------|------| |
| 50 | +| mwaskom/seaborn | 3 | 4 | 75.0% | |
| 51 | +| django/django | 84 | 114 | 73.7% | |
| 52 | +| pytest-dev/pytest | 12 | 17 | 70.6% | |
| 53 | +| sphinx-doc/sphinx | 11 | 16 | 68.8% | |
| 54 | +| astropy/astropy | 4 | 6 | 66.7% | |
| 55 | +| pallets/flask | 2 | 3 | 66.7% | |
| 56 | +| matplotlib/matplotlib | 15 | 23 | 65.2% | |
| 57 | +| sympy/sympy | 48 | 77 | 62.3% | |
| 58 | +| scikit-learn/scikit-learn | 14 | 23 | 60.9% | |
| 59 | +| pylint-dev/pylint | 3 | 6 | 50.0% | |
| 60 | +| pydata/xarray | 2 | 5 | 40.0% | |
| 61 | +| psf/requests | 0 | 6 | 0.0% | |
| 62 | + |
| 63 | +## System Architecture |
| 64 | + |
| 65 | +Squad is a multi-agent orchestration framework built on [GitHub Copilot CLI](https://github.com/features/copilot). Unlike single-agent approaches that give one model the entire problem, Squad decomposes work through a team of specialized agents with distinct roles. |
| 66 | + |
| 67 | +### The Team |
| 68 | + |
| 69 | +For SWE-bench, Squad used its standard team configuration — no special tuning for the benchmark: |
| 70 | + |
| 71 | +``` |
| 72 | +🏗️ Picard — Lead Architecture decisions, code review, routing |
| 73 | +💻 Data — Code Expert C#, Go, .NET, Python — implementation |
| 74 | +📋 Scribe — Session Logger Memory, decisions, context sharing |
| 75 | +``` |
| 76 | + |
| 77 | +### How It Works |
| 78 | + |
| 79 | +```mermaid |
| 80 | +graph TD |
| 81 | + A[SWE-bench Issue] --> B[Squad Coordinator] |
| 82 | + B --> C{Route to specialist} |
| 83 | + C --> D[Data - Code Expert] |
| 84 | + D --> E[Analyze codebase] |
| 85 | + E --> F[Identify root cause] |
| 86 | + F --> G[Generate patch] |
| 87 | + G --> H[Commit fix] |
| 88 | +``` |
| 89 | + |
| 90 | +1. **Coordinator receives the task** — The Squad coordinator agent gets the GitHub issue description and the repository checkout. |
| 91 | + |
| 92 | +2. **Routing** — The coordinator reads `routing.md` (which maps task types to agents) and dispatches to Data (Code Expert) — the agent specialized in implementation work. |
| 93 | + |
| 94 | +3. **Agent context loading** — Data receives: |
| 95 | + - Its charter (role definition, boundaries, capabilities) |
| 96 | + - Team decisions (architectural choices, conventions) |
| 97 | + - History (learnings from past tasks) |
| 98 | + - The full problem statement |
| 99 | + |
| 100 | +4. **Implementation** — Data analyzes the issue, navigates the codebase using available tools (grep, view, LSP), identifies the root cause, and generates a fix. |
| 101 | + |
| 102 | +5. **Patch output** — The agent commits the fix as a git diff, which becomes the prediction. |
| 103 | + |
| 104 | +### Why Multi-Agent Beats Single-Agent |
| 105 | + |
| 106 | +The key insight is **separation of concerns**: |
| 107 | + |
| 108 | +- The **Coordinator** handles routing logic, team state, and decisions — it never writes code. |
| 109 | +- The **Code Expert** focuses purely on implementation — it doesn't need to manage its own context or decide what to work on. |
| 110 | +- **Charters** prevent agents from drifting — Data knows exactly what it's responsible for and what's out of scope. |
| 111 | +- **Decisions memory** accumulates patterns across tasks — even within a single benchmark run, the team builds institutional knowledge. |
| 112 | + |
| 113 | +This mirrors how effective human engineering teams work: specialists with clear ownership, guided by shared architectural decisions. |
| 114 | + |
| 115 | +## Configuration |
| 116 | + |
| 117 | +```yaml |
| 118 | +# Squad SWE-bench runner configuration |
| 119 | +model: gpt-4o |
| 120 | +agent: squad |
| 121 | +mode: autopilot (--yolo) |
| 122 | +max_autopilot_continues: 50 |
| 123 | +timeout_seconds: 1800 # 30 minutes per task |
| 124 | +workers: 4 # parallel task execution |
| 125 | +total_runtime: ~21 hours |
| 126 | +``` |
| 127 | +
|
| 128 | +### Key Parameters |
| 129 | +
|
| 130 | +- **Model: gpt-4o** — Used for all agents (coordinator, lead, and code expert). No model mixing. |
| 131 | +- **Autopilot mode** — Agents work without human intervention. The `--yolo` flag prevents confirmation prompts. |
| 132 | +- **50 continuation limit** — Prevents infinite loops. Most tasks complete in 10-20 continuations. |
| 133 | +- **30-minute timeout** — Generous but bounded. 31 tasks hit the timeout; 20 produced no patch at all, 11 produced partial patches that failed to apply cleanly. |
| 134 | +- **4 parallel workers** — Processes 4 issues concurrently. Each worker gets its own git worktree. |
| 135 | + |
| 136 | +## Methodology |
| 137 | + |
| 138 | +### Pass@1 Submission |
| 139 | + |
| 140 | +Each instance is attempted exactly once. No retries, no best-of-k selection, no evaluation feedback loops. |
| 141 | + |
| 142 | +### No Test Knowledge |
| 143 | + |
| 144 | +The system does NOT use: |
| 145 | +- `PASS_TO_PASS` tests (tests that should continue passing) |
| 146 | +- `FAIL_TO_PASS` tests (tests that should start passing after the fix) |
| 147 | +- `hints_text` field (human hints about the solution) |
| 148 | + |
| 149 | +### No Web Browsing |
| 150 | + |
| 151 | +Squad agents do not have web browsing capabilities. They work purely on: |
| 152 | +- The local repository checkout |
| 153 | +- The issue description provided by SWE-bench |
| 154 | +- Their built-in knowledge from training |
| 155 | + |
| 156 | +### Evaluation |
| 157 | + |
| 158 | +Evaluation used the official `swebench.harness.run_evaluation` Docker harness: |
| 159 | + |
| 160 | +```bash |
| 161 | +docker run --rm \ |
| 162 | + -v /workspace/predictions.json:/workspace/predictions.json \ |
| 163 | + -v /var/run/docker.sock:/var/run/docker.sock \ |
| 164 | + swebench-eval \ |
| 165 | + --dataset_name princeton-nlp/SWE-bench_Lite \ |
| 166 | + --predictions_path /workspace/predictions.json \ |
| 167 | + --run_id squad_v1 \ |
| 168 | + --max_workers 4 |
| 169 | +``` |
| 170 | + |
| 171 | +The harness: |
| 172 | +1. Checks out each repository at the correct commit |
| 173 | +2. Applies the predicted patch |
| 174 | +3. Runs the repository's test suite |
| 175 | +4. Verifies that previously-failing tests now pass |
| 176 | +5. Verifies that previously-passing tests still pass |
| 177 | + |
| 178 | +## Integrity Verification |
| 179 | + |
| 180 | +We performed a 12-point integrity check on the results: |
| 181 | + |
| 182 | +1. ✅ All 300 SWE-bench Lite instances present in predictions |
| 183 | +2. ✅ Report math checks out (198 + 44 + 38 + 20 = 300) |
| 184 | +3. ✅ 300 unique instance_ids, all 12 repos represented |
| 185 | +4. ✅ No `.squad/` or `.github/agents/` contamination in patches |
| 186 | +5. ✅ All 198 resolved IDs have non-empty patches |
| 187 | +6. ✅ Worker logs confirm Squad Team Mode active (sampled) |
| 188 | +7. ✅ Data subagent confirmed spawned across sampled tasks |
| 189 | +8. ✅ Per-repo rates are internally consistent |
| 190 | +9. ✅ Evaluation via official swebench Docker harness |
| 191 | +10. ✅ No test knowledge used (PASS_TO_PASS, FAIL_TO_PASS) |
| 192 | +11. ✅ No hints field used |
| 193 | +12. ✅ No web browsing capabilities available to agents |
| 194 | + |
| 195 | +## Error Analysis |
| 196 | + |
| 197 | +### Timeout Cases (31/300, 10.3%) |
| 198 | + |
| 199 | +31 tasks exceeded the 30-minute timeout. Of these: |
| 200 | +- 20 produced no patch at all (the "no generation" category) |
| 201 | +- 11 produced partial patches that failed to apply cleanly (included in the 38 "patch apply errors") |
| 202 | + |
| 203 | +Tasks that timed out typically involved: |
| 204 | +- Very large codebases requiring extensive navigation |
| 205 | +- Complex multi-file changes where the agent explored many approaches |
| 206 | +- Issues requiring deep domain knowledge the model lacked |
| 207 | + |
| 208 | +### Patch Apply Errors (38/300, 12.7%) |
| 209 | + |
| 210 | +Generated diffs that couldn't cleanly apply to the target commit. Common causes: |
| 211 | +- Line number offsets from generated vs actual file state |
| 212 | +- Fuzzy matching failures on complex edits |
| 213 | +- Patches generated against a slightly different file state |
| 214 | + |
| 215 | +### Unresolved — Tests Fail (44/300, 14.7%) |
| 216 | + |
| 217 | +Patches that applied cleanly but didn't fix the issue. These represent cases where the agent: |
| 218 | +- Fixed a symptom but not the root cause |
| 219 | +- Made a partial fix that didn't satisfy all test cases |
| 220 | +- Misunderstood the issue requirements |
| 221 | + |
| 222 | +### No Generation (20/300, 6.7%) |
| 223 | + |
| 224 | +Tasks where the agent couldn't produce a patch within the time limit. All 20 overlap with the timeout cases above — no task failed to generate a patch for non-timeout reasons. |
| 225 | + |
| 226 | +## Limitations & Attribution |
| 227 | + |
| 228 | +### Model vs. Orchestration |
| 229 | + |
| 230 | +Squad's 66% result uses GPT-4o as the base model. An important question: how much of this performance comes from the model vs. the multi-agent orchestration? |
| 231 | + |
| 232 | +What we know: |
| 233 | +- **GPT-4o single-agent** scores in the low-to-mid 40s% on SWE-bench Lite (based on public leaderboard entries for single-agent GPT-4o systems) |
| 234 | +- **Squad + GPT-4o** achieved 66% — a ~20+ point improvement over the same base model used alone |
| 235 | +- **Claude Opus 4.6 single-agent** (a stronger model) achieves 62.7% — still below Squad + GPT-4o |
| 236 | + |
| 237 | +This suggests the multi-agent architecture contributes meaningfully, but we cannot claim the full 66% is due to orchestration alone. A rigorous ablation study — running the same tasks with GPT-4o in a single-agent configuration using the same tools and timeout — is planned as a follow-up. |
| 238 | + |
| 239 | +### Other Limitations |
| 240 | + |
| 241 | +- **Single model tested** — We haven't yet run Squad with other base models (Claude, o3). It's possible the improvement varies by model. |
| 242 | +- **Python-only benchmark** — SWE-bench Lite contains only Python repositories. Squad's performance on other languages is untested. |
| 243 | +- **Cost not reported** — We haven't published per-task API costs, which matter for practical comparison. |
| 244 | + |
| 245 | +## What's Next |
| 246 | + |
| 247 | +- **Single-agent baseline** — Run GPT-4o without Squad orchestration as an ablation |
| 248 | +- **Formal leaderboard submission** via sb-cli |
| 249 | +- **SWE-bench Verified** run (500 instances, harder tasks) |
| 250 | +- **Full SWE-bench** run (2,294 instances) |
| 251 | +- **Model comparison** — Testing with Claude, o3, and other models as the base LLM |
| 252 | +- **Team configuration experiments** — Does adding a dedicated Tester agent improve results? |
| 253 | + |
| 254 | +## Reproducing the Results |
| 255 | + |
| 256 | +All artifacts — predictions, evaluation report, worker logs, runner code, and agent configuration — are publicly available: |
| 257 | + |
| 258 | +**📦 [github.com/tamirdresher/squad-swe-bench](https://github.com/tamirdresher/squad-swe-bench)** |
| 259 | + |
| 260 | +The repository contains: |
| 261 | + |
| 262 | +``` |
| 263 | +benchmarks/swe-bench/ |
| 264 | +├── squad_swebench_runner.py # Main orchestrator |
| 265 | +├── config.yaml # Configuration |
| 266 | +├── squad-scaffold/ # Agent charters & team config |
| 267 | +│ └── .squad/ |
| 268 | +│ ├── team.md |
| 269 | +│ ├── routing.md |
| 270 | +│ └── agents/ |
| 271 | +├── output/ |
| 272 | +│ ├── predictions.json # 300-task predictions |
| 273 | +│ └── squad-v1.squad_v1.json # Evaluation report |
| 274 | +└── submission/ # Leaderboard submission package |
| 275 | +``` |
| 276 | + |
| 277 | +## About Squad |
| 278 | + |
| 279 | +Squad is a multi-agent orchestration framework that turns AI coding assistants into coordinated engineering teams. Each agent has a persistent identity, charter, and memory — enabling the kind of specialization and institutional knowledge that makes human teams effective. |
| 280 | + |
| 281 | +Learn more at [bradygaster.github.io/squad](https://bradygaster.github.io/squad/). |
| 282 | + |
| 283 | +--- |
| 284 | + |
| 285 | +*This post serves as the technical report for Squad's SWE-bench Lite leaderboard submission (June 2026).* |
0 commit comments