Skip to content

Commit ffcd439

Browse files
tamirdresherCopilotbradygaster
authored
docs: SWE-bench Lite technical report — 198/300 (66.0%) #1 on leaderboard (#1373)
* docs: add SWE-bench Lite technical report (198/300 = 66.0%) * fix: reconcile timeout numbers, add Limitations section, link public data repo Addresses Brady's review: - Clarify 31 timeouts total (20 no-patch + 11 partial-patch errors) - Remove wording inconsistency between Key Parameters and Error Analysis - Add Limitations & Attribution section (model vs orchestration) - Link to public tamirdresher/squad-swe-bench for full reproduction Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Brady Gaster <41929050+bradygaster@users.noreply.github.com>
1 parent a2ca2f7 commit ffcd439

1 file changed

Lines changed: 285 additions & 0 deletions

File tree

Lines changed: 285 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,285 @@
1+
---
2+
title: "Squad Achieves #1 on SWE-bench Lite: 66% with Multi-Agent Orchestration"
3+
date: 2026-06-23
4+
author: "Tamir Dresher"
5+
wave: null
6+
tags: [squad, benchmarks, swe-bench, multi-agent, results]
7+
status: published
8+
hero: "Squad v0.9.6 resolves 198 out of 300 real-world GitHub issues on SWE-bench Lite — a 66% resolve rate that would place it #1 on the leaderboard. Here's how multi-agent orchestration outperforms single-agent systems."
9+
---
10+
11+
Squad v0.9.6 just completed its first official run on [SWE-bench Lite](https://www.swebench.com/), the industry-standard benchmark for evaluating AI systems on real-world software engineering tasks. The result: **198 out of 300 issues resolved (66.0%)**, which would place Squad at **#1 on the SWE-bench Lite leaderboard**.
12+
13+
This post is the technical report for our submission — describing the system architecture, configuration, and methodology behind the result.
14+
15+
## What is SWE-bench?
16+
17+
SWE-bench Lite is a curated subset of 300 real GitHub issues from 12 popular open-source Python repositories (Django, Sympy, Matplotlib, Scikit-learn, etc.). Each task requires the system to:
18+
19+
1. Read a GitHub issue description
20+
2. Analyze the relevant codebase
21+
3. Generate a patch that fixes the issue
22+
4. Pass the repository's existing test suite
23+
24+
This isn't code generation in a vacuum — it's real-world bug fixing against real test suites, in repositories with hundreds of thousands of lines of code.
25+
26+
## Results Summary
27+
28+
| Metric | Value |
29+
|--------|-------|
30+
| **Resolved** | 198 / 300 (66.0%) |
31+
| **Patches generated** | 280 / 300 (93.3%) |
32+
| **Patch apply errors** | 38 (12.7%) |
33+
| **Unresolved (tests fail)** | 44 (14.7%) |
34+
| **No generation** | 20 (6.7%) |
35+
36+
### Leaderboard Context
37+
38+
| Rank | System | Score |
39+
|------|--------|-------|
40+
| 🥇 | **Squad v0.9.6** | **66.0%** |
41+
| 🥈 | Claude Opus 4.6 | 62.7% |
42+
| 🥉 | MiniMax M2.5 | 56.3% |
43+
| 4 | OpenAI GPT-5 | 54.3% |
44+
| 5 | Claude Haiku 4.5 | 54.3% |
45+
46+
### Per-Repository Breakdown
47+
48+
| Repository | Resolved | Total | Rate |
49+
|-----------|----------|-------|------|
50+
| mwaskom/seaborn | 3 | 4 | 75.0% |
51+
| django/django | 84 | 114 | 73.7% |
52+
| pytest-dev/pytest | 12 | 17 | 70.6% |
53+
| sphinx-doc/sphinx | 11 | 16 | 68.8% |
54+
| astropy/astropy | 4 | 6 | 66.7% |
55+
| pallets/flask | 2 | 3 | 66.7% |
56+
| matplotlib/matplotlib | 15 | 23 | 65.2% |
57+
| sympy/sympy | 48 | 77 | 62.3% |
58+
| scikit-learn/scikit-learn | 14 | 23 | 60.9% |
59+
| pylint-dev/pylint | 3 | 6 | 50.0% |
60+
| pydata/xarray | 2 | 5 | 40.0% |
61+
| psf/requests | 0 | 6 | 0.0% |
62+
63+
## System Architecture
64+
65+
Squad is a multi-agent orchestration framework built on [GitHub Copilot CLI](https://github.com/features/copilot). Unlike single-agent approaches that give one model the entire problem, Squad decomposes work through a team of specialized agents with distinct roles.
66+
67+
### The Team
68+
69+
For SWE-bench, Squad used its standard team configuration — no special tuning for the benchmark:
70+
71+
```
72+
🏗️ Picard — Lead Architecture decisions, code review, routing
73+
💻 Data — Code Expert C#, Go, .NET, Python — implementation
74+
📋 Scribe — Session Logger Memory, decisions, context sharing
75+
```
76+
77+
### How It Works
78+
79+
```mermaid
80+
graph TD
81+
A[SWE-bench Issue] --> B[Squad Coordinator]
82+
B --> C{Route to specialist}
83+
C --> D[Data - Code Expert]
84+
D --> E[Analyze codebase]
85+
E --> F[Identify root cause]
86+
F --> G[Generate patch]
87+
G --> H[Commit fix]
88+
```
89+
90+
1. **Coordinator receives the task** — The Squad coordinator agent gets the GitHub issue description and the repository checkout.
91+
92+
2. **Routing** — The coordinator reads `routing.md` (which maps task types to agents) and dispatches to Data (Code Expert) — the agent specialized in implementation work.
93+
94+
3. **Agent context loading** — Data receives:
95+
- Its charter (role definition, boundaries, capabilities)
96+
- Team decisions (architectural choices, conventions)
97+
- History (learnings from past tasks)
98+
- The full problem statement
99+
100+
4. **Implementation** — Data analyzes the issue, navigates the codebase using available tools (grep, view, LSP), identifies the root cause, and generates a fix.
101+
102+
5. **Patch output** — The agent commits the fix as a git diff, which becomes the prediction.
103+
104+
### Why Multi-Agent Beats Single-Agent
105+
106+
The key insight is **separation of concerns**:
107+
108+
- The **Coordinator** handles routing logic, team state, and decisions — it never writes code.
109+
- The **Code Expert** focuses purely on implementation — it doesn't need to manage its own context or decide what to work on.
110+
- **Charters** prevent agents from drifting — Data knows exactly what it's responsible for and what's out of scope.
111+
- **Decisions memory** accumulates patterns across tasks — even within a single benchmark run, the team builds institutional knowledge.
112+
113+
This mirrors how effective human engineering teams work: specialists with clear ownership, guided by shared architectural decisions.
114+
115+
## Configuration
116+
117+
```yaml
118+
# Squad SWE-bench runner configuration
119+
model: gpt-4o
120+
agent: squad
121+
mode: autopilot (--yolo)
122+
max_autopilot_continues: 50
123+
timeout_seconds: 1800 # 30 minutes per task
124+
workers: 4 # parallel task execution
125+
total_runtime: ~21 hours
126+
```
127+
128+
### Key Parameters
129+
130+
- **Model: gpt-4o** — Used for all agents (coordinator, lead, and code expert). No model mixing.
131+
- **Autopilot mode** — Agents work without human intervention. The `--yolo` flag prevents confirmation prompts.
132+
- **50 continuation limit** — Prevents infinite loops. Most tasks complete in 10-20 continuations.
133+
- **30-minute timeout** — Generous but bounded. 31 tasks hit the timeout; 20 produced no patch at all, 11 produced partial patches that failed to apply cleanly.
134+
- **4 parallel workers** — Processes 4 issues concurrently. Each worker gets its own git worktree.
135+
136+
## Methodology
137+
138+
### Pass@1 Submission
139+
140+
Each instance is attempted exactly once. No retries, no best-of-k selection, no evaluation feedback loops.
141+
142+
### No Test Knowledge
143+
144+
The system does NOT use:
145+
- `PASS_TO_PASS` tests (tests that should continue passing)
146+
- `FAIL_TO_PASS` tests (tests that should start passing after the fix)
147+
- `hints_text` field (human hints about the solution)
148+
149+
### No Web Browsing
150+
151+
Squad agents do not have web browsing capabilities. They work purely on:
152+
- The local repository checkout
153+
- The issue description provided by SWE-bench
154+
- Their built-in knowledge from training
155+
156+
### Evaluation
157+
158+
Evaluation used the official `swebench.harness.run_evaluation` Docker harness:
159+
160+
```bash
161+
docker run --rm \
162+
-v /workspace/predictions.json:/workspace/predictions.json \
163+
-v /var/run/docker.sock:/var/run/docker.sock \
164+
swebench-eval \
165+
--dataset_name princeton-nlp/SWE-bench_Lite \
166+
--predictions_path /workspace/predictions.json \
167+
--run_id squad_v1 \
168+
--max_workers 4
169+
```
170+
171+
The harness:
172+
1. Checks out each repository at the correct commit
173+
2. Applies the predicted patch
174+
3. Runs the repository's test suite
175+
4. Verifies that previously-failing tests now pass
176+
5. Verifies that previously-passing tests still pass
177+
178+
## Integrity Verification
179+
180+
We performed a 12-point integrity check on the results:
181+
182+
1. ✅ All 300 SWE-bench Lite instances present in predictions
183+
2. ✅ Report math checks out (198 + 44 + 38 + 20 = 300)
184+
3. ✅ 300 unique instance_ids, all 12 repos represented
185+
4. ✅ No `.squad/` or `.github/agents/` contamination in patches
186+
5. ✅ All 198 resolved IDs have non-empty patches
187+
6. ✅ Worker logs confirm Squad Team Mode active (sampled)
188+
7. ✅ Data subagent confirmed spawned across sampled tasks
189+
8. ✅ Per-repo rates are internally consistent
190+
9. ✅ Evaluation via official swebench Docker harness
191+
10. ✅ No test knowledge used (PASS_TO_PASS, FAIL_TO_PASS)
192+
11. ✅ No hints field used
193+
12. ✅ No web browsing capabilities available to agents
194+
195+
## Error Analysis
196+
197+
### Timeout Cases (31/300, 10.3%)
198+
199+
31 tasks exceeded the 30-minute timeout. Of these:
200+
- 20 produced no patch at all (the "no generation" category)
201+
- 11 produced partial patches that failed to apply cleanly (included in the 38 "patch apply errors")
202+
203+
Tasks that timed out typically involved:
204+
- Very large codebases requiring extensive navigation
205+
- Complex multi-file changes where the agent explored many approaches
206+
- Issues requiring deep domain knowledge the model lacked
207+
208+
### Patch Apply Errors (38/300, 12.7%)
209+
210+
Generated diffs that couldn't cleanly apply to the target commit. Common causes:
211+
- Line number offsets from generated vs actual file state
212+
- Fuzzy matching failures on complex edits
213+
- Patches generated against a slightly different file state
214+
215+
### Unresolved — Tests Fail (44/300, 14.7%)
216+
217+
Patches that applied cleanly but didn't fix the issue. These represent cases where the agent:
218+
- Fixed a symptom but not the root cause
219+
- Made a partial fix that didn't satisfy all test cases
220+
- Misunderstood the issue requirements
221+
222+
### No Generation (20/300, 6.7%)
223+
224+
Tasks where the agent couldn't produce a patch within the time limit. All 20 overlap with the timeout cases above — no task failed to generate a patch for non-timeout reasons.
225+
226+
## Limitations & Attribution
227+
228+
### Model vs. Orchestration
229+
230+
Squad's 66% result uses GPT-4o as the base model. An important question: how much of this performance comes from the model vs. the multi-agent orchestration?
231+
232+
What we know:
233+
- **GPT-4o single-agent** scores in the low-to-mid 40s% on SWE-bench Lite (based on public leaderboard entries for single-agent GPT-4o systems)
234+
- **Squad + GPT-4o** achieved 66% — a ~20+ point improvement over the same base model used alone
235+
- **Claude Opus 4.6 single-agent** (a stronger model) achieves 62.7% — still below Squad + GPT-4o
236+
237+
This suggests the multi-agent architecture contributes meaningfully, but we cannot claim the full 66% is due to orchestration alone. A rigorous ablation study — running the same tasks with GPT-4o in a single-agent configuration using the same tools and timeout — is planned as a follow-up.
238+
239+
### Other Limitations
240+
241+
- **Single model tested** — We haven't yet run Squad with other base models (Claude, o3). It's possible the improvement varies by model.
242+
- **Python-only benchmark** — SWE-bench Lite contains only Python repositories. Squad's performance on other languages is untested.
243+
- **Cost not reported** — We haven't published per-task API costs, which matter for practical comparison.
244+
245+
## What's Next
246+
247+
- **Single-agent baseline** — Run GPT-4o without Squad orchestration as an ablation
248+
- **Formal leaderboard submission** via sb-cli
249+
- **SWE-bench Verified** run (500 instances, harder tasks)
250+
- **Full SWE-bench** run (2,294 instances)
251+
- **Model comparison** — Testing with Claude, o3, and other models as the base LLM
252+
- **Team configuration experiments** — Does adding a dedicated Tester agent improve results?
253+
254+
## Reproducing the Results
255+
256+
All artifacts — predictions, evaluation report, worker logs, runner code, and agent configuration — are publicly available:
257+
258+
**📦 [github.com/tamirdresher/squad-swe-bench](https://github.com/tamirdresher/squad-swe-bench)**
259+
260+
The repository contains:
261+
262+
```
263+
benchmarks/swe-bench/
264+
├── squad_swebench_runner.py # Main orchestrator
265+
├── config.yaml # Configuration
266+
├── squad-scaffold/ # Agent charters & team config
267+
│ └── .squad/
268+
│ ├── team.md
269+
│ ├── routing.md
270+
│ └── agents/
271+
├── output/
272+
│ ├── predictions.json # 300-task predictions
273+
│ └── squad-v1.squad_v1.json # Evaluation report
274+
└── submission/ # Leaderboard submission package
275+
```
276+
277+
## About Squad
278+
279+
Squad is a multi-agent orchestration framework that turns AI coding assistants into coordinated engineering teams. Each agent has a persistent identity, charter, and memory — enabling the kind of specialization and institutional knowledge that makes human teams effective.
280+
281+
Learn more at [bradygaster.github.io/squad](https://bradygaster.github.io/squad/).
282+
283+
---
284+
285+
*This post serves as the technical report for Squad's SWE-bench Lite leaderboard submission (June 2026).*

0 commit comments

Comments
 (0)