Commit c1bd5e3
root
v1.2.0 — Empirical tier restructure, human-proof agentic challenges
Major changes since v0.8.0:
Tier restructure (empirically calibrated):
Only types where GPT-5.2 achieves 100% are active.
Easy (4): simple_math, string_math, binary, pattern
Medium (2): sorting, word_math
Hard (3): nested_operations, base_conversion_chain, power_mod [NEW]
Agentic (1): chained_arithmetic [NEW]
Shelved (17): All character-manipulation types reserved for future models
New challenge types:
- chained_arithmetic: 4-step chain with 4 operation patterns
GPT-5.2: 100% | GPT-4o: 30% | Humans: 15-20s without paper
- power_mod: base^exp mod m
GPT-5.2: 100% | GPT-4o: 80%
Dynamic mode:
- Generates multi-step arithmetic (no character manipulation)
- Iteratively refined through 4 rounds against real API
- GPT-5.2 verified at 100% solve rate
Frontend (challenge.llm.kaveenk.com):
- Human/Agent toggle (agent mode hides everything, shows SKILL.md)
- All tiers displayed with empirical percentages
- GPT-5.2 row in model compatibility table
- Shelved types section (greyed out)
- Lock mode TTL: 5 seconds
- Toggle centered properly
Tests: 219 unit + 23 E2E, 7/7 CI green
GPT-5.2 calibration data: tests/calibration_gpt52.json1 parent d9758aa commit c1bd5e3
3 files changed
Lines changed: 3 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | | - | |
| 37 | + | |
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| |||
0 commit comments