Skip to content

Commit 324bbcb

Browse files
ostenbomclaude
andcommitted
Correct which Clairic variant passed longer flows
Clairic with Clef also passed activity-after-account-update 10/10, so the 2/10 ceiling for multi-step scenarios applies to the naive loop only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent bfaf911 commit 324bbcb

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

‎scripts/jev/BENCHMARK_RESULTS_CLAIRIC.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ The Jev reference reproduces the October 16 measurement of the same controller (
9292

9393
**Without that shortcut, it does not know when to stop.** With Clef judging completion, false completions dropped to zero, but 97 of 150 runs stalled or hit the step limit. The action log says what was clicked, not what the page answered, so after "Save Changes" Clairic saw no reason to move on and clicked Save again until the repeat guard ended the run. Clef's completion score stayed low: across the 97 failed runs, the median of each run's highest score was 0.02 (90th percentile 0.04).
9494

95-
**Multi-step flows mostly never finish.** Apart from delete-account, every scenario that needs more than a couple of actions scored at most 2/10 in every Clairic variant. Typical traces bounce between pages (plan-upgrade: Change Plan → open /dashboard → Change Plan again): 25–35% of all steps opened a direct URL, and 95% of those in `clairic` reopened `/dashboard`. Checkout was reached in about half of the plan-upgrade and payment-history attempts and never completed.
95+
**Multi-step flows mostly never finish.** In `clairic`, apart from delete-account, every scenario that needs more than a couple of actions scored at most 2/10; `clairic-clef` also passed activity-after-account-update 10/10, and nothing longer. Typical traces bounce between pages (plan-upgrade: Change Plan → open /dashboard → Change Plan again): 25–35% of all steps opened a direct URL, and 95% of those in `clairic` reopened `/dashboard`. Checkout was reached in about half of the plan-upgrade and payment-history attempts and never completed.
9696

9797
**The screenshot made it worse again.** As with Jev, adding the screenshot lowered the pass rate (28% → 20%), raised cost 2.4×, and added more stops and stalls.
9898

0 commit comments

Comments
 (0)