Skip to content

Commit 93f2fca

Browse files
committed
merge dev-merge: publish Tau3 benchmark results
2 parents ef33263 + afa69b2 commit 93f2fca

2 files changed

Lines changed: 29 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,21 @@ control rather than hidden implementation details.
119119
<em>Ablation-derived component contributions across task success, API-cost saving, and latency saving.</em>
120120
</p>
121121

122+
## Tau3 benchmark results
123+
124+
The latest three-way comparison was run against Hypha `main@ef33263` with one trial per task
125+
(385 tasks total). The base model was `deepseek-v4-flash`. Hypha is the framework result and
126+
Direct is the direct model-call baseline. Neither arm reported a system error.
127+
128+
| Benchmark | Tasks | Hypha | Direct |
129+
| --------- | ----: | ----: | ------: |
130+
| Mock | 10 | 0.800 (8/10) | 0.800 (8/10) |
131+
| Airline | 50 | 0.880 (44/50) | 0.900 (45/50) |
132+
| Retail | 114 | 0.912 (104/114) | 0.895 (102/114) |
133+
| Telecom | 114 | 0.605 (69/114) | 0.588 (67/114) |
134+
| Banking knowledge | 97 | 0.206 (20/97) | 0.196 (19/97) |
135+
| **Total** | **385** | **0.636 (245/385)** | **0.626 (241/385)** |
136+
122137
## Product model
123138

124139
| Concept | Responsibility |

‎README.zh-CN.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,20 @@ Invalidation 与 Bypass 都属于 Runtime 控制,而不是隐藏的实现细
114114
<em>通过 Ablation 得出的组件贡献,覆盖任务成功率、API 成本节省与延迟节省。</em>
115115
</p>
116116

117+
## Tau3 Benchmark 结果
118+
119+
最新三方对比基于 Hypha `main@ef33263`,每个任务运行一次,共 385 个任务。Base model 为
120+
`deepseek-v4-flash`。Hypha 列为框架结果,Direct 列为直接模型调用基线;两组均未报告系统错误。
121+
122+
| Benchmark | 任务数 | Hypha | Direct |
123+
| --------- | ----: | ----: | ------: |
124+
| Mock | 10 | 0.800(8/10) | 0.800(8/10) |
125+
| Airline | 50 | 0.880(44/50) | 0.900(45/50) |
126+
| Retail | 114 | 0.912(104/114) | 0.895(102/114) |
127+
| Telecom | 114 | 0.605(69/114) | 0.588(67/114) |
128+
| Banking knowledge | 97 | 0.206(20/97) | 0.196(19/97) |
129+
| **合计** | **385** | **0.636(245/385)** | **0.626(241/385)** |
130+
117131
## 产品模型
118132

119133
| 概念 | 职责 |

0 commit comments

Comments
 (0)