Skip to content

Commit fab121e

Browse files
committed
Add Python quick start and update README presentation and links
1 parent da71216 commit fab121e

1 file changed

Lines changed: 40 additions & 5 deletions

File tree

‎README.md‎

Lines changed: 40 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,15 @@ Haizhong Zheng<sup>1</sup>, Yizhuo Di<sup>1</sup>, Ranajoy Sadhukhan<sup>1</sup>
88

99
<sup>1</sup>Carnegie Mellon University &nbsp; <sup>2</sup>Amazon
1010

11-
[![Website](https://img.shields.io/badge/website-online-9cf.svg)](https://haizhongzheng.github.io/ServeLearnBench/)
11+
[![Website](https://img.shields.io/badge/website-online-9cf.svg)](https://infini-ai-lab.github.io/ServeLearnBench/)
1212
[![Dataset on HF](https://img.shields.io/badge/%F0%9F%A4%97-dataset-yellow.svg)](https://huggingface.co/datasets/haizhongzheng/ServeLearnBench)
1313
[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](./LICENSE)
1414
[![Python](https://img.shields.io/badge/python-3.12%2B-blue.svg)](pyproject.toml)
1515

1616
</div>
1717

18+
<p align="center"><img src="docs/assets/frontier.png" width="92%" alt="Hidden reward versus token and API cost per task"></p>
19+
1820
The knowledge an agent needs in deployment is often implicit, undisclosed, and subject to change. ServeLearnBench
1921
measures whether an agent can **infer that knowledge from its own serving experience, revise it when it goes stale,
2022
and keep behaving correctly on everything that did not change**.
@@ -38,7 +40,7 @@ Nine scenarios, **53 windows, 4,508 serving tasks and 3,210 held-out test tasks*
3840
## Installation
3941

4042
```bash
41-
git clone https://github.com/haizhongzheng/ServeLearnBench.git
43+
git clone https://github.com/Infini-AI-Lab/ServeLearnBench.git
4244
cd ServeLearnBench
4345
pip install -e . # Python 3.12+
4446
slb verify # regenerates all nine scenarios and checks them against the paper's
@@ -47,6 +49,10 @@ slb verify # regenerates all nine scenarios and checks them aga
4749
## Quick start
4850

4951
Any OpenAI-compatible endpoint with native tool calling works. The key is read from `SLB_API_KEY`.
52+
Use the command line for a quick baseline run, or the Python API to configure a run in code. Both use the same
53+
task streams, learning schedule, and evaluator.
54+
55+
### Command line
5056

5157
```bash
5258
export SLB_API_KEY=...
@@ -62,7 +68,36 @@ slb run --scenario banking_l1 --method rag --model accounts/fireworks/models/glm
6268
slb score results/*/*.jsonl
6369
```
6470

65-
Results are written to `results/<model>/<scenario>_<method>.jsonl`, one line per episode with the full trajectory.
71+
### Python API
72+
73+
With `SLB_API_KEY` set, run the same RAG evaluation and inspect its scores directly:
74+
75+
```python
76+
from servelearnbench.llm import ModelConfig
77+
from servelearnbench.methods import RAG
78+
from servelearnbench.runner import run
79+
from servelearnbench.scoring import read_rows, setting_scores
80+
81+
model = ModelConfig(
82+
model="accounts/fireworks/models/glm-5p3-flash",
83+
base_url="https://api.fireworks.ai/inference/v1",
84+
)
85+
output = run(
86+
scenario="banking_l1",
87+
method=RAG(),
88+
model_cfg=model,
89+
out="results/banking_l1_rag.jsonl",
90+
)
91+
_, tasks = read_rows(output)
92+
print(setting_scores(tasks))
93+
```
94+
95+
Replace `RAG()` with `Blind()` or `Oracle()` (imported from the same methods module) to run a reference baseline.
96+
To implement your own learning method, see [examples/custom_method.py](examples/custom_method.py) and
97+
[the method interface](docs/methods.md).
98+
99+
The CLI writes results to `results/<model>/<scenario>_<method>.jsonl`; the Python API uses the path supplied as
100+
`out`. Each file records task rewards and full trajectories.
66101
Pitch is scored by an LLM judge (`--judge-model`, default `glm-5p3-flash` at high reasoning effort, as in the paper).
67102
See [docs/getting-started.md](docs/getting-started.md) for all options and local servers.
68103

@@ -79,7 +114,7 @@ The paper also evaluates Mem0, SkillOpt, Continual Harness and Prime; their resu
79114

80115
## Results
81116

82-
Scores of every model and method evaluated in the paper are on the [project website](https://haizhongzheng.github.io/ServeLearnBench/)
117+
Scores of every model and method evaluated in the paper are on the [project website](https://infini-ai-lab.github.io/ServeLearnBench/)
83118
and in [docs/leaderboard.csv](docs/leaderboard.csv).
84119

85120
## Documentation
@@ -124,4 +159,4 @@ If you find ServeLearnBench useful, please cite:
124159

125160
## License
126161

127-
Apache 2.0. The retail domain's tool interface follows the design of [τ-bench](https://github.com/sierra-research/tau-bench).
162+
Apache 2.0.

0 commit comments

Comments
 (0)