@@ -8,13 +8,15 @@ Haizhong Zheng<sup>1</sup>, Yizhuo Di<sup>1</sup>, Ranajoy Sadhukhan<sup>1</sup>
88
99<sup >1</sup >Carnegie Mellon University   ; <sup >2</sup >Amazon
1010
11- [ ![ Website] ( https://img.shields.io/badge/website-online-9cf.svg )] ( https://haizhongzheng .github.io/ServeLearnBench/ )
11+ [ ![ Website] ( https://img.shields.io/badge/website-online-9cf.svg )] ( https://infini-ai-lab .github.io/ServeLearnBench/ )
1212[ ![ Dataset on HF] ( https://img.shields.io/badge/%F0%9F%A4%97-dataset-yellow.svg )] ( https://huggingface.co/datasets/haizhongzheng/ServeLearnBench )
1313[ ![ License] ( https://img.shields.io/badge/license-Apache%202.0-green.svg )] ( ./LICENSE )
1414[ ![ Python] ( https://img.shields.io/badge/python-3.12%2B-blue.svg )] ( pyproject.toml )
1515
1616</div >
1717
18+ <p align =" center " ><img src =" docs/assets/frontier.png " width =" 92% " alt =" Hidden reward versus token and API cost per task " ></p >
19+
1820The knowledge an agent needs in deployment is often implicit, undisclosed, and subject to change. ServeLearnBench
1921measures whether an agent can ** infer that knowledge from its own serving experience, revise it when it goes stale,
2022and keep behaving correctly on everything that did not change** .
@@ -38,7 +40,7 @@ Nine scenarios, **53 windows, 4,508 serving tasks and 3,210 held-out test tasks*
3840## Installation
3941
4042``` bash
41- git clone https://github.com/haizhongzheng /ServeLearnBench.git
43+ git clone https://github.com/Infini-AI-Lab /ServeLearnBench.git
4244cd ServeLearnBench
4345pip install -e . # Python 3.12+
4446slb verify # regenerates all nine scenarios and checks them against the paper's
@@ -47,6 +49,10 @@ slb verify # regenerates all nine scenarios and checks them aga
4749## Quick start
4850
4951Any OpenAI-compatible endpoint with native tool calling works. The key is read from ` SLB_API_KEY ` .
52+ Use the command line for a quick baseline run, or the Python API to configure a run in code. Both use the same
53+ task streams, learning schedule, and evaluator.
54+
55+ ### Command line
5056
5157``` bash
5258export SLB_API_KEY=...
@@ -62,7 +68,36 @@ slb run --scenario banking_l1 --method rag --model accounts/fireworks/models/glm
6268slb score results/* /* .jsonl
6369```
6470
65- Results are written to ` results/<model>/<scenario>_<method>.jsonl ` , one line per episode with the full trajectory.
71+ ### Python API
72+
73+ With ` SLB_API_KEY ` set, run the same RAG evaluation and inspect its scores directly:
74+
75+ ``` python
76+ from servelearnbench.llm import ModelConfig
77+ from servelearnbench.methods import RAG
78+ from servelearnbench.runner import run
79+ from servelearnbench.scoring import read_rows, setting_scores
80+
81+ model = ModelConfig(
82+ model = " accounts/fireworks/models/glm-5p3-flash" ,
83+ base_url = " https://api.fireworks.ai/inference/v1" ,
84+ )
85+ output = run(
86+ scenario = " banking_l1" ,
87+ method = RAG(),
88+ model_cfg = model,
89+ out = " results/banking_l1_rag.jsonl" ,
90+ )
91+ _, tasks = read_rows(output)
92+ print (setting_scores(tasks))
93+ ```
94+
95+ Replace ` RAG() ` with ` Blind() ` or ` Oracle() ` (imported from the same methods module) to run a reference baseline.
96+ To implement your own learning method, see [ examples/custom_method.py] ( examples/custom_method.py ) and
97+ [ the method interface] ( docs/methods.md ) .
98+
99+ The CLI writes results to ` results/<model>/<scenario>_<method>.jsonl ` ; the Python API uses the path supplied as
100+ ` out ` . Each file records task rewards and full trajectories.
66101Pitch is scored by an LLM judge (` --judge-model ` , default ` glm-5p3-flash ` at high reasoning effort, as in the paper).
67102See [ docs/getting-started.md] ( docs/getting-started.md ) for all options and local servers.
68103
@@ -79,7 +114,7 @@ The paper also evaluates Mem0, SkillOpt, Continual Harness and Prime; their resu
79114
80115## Results
81116
82- Scores of every model and method evaluated in the paper are on the [ project website] ( https://haizhongzheng .github.io/ServeLearnBench/ )
117+ Scores of every model and method evaluated in the paper are on the [ project website] ( https://infini-ai-lab .github.io/ServeLearnBench/ )
83118and in [ docs/leaderboard.csv] ( docs/leaderboard.csv ) .
84119
85120## Documentation
@@ -124,4 +159,4 @@ If you find ServeLearnBench useful, please cite:
124159
125160## License
126161
127- Apache 2.0. The retail domain's tool interface follows the design of [ τ-bench ] ( https://github.com/sierra-research/tau-bench ) .
162+ Apache 2.0.
0 commit comments