Skip to content

Commit 4beb5a6

Browse files
docs: update observations pagination and batch evaluation docs (#2098)
Since langfuse/langfuse#19043, observations v2 pages can be short or empty while meta.cursor is set, a request can take up to 45 s, and queries without a lower start-time bound stop at 2020-01-01. Document how to read every page in the Langfuse.api docs, and correct batch evaluation docs that claimed filter=None fetches everything, that more items definitely exist, and that all fetch failures are retried. Replace the resume example that looped forever on a persistent fetch failure, and fix the inclusive resume timestamp description. Co-authored-by: Hassieb Pakzad <68423100+hassiebp@users.noreply.github.com>
1 parent 8faf84d commit 4beb5a6

2 files changed

Lines changed: 63 additions & 26 deletions

File tree

‎langfuse/_client/client.py‎

Lines changed: 46 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -444,9 +444,34 @@ def api(self) -> LangfuseAPI:
444444
user, session, tags and input/output are on its root observation. Read
445445
scores with `api.scores.get_many` and experiments with `api.experiments`.
446446
447+
- **Read observation pages until `meta.cursor` is absent.** Pages can be
448+
short or empty before the end. The server scans for up to 45 seconds per
449+
request, so pass `request_options={"timeout_in_seconds": 60}`, and pass
450+
`from_start_time` when you know a lower bound (otherwise the server
451+
scans back to 2020-01-01). A 422 means the request hit a server resource
452+
limit: narrow the time range rather than retrying unchanged. A `startTime`
453+
condition in the JSON `filter` overrides `from_start_time` and
454+
`to_start_time`.
455+
447456
- For large-scale aggregation (usage/cost by model, user, etc.), prefer the
448457
v2 Metrics API (`api.metrics.metrics(...)`) over paginating row-level data.
449458
459+
Reading all observations of a trace:
460+
```python
461+
observations, cursor = [], None
462+
while True:
463+
page = langfuse.api.observations.get_many(
464+
trace_id=trace_id,
465+
from_start_time=since,
466+
cursor=cursor,
467+
request_options={"timeout_in_seconds": 60},
468+
)
469+
observations.extend(page.data)
470+
cursor = page.meta.cursor
471+
if cursor is None:
472+
break
473+
```
474+
450475
451476
See also: `async_api`,
452477
https://langfuse.com/docs/api-and-data-platform/features/query-via-sdk
@@ -2226,8 +2251,8 @@ def flush(self) -> None:
22262251
*readable* yet: server-side ingestion is asynchronous, so flushed data
22272252
may not be queryable for 15-30 seconds —
22282253
`api.observations.get_many(trace_id=...)` may return empty results right
2229-
after a successful flush. See the `api` property docs for a bounded retry
2230-
pattern, or
2254+
after a successful flush. Poll with a deadline instead of a fixed sleep;
2255+
see
22312256
https://langfuse.com/docs/api-and-data-platform/features/query-via-sdk#ingestion-lag
22322257
"""
22332258
if self._resources is not None:
@@ -3196,7 +3221,9 @@ def run_batched_evaluation(
31963221
- '[{"type": "arrayOptions", "column": "tags", "operator": "any of", "value": ["production"]}]'
31973222
- '[{"type": "string", "column": "traceName", "operator": "=", "value": "chat"}]'
31983223
- '[{"type": "datetime", "column": "startTime", "operator": ">=", "value": "2026-01-01T00:00:00Z"}]'
3199-
Default: None (fetches all observations).
3224+
Without a `startTime` lower bound, the server only reads back to
3225+
2020-01-01, and a selective filter can take many slow requests to get
3226+
there. Add a `startTime` lower bound when you can. Default: None.
32003227
fetch_batch_size: Number of items to fetch per API call and hold in memory.
32013228
Larger values may be faster but use more memory. Maximum 1000. Default: 50.
32023229
fields: Comma-separated list of observation field groups to fetch. Available
@@ -3216,8 +3243,9 @@ def run_batched_evaluation(
32163243
combined metrics. Default: None.
32173244
metadata: Optional metadata dict to add to all created scores. Useful for
32183245
tracking evaluation runs, versions, or other context. Default: None.
3219-
max_retries: Maximum number of retry attempts for failed batch fetches.
3220-
Default: 3.
3246+
max_retries: Maximum retries of a batch fetch on HTTP 408, 409, 429 and
3247+
5xx responses. Other fetch errors, such as timeouts or a 422, stop
3248+
the run with `completed=False` and a resume token. Default: 3.
32213249
verbose: If True, logs progress information to console. Useful for monitoring
32223250
long-running evaluations. Default: False.
32233251
resume_from: Optional resume token from a previous run that stopped early
@@ -3236,12 +3264,12 @@ def run_batched_evaluation(
32363264
- total_evaluations_failed: Individual evaluator failures
32373265
- evaluator_stats: Per-evaluator statistics (success rate, scores created)
32383266
- resume_token: Token for continuing the run (set after a fetch failure
3239-
or when max_items was reached while more items exist)
3267+
or when max_items was reached while more items may exist)
32403268
- completed: False if the run stopped because a fetch failed
32413269
- duration_seconds: Total execution time
32423270
- failed_item_ids: Observation IDs of items that failed
32433271
- error_summary: Error types and counts
3244-
- has_more_items: True if max_items reached but more exist
3272+
- has_more_items: True if max_items was reached and more items may exist
32453273
- item_evaluations: Evaluations per observation ID
32463274
32473275
Raises:
@@ -3317,27 +3345,34 @@ def composite_evaluator(*, input, output, expected_output, metadata, evaluations
33173345
)
33183346
```
33193347
3320-
Continuing a run that stopped early:
3348+
Processing in chunks of `max_items`:
33213349
```python
33223350
result = client.run_batched_evaluation(
33233351
mapper=generation_mapper,
33243352
evaluators=[accuracy_evaluator],
33253353
max_items=10000,
33263354
)
33273355
3328-
while result.resume_token:
3356+
while result.has_more_items:
33293357
result = client.run_batched_evaluation(
3330-
mapper=generation_mapper,
3358+
mapper=generation_mapper,
33313359
evaluators=[accuracy_evaluator],
33323360
max_items=10000,
33333361
resume_from=result.resume_token,
33343362
)
3363+
3364+
if not result.completed:
3365+
# resume_from=result.resume_token retries the failed page. A 422
3366+
# usually repeats, and a token only resumes its own filter, so
3367+
# start a new run whose filter adds a narrower startTime range.
3368+
...
33353369
```
33363370
33373371
Note:
33383372
- Evaluator failures are logged but don't stop the batch evaluation
33393373
- Individual item failures are tracked but don't stop processing
3340-
- Fetch failures are retried up to `max_retries` times
3374+
- Fetches are retried up to `max_retries` times on HTTP 408, 409, 429
3375+
and 5xx responses; other fetch errors stop the run
33413376
- All scores are automatically flushed to Langfuse at the end
33423377
- Resuming uses the pagination cursor, so items are neither skipped nor
33433378
evaluated twice

‎langfuse/batch_evaluation.py‎

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -471,17 +471,18 @@ class BatchEvaluationResumeToken:
471471
exactly where the previous one stopped, without re-evaluating or skipping
472472
items, even if new observations were ingested in the meantime.
473473
474-
A token is returned when a run stops because a batch fetch failed after all
475-
retries (`completed=False`) or because `max_items` was reached while more
476-
items exist (`has_more_items=True`).
474+
A token is returned when a run stops because a batch fetch failed
475+
(`completed=False`) or because `max_items` was reached while more items may
476+
exist (`has_more_items=True`).
477477
478478
Attributes:
479479
filter: The original JSON filter string used to query items. Pass the
480480
same filter when resuming.
481481
cursor: Cursor of the next page to fetch. None if no page was fetched yet.
482482
last_processed_timestamp: ISO 8601 start time of the oldest processed
483483
observation. Only used to resume when `cursor` is None, by fetching
484-
observations that started strictly before this timestamp.
484+
observations that started at or before this timestamp, skipping
485+
`last_processed_id`.
485486
last_processed_id: The ID of the last processed observation, for reference.
486487
last_page_ids: IDs of the observations on the last processed page. A
487488
resumed run skips these, because unmerged duplicate rows of one
@@ -572,13 +573,13 @@ class BatchEvaluationResult:
572573
evaluator_stats: List of per-evaluator statistics (success/failure rates, scores created).
573574
resume_token: Token for continuing the run. Set when a batch fetch failed
574575
(`completed=False`) or when `max_items` was reached while more items
575-
exist (`has_more_items=True`); None otherwise.
576+
may exist (`has_more_items=True`); None otherwise.
576577
completed: False if the run stopped because a batch fetch failed, True otherwise
577578
(including when it stopped at `max_items`).
578579
duration_seconds: Total time taken to execute the batch evaluation.
579580
failed_item_ids: List of observation IDs for items that failed evaluation.
580581
error_summary: Dictionary mapping error types to occurrence counts.
581-
has_more_items: True if max_items limit was reached but more items exist.
582+
has_more_items: True if the run stopped at max_items and more items may exist.
582583
item_evaluations: Dictionary mapping observation IDs to their evaluation results (both regular and composite).
583584
584585
Examples:
@@ -632,7 +633,7 @@ class BatchEvaluationResult:
632633
print("Pass it as resume_from to continue")
633634
634635
if result.has_more_items:
635-
print(f"ℹ️ More items available beyond max_items limit")
636+
print(f"ℹ️ More items may be available beyond max_items limit")
636637
```
637638
638639
Performance monitoring:
@@ -691,7 +692,7 @@ def __init__(
691692
duration_seconds: Total execution time.
692693
failed_item_ids: IDs of failed items.
693694
error_summary: Error types and counts.
694-
has_more_items: Whether more items exist beyond max_items.
695+
has_more_items: Whether more items may exist beyond max_items.
695696
item_evaluations: Dictionary mapping item IDs to their evaluation results.
696697
697698
Note:
@@ -790,7 +791,7 @@ def __str__(self) -> str:
790791
lines.append(" Use resume_from parameter to continue")
791792

792793
if self.has_more_items:
793-
lines.append("\nNote: More items available beyond max_items limit")
794+
lines.append("\nNote: More items may be available beyond max_items limit")
794795

795796
lines.append("=" * 60)
796797
return "\n".join(lines)
@@ -855,7 +856,8 @@ async def run_async(
855856
metadata: Metadata to add to all created scores.
856857
_add_observation_scores_to_trace: Private option to duplicate
857858
observation-level scores onto the parent trace.
858-
max_retries: Maximum retries for failed batch fetches.
859+
max_retries: Maximum retries of a batch fetch on HTTP 408, 409, 429
860+
and 5xx responses. Other fetch errors are not retried.
859861
verbose: If True, log progress to console.
860862
resume_from: Resume token from a previous run. If `filter` is omitted,
861863
the token's filter is reused.
@@ -1104,15 +1106,15 @@ async def _fetch_batch_with_retry(
11041106
filter: JSON filter string for querying.
11051107
cursor: Cursor returned with the previous page; None for the first page.
11061108
limit: Number of items to request.
1107-
max_retries: Maximum number of retry attempts.
1109+
max_retries: Maximum retries on HTTP 408, 409, 429 and 5xx responses.
11081110
fields: Comma-separated v2 field groups to include.
11091111
11101112
Returns:
1111-
Tuple of the page's observations and the cursor for the next page
1112-
(None when there are no more pages).
1113+
Tuple of the page's observations, which may be empty, and the cursor
1114+
for the next page (None when there are no more pages).
11131115
11141116
Raises:
1115-
Exception: If all retry attempts fail.
1117+
Exception: If the fetch fails.
11161118
"""
11171119
response = await asyncio.to_thread(
11181120
self.client.api.observations.get_many,
@@ -1481,7 +1483,7 @@ def _build_result(
14811483
start_time: Start time (unix timestamp).
14821484
failed_item_ids: IDs of failed items.
14831485
error_summary: Error type counts.
1484-
has_more_items: Whether more items exist beyond max_items.
1486+
has_more_items: Whether more items may exist beyond max_items.
14851487
item_evaluations: Dictionary mapping item IDs to their evaluation results.
14861488
14871489
Returns:

0 commit comments

Comments
 (0)