[codex] Fix local experiment ID stability

[hassiebp]

Summary

• generate one fallback experimentId per experiment.run() invocation and reuse it across all local-data items
• keep preferring datasetRunId for dataset-backed experiments and expose the stable run identifier on ExperimentResult
• add regression coverage for local multi-item propagation and assertions for the new experimentId contract

Root Cause

For local data, runItem() generated createExperimentId() inside each item execution when no datasetRunId was available. That meant all items in the same experiment shared a runName but not a stable langfuse.experiment.id.

Impact

Local experiments now propagate one stable experiment ID across all item traces and child spans. Dataset-backed experiments continue to use the dataset run ID, and callers can inspect the stable identifier directly via result.experimentId.

Validation

• pnpm build
• pnpm exec vitest run --project=integration tests/integration/experiment-propagation.integration.test.ts
• pnpm typecheck
• pnpm exec eslint packages/client/src/experiment/ExperimentManager.ts packages/client/src/experiment/types.ts tests/integration/experiment-propagation.integration.test.ts tests/e2e/experiments.e2e.test.ts
• pnpm exec prettier --check packages/client/src/experiment/ExperimentManager.ts packages/client/src/experiment/types.ts tests/integration/experiment-propagation.integration.test.ts tests/e2e/experiments.e2e.test.ts

Notes

• This mirrors the recent Python SDK fix for unstable local experiment IDs.
• I did not run the E2E suite here because those tests require a configured Langfuse backend in the environment.

Disclaimer: Experimental PR review

Greptile Summary

This PR fixes a bug where local-data experiments generated a new experimentId for every individual item, meaning items within the same run had unrelated IDs. The fix generates one stable fallbackExperimentId at the top of run() and threads it through every runItem() call, while dataset-backed experiments continue to derive their ID from the authoritative datasetRunId.

Key changes:

• ExperimentManager.ts: createExperimentId() is called once before the batch loop; the result is passed as fallbackExperimentId to every runItem(), which uses datasetRunId || fallbackExperimentId. The datasetRunId extraction is also improved from itemResults[0].datasetRunId (fragile when the first item is skipped) to itemResults.find(item => item.datasetRunId)?.datasetRunId.
• types.ts: ExperimentResult gains a required experimentId: string field with clear documentation about the dataset-vs-local semantics.
• Tests: Integration test renamed and updated to assert shared IDs; four new E2E assertions cover the local, dataset, custom-name, and empty-data cases.

Confidence Score: 5/5

Safe to merge — the fix is minimal, well-tested, and introduces no breaking changes to the public API (the new experimentId field is additive).

No P0 or P1 issues found. The core logic change is a one-liner substitution (params.fallbackExperimentId instead of await createExperimentId()). The datasetRunId extraction improvement with find() is strictly safer than the previous [0] access. Both integration and E2E tests cover the new contract thoroughly.

No files require special attention.

Important Files Changed

Filename	Overview
packages/client/src/experiment/ExperimentManager.ts	Generates one stable fallbackExperimentId per run() invocation and threads it through all runItem() calls; also improves datasetRunId extraction with find() instead of [0]
packages/client/src/experiment/types.ts	Adds the new experimentId: string field to ExperimentResult with clear JSDoc explaining the dataset vs local-data semantics
tests/integration/experiment-propagation.integration.test.ts	Updates the multi-item test to assert that all local items now share a single experiment ID while still keeping distinct item IDs; adds result.experimentId assertion
tests/e2e/experiments.e2e.test.ts	Adds experimentId assertions across all four test scenarios: local datasets, dataset-backed runs, custom run names, and empty data

Sequence Diagram

sequenceDiagram
    participant Caller
    participant RunMethod as ExperimentManager run
    participant RunItem as ExperimentManager runItem
    participant API as Langfuse API

    Caller->>RunMethod: experiment.run(config)
    RunMethod->>RunMethod: createExperimentRunName()
    RunMethod->>RunMethod: fallbackExperimentId = await createExperimentId()

    loop for each item in batch
        RunMethod->>RunItem: params including fallbackExperimentId
        alt DatasetItem has datasetItemId
            RunItem->>API: datasetRunItems.create(...)
            API-->>RunItem: datasetRunId
            RunItem->>RunItem: experimentId = datasetRunId
        else Local item
            RunItem->>RunItem: experimentId = fallbackExperimentId
        end
        RunItem->>RunItem: propagateAttributes with experimentId
        RunItem-->>RunMethod: ExperimentItemResult with optional datasetRunId
    end

    RunMethod->>RunMethod: datasetRunId = itemResults.find first with datasetRunId
    RunMethod->>RunMethod: experimentId = datasetRunId OR fallbackExperimentId
    RunMethod-->>Caller: ExperimentResult with stable experimentId

Loading

_{Reviews (1): Last reviewed commit: "fix local experiment ID stability" | Re-trigger Greptile}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
langfuse-js	Ready	Preview	Apr 1, 2026 9:07am

[github-actions]

@claude review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6cb137e762

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[claude]

LGTM — clean, well-scoped bug fix.

Extended reasoning...

Overview

This PR touches ExperimentManager.ts, types.ts, and two test files. It moves createExperimentId() from per-item call sites inside runItem() up to a single call at the top of run(), then threads the result (fallbackExperimentId) down through params. The stable ID is also exposed on ExperimentResult as experimentId.

Security risks

None. The change only affects internal ID generation and result shape — no auth, crypto, permissions, or external data paths are involved.

Level of scrutiny

Low. This is a straightforward refactor with a clear root cause and minimal blast radius. The only observable behavior change is that local-experiment items now share a single experimentId (the intended fix), and callers gain a new experimentId field on the result object (purely additive, no breaking change). The datasetRunId retrieval improvement from [0] to .find() is a minor robustness improvement that handles the case where the first item may have been skipped.

Other factors

The integration test directly verifies the fix (experimentIds[0] === experimentIds[1] and result.experimentId === experimentIds[0]). E2E tests add contract assertions for the new field across all relevant scenarios (local, dataset-backed, empty dataset, custom runName). Build, typecheck, lint, and integration test validation are all reported as passing.