[hassiebp]

What does this PR do?

Quarantines the three tests that fail intermittently in CI for reasons outside the SDK. They still run, but are marked as non-strict xfail: a failure shows as XFAIL and a pass as XPASS, and neither can turn CI red.

Test	Isolated failures (all time)	Same commit also passed	Failures since Oct 10	Failure mode
test_openai.py::test_audio_input_and_output	18	6	3 of 95	OpenAI audio call fails
test_langchain.py::test_multimodal	8	0	3 of 95	OpenAI call hangs until the 180 s timeout and crashes the xdist worker
test_openai.py::test_base_64_image_input	4	3	1 of 95	OpenAI image input fails

How the tests were picked. I parsed the failed test names from every CI log over the last three months (Jul 27 to Oct 22), then removed the noise:

• Fork and dependabot PRs, which have no provider secrets.
• Batches of more than 10 simultaneous failures, which are environment breakages.

A test only counts as flaky if it either passed and failed on the same commit, or failed in isolation across several unrelated branches.

Mechanism. QUARANTINED_NODEIDS in tests/conftest.py mirrors the existing SERIAL_E2E_NODEIDS. Quarantined tests also get a 60 s signal-based timeout. A hang then raises inside the test and counts as XFAIL, instead of the suite's thread-based timeout killing the pytest-xdist worker, which xdist reports as a hard failure.

Considered but not quarantined:

• Four test_decorators.py tests asserting total_cost > 0 (test_nested_observations and three siblings). They failed about 4% of the time until Oct 10 (9 of roughly 200 runs) and 0 of roughly 96 since. That looks fixed upstream in the Langfuse server, which CI pulls from HEAD. The test still races the server's asynchronous cost calculation, so it may need a better wait condition later.
• test_openai_function_call_streamed. Two OpenAI 500 errors on one commit, which could hit any provider test.
• Deterministic waves, which are real breakages rather than flakes: the scores_v3 API change, the KeyError: 'response_format' SDK change, KeyError: 'context' in the media tests, and the retired gpt-3.5-turbo-instruct model.

Type of change

• Bug fix
• New feature
• Breaking change
• Refactor
• Documentation update
• Tooling, CI, or repo maintenance

Verification

uv run --frozen pytest --collect-only -q -m xfail tests/live_provider tests/e2e
# 3/232 tests collected: exactly the three quarantined node ids
uv run --frozen pytest -n auto -q tests/unit
# 963 passed, 2 skipped
uv run --frozen ruff check tests/conftest.py && uv run --frozen ruff format --check tests/conftest.py
# All checks passed! / 1 file already formatted

I prototyped the mechanism in a scratch directory under xdist, with --timeout-method=thread as in CI. A hanging test, a failing test and a passing test reported 2 xfailed, 1 xpassed with exit code 0.

Checklist

• I self-reviewed the diff using code_review.md.
• I added or updated tests for behavior changes.
• I updated docs, examples, or .env.template if needed.
• I did not hand-edit generated files; if generated files changed, I used the upstream regeneration path.
• I did not commit secrets or credentials.

  

Confidence Score: 5/5

The PR appears safe to merge.

Summary

Quarantines three flaky live-provider tests without skipping them.

• Three live-provider tests no longer fail CI on flaky results.

The supplied PR description by hassiebp explicitly acknowledges that suppressing failures in these three tests is intentional.

_{Reviews (1) · Last reviewed commit: "test: quarantine flaky live-provider tes..." · Reviewed by Greptile}

[github-actions]

@claude review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-23T12:45:42.246622Z	cf716ad	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[claude]

Claude Code Review

No review was started: this request came from a bot account. Manual reviews can only be requested by someone with write access to this repository. Ask a maintainer to comment @claude review, or have your automation post the comment from a user account with write access.

_{Tip: disable this comment in your organization's Code Review settings.}

[claude]

Code review found no issues

No high-confidence issues detected in this change.