Skip to content

Commit 87f2aad

Browse files
ci: skip live-provider tests on fork pull requests (#2036)
* test: move OpenAI-backed decorator tests into live_provider suite Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * ci: skip live-provider tests on fork pull requests Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
1 parent 0658c20 commit 87f2aad

5 files changed

Lines changed: 239 additions & 187 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,23 +152,34 @@ jobs:
152152
# SERPAPI_API_KEY: ${{ secrets.SERPAPI_API_KEY }}
153153
HUGGINGFACEHUB_API_TOKEN: ${{ secrets.HUGGINGFACEHUB_API_TOKEN }}
154154
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
155+
# GitHub withholds secrets from fork PRs, so provider calls there can only fail.
156+
# The suite still runs on same-repo PRs and on main, where a missing key fails loudly.
157+
SKIP_LIVE_PROVIDER: ${{ matrix.suite == 'live_provider' && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository }}
155158

156159
name: ${{ matrix.job_name }}
157160
steps:
161+
- name: Skip live-provider tests on fork PRs
162+
if: ${{ env.SKIP_LIVE_PROVIDER == 'true' }}
163+
run: echo "::notice::Live-provider tests are skipped on fork PRs because provider secrets are unavailable. They run after merge on main."
158164
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
165+
if: ${{ env.SKIP_LIVE_PROVIDER != 'true' }}
159166
with:
160167
persist-credentials: false
161168
- name: Install uv and set Python version
169+
if: ${{ env.SKIP_LIVE_PROVIDER != 'true' }}
162170
uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
163171
with:
164172
version: "0.11.2"
165173
python-version: "3.13"
166174
enable-cache: true # zizmor: ignore[cache-poisoning] CI-only, no artifacts published
167175
- name: Install the project dependencies
176+
if: ${{ env.SKIP_LIVE_PROVIDER != 'true' }}
168177
run: uv sync --locked
169178
- name: Check uv Python version
179+
if: ${{ env.SKIP_LIVE_PROVIDER != 'true' }}
170180
run: uv run --frozen python --version
171181
- name: Prepare langfuse server compose
182+
if: ${{ env.SKIP_LIVE_PROVIDER != 'true' }}
172183
run: |
173184
mkdir -p ./langfuse-server
174185
LANGFUSE_SERVER_SHA="$(git ls-remote https://github.com/langfuse/langfuse.git HEAD | cut -f1)"
@@ -177,6 +188,7 @@ jobs:
177188
echo "${LANGFUSE_SERVER_SHA}"
178189
179190
- name: Run langfuse server
191+
if: ${{ env.SKIP_LIVE_PROVIDER != 'true' }}
180192
run: |
181193
cd ./langfuse-server
182194
@@ -194,6 +206,7 @@ jobs:
194206
echo "::endgroup::"
195207
196208
- name: Health check for langfuse server
209+
if: ${{ env.SKIP_LIVE_PROVIDER != 'true' }}
197210
run: |
198211
echo "Checking if the langfuse server is up..."
199212
retry_count=0
@@ -264,7 +277,7 @@ jobs:
264277
fi
265278
266279
- name: Run live-provider tests
267-
if: ${{ matrix.suite == 'live_provider' }}
280+
if: ${{ matrix.suite == 'live_provider' && env.SKIP_LIVE_PROVIDER != 'true' }}
268281
run: |
269282
uv run --frozen python --version
270283
uv run --frozen pytest \

‎AGENTS.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,8 @@ Minimum verification matrix:
100100

101101
### `tests/live_provider`
102102

103-
- This suite uses real provider calls and always runs as one dedicated CI suite.
103+
- This suite uses real provider calls and runs as one dedicated CI suite. CI skips it on fork PRs, which have no provider secrets.
104+
- Any test that calls a model provider belongs here, never in `tests/e2e`, so fork PRs stay green on the e2e shards.
104105
- Do not split or shard `tests/live_provider` into separate smoke and extended jobs unless the team explicitly changes that policy.
105106
- Keep assertions focused on stable provider-facing behavior rather than brittle observation counts.
106107

@@ -111,7 +112,7 @@ The main CI workflow currently runs:
111112
- mypy on Python 3.13
112113
- `tests/unit` on a Python 3.11-3.15 matrix
113114
- `tests/e2e` in 2 mechanical shards plus a serial subset inside each shard
114-
- `tests/live_provider` as one always-on suite
115+
- `tests/live_provider` as one suite, skipped on fork PRs because GitHub withholds secrets there
115116
- PR title validation for Conventional Commits
116117

117118
If you change the e2e split:

‎CONTRIBUTING.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@ Live-provider tests make real provider calls and require provider API keys:
4949
uv run --frozen pytest -n 4 --dist worksteal tests/live_provider -m "live_provider"
5050
```
5151

52+
GitHub does not expose repository secrets to pull requests from forks, so CI skips the live-provider suite on those PRs. It runs on same-repo PRs and again after merge on `main`. Any test that calls a model provider belongs in `tests/live_provider`, not `tests/e2e`.
53+
5254
Run a specific test with:
5355

5456
```bash

‎tests/e2e/test_decorators.py‎

Lines changed: 0 additions & 184 deletions
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,11 @@
66
from typing import Optional
77

88
import pytest
9-
from langchain_core.prompts import ChatPromptTemplate
10-
from langchain_openai import ChatOpenAI
119
from opentelemetry import trace
1210

1311
from langfuse import Langfuse, get_client, observe, propagate_attributes
1412
from langfuse._client.environment_variables import LANGFUSE_PUBLIC_KEY
1513
from langfuse._client.resource_manager import LangfuseResourceManager
16-
from langfuse.langchain import CallbackHandler
1714
from langfuse.media import LangfuseMedia
1815
from tests.support.utils import (
1916
get_observations,
@@ -389,88 +386,6 @@ def level_1_function(*args, **kwargs):
389386
assert level_3_observation.total_cost > 0
390387

391388

392-
def test_decorators_langchain():
393-
mock_name = "test_decorators_langchain"
394-
langfuse = get_client()
395-
mock_trace_id = langfuse.create_trace_id()
396-
397-
@observe()
398-
def langchain_operations(*args, **kwargs):
399-
# Get langfuse callback handler for LangChain
400-
handler = CallbackHandler()
401-
prompt = ChatPromptTemplate.from_template("tell me a short joke about {topic}")
402-
model = ChatOpenAI(temperature=0)
403-
404-
chain = prompt | model
405-
406-
return chain.invoke(
407-
{"topic": kwargs["topic"]},
408-
config={
409-
"callbacks": [handler],
410-
},
411-
)
412-
413-
@observe()
414-
def level_3_function(*args, **kwargs):
415-
langfuse.update_current_span(metadata=mock_metadata)
416-
langfuse.update_current_span(metadata=mock_deep_metadata)
417-
return langchain_operations(*args, **kwargs)
418-
419-
@observe()
420-
def level_2_function(*args, **kwargs):
421-
langfuse.update_current_span(metadata=mock_metadata)
422-
423-
return level_3_function(*args, **kwargs)
424-
425-
@observe()
426-
def level_1_function(*args, **kwargs):
427-
return level_2_function(*args, **kwargs)
428-
429-
with propagate_attributes(session_id=mock_session_id, trace_name=mock_name):
430-
level_1_function(topic="socks", langfuse_trace_id=mock_trace_id)
431-
432-
langfuse.flush()
433-
434-
trace_data = wait_for_trace_snapshot(
435-
mock_trace_id,
436-
is_result_ready=lambda trace: (
437-
trace.session_id == mock_session_id
438-
and trace.name == mock_name
439-
and {
440-
"level_1_function",
441-
"level_2_function",
442-
"level_3_function",
443-
"langchain_operations",
444-
"ChatPromptTemplate",
445-
}.issubset({observation.name for observation in trace.observations})
446-
),
447-
)
448-
assert len(trace_data.observations) > 2
449-
450-
# trace parameters if set anywhere in the call stack
451-
assert trace_data.session_id == mock_session_id
452-
assert trace_data.name == mock_name
453-
454-
level_1_observation = _get_observation_by_name(trace_data, "level_1_function")
455-
level_2_observation = _get_observation_by_name(trace_data, "level_2_function")
456-
level_3_observation = _get_observation_by_name(trace_data, "level_3_function")
457-
langchain_observation = _get_observation_by_name(trace_data, "langchain_operations")
458-
prompt_observation = _get_observation_by_name(trace_data, "ChatPromptTemplate")
459-
460-
assert level_1_observation.name == "level_1_function"
461-
assert _is_descendant(trace_data, level_2_observation.id, level_1_observation.id)
462-
assert level_2_observation.name == "level_2_function"
463-
assert level_2_observation.metadata["key"] == mock_metadata["key"]
464-
assert _is_descendant(trace_data, level_3_observation.id, level_2_observation.id)
465-
assert level_3_observation.name == "level_3_function"
466-
assert level_3_observation.metadata["key"] == mock_deep_metadata["key"]
467-
assert _is_descendant(trace_data, langchain_observation.id, level_3_observation.id)
468-
assert langchain_observation.name == "langchain_operations"
469-
470-
# Check that LangChain components are captured
471-
assert _is_descendant(trace_data, prompt_observation.id, langchain_observation.id)
472-
473-
474389
def test_get_current_trace_url():
475390
langfuse = get_client()
476391
mock_trace_id = langfuse.create_trace_id()
@@ -860,105 +775,6 @@ async def main_async(**kwargs):
860775
assert nested_observation.output == "Hello--, async --World!"
861776

862777

863-
@pytest.mark.asyncio
864-
async def test_async_nested_openai_chat_stream():
865-
from langfuse.openai import AsyncOpenAI
866-
867-
mock_name = "test_async_nested_openai_chat_stream"
868-
langfuse = get_client()
869-
mock_trace_id = langfuse.create_trace_id()
870-
mock_tags = ["tag1", "tag2"]
871-
mock_session_id = "session-id-1"
872-
mock_user_id = "user-id-1"
873-
874-
@observe(capture_output=False)
875-
async def level_2_function():
876-
gen = await AsyncOpenAI().chat.completions.create(
877-
model="gpt-3.5-turbo",
878-
messages=[{"role": "user", "content": "1 + 1 = "}],
879-
temperature=0,
880-
metadata={"someKey": "someResponse"},
881-
stream=True,
882-
)
883-
884-
with propagate_attributes(
885-
session_id=mock_session_id,
886-
user_id=mock_user_id,
887-
tags=mock_tags,
888-
trace_name=mock_name,
889-
):
890-
async for c in gen:
891-
print(c)
892-
893-
langfuse.update_current_span(metadata=mock_metadata)
894-
895-
return "level_2"
896-
897-
@observe()
898-
async def level_1_function(*args, **kwargs):
899-
await level_2_function()
900-
901-
return "level_1"
902-
903-
result = await level_1_function(
904-
*mock_args, **mock_kwargs, langfuse_trace_id=mock_trace_id
905-
)
906-
langfuse.flush()
907-
908-
assert result == "level_1" # Wrapped function returns correctly
909-
910-
# ID setting for span or trace
911-
trace_data = wait_for_trace_snapshot(
912-
mock_trace_id,
913-
is_result_ready=lambda trace: (
914-
trace.session_id == mock_session_id
915-
and trace.name == mock_name
916-
and {
917-
"level_1_function",
918-
"level_2_function",
919-
"OpenAI-generation",
920-
}.issubset({observation.name for observation in trace.observations})
921-
),
922-
)
923-
assert len(trace_data.observations) >= 3
924-
925-
# trace parameters if set anywhere in the call stack
926-
assert trace_data.session_id == mock_session_id
927-
assert trace_data.name == mock_name
928-
929-
level_1_observation = _get_observation_by_name(trace_data, "level_1_function")
930-
level_2_observation = _get_observation_by_name(trace_data, "level_2_function")
931-
level_3_observation = _get_observation_by_name(trace_data, "OpenAI-generation")
932-
assert _is_descendant(trace_data, level_2_observation.id, level_1_observation.id)
933-
assert _is_descendant(trace_data, level_3_observation.id, level_2_observation.id)
934-
935-
assert level_2_observation.metadata["key"] == mock_metadata["key"]
936-
937-
generation = level_3_observation
938-
939-
assert generation.name == "OpenAI-generation"
940-
assert generation.metadata["someKey"] == "someResponse"
941-
assert generation.input == [{"content": "1 + 1 = ", "role": "user"}]
942-
assert generation.type == "GENERATION"
943-
assert "gpt-3.5-turbo" in generation.model
944-
assert generation.start_time is not None
945-
assert generation.end_time is not None
946-
assert generation.start_time < generation.end_time
947-
assert generation.model_parameters == {
948-
"service_tier": "default",
949-
"temperature": 0,
950-
"top_p": 1,
951-
"frequency_penalty": 0,
952-
"max_tokens": "Infinity",
953-
"presence_penalty": 0,
954-
}
955-
assert generation.usage_details["input"] is not None
956-
assert generation.usage_details["output"] is not None
957-
assert generation.usage_details["total"] is not None
958-
print(generation)
959-
assert generation.output == "2"
960-
961-
962778
def test_generator_as_function_input():
963779
langfuse = get_client()
964780
mock_trace_id = langfuse.create_trace_id()

0 commit comments

Comments
 (0)