Skip to content

Commit 0499aa6

Browse files
committed
test(ofrep): run the provider conformance suite against flagd's OFREP API
Adopts the OpenFeature provider conformance suite in the OFREP provider. No new infrastructure. flagd serves the OFREP API on port 8016 alongside its own protocols, and flagd-testbed's compose file already publishes it, so the OFREP provider runs against the existing testbed, seeded with the same canonical flag set, driven through the same launchpad control API as the flagd suites. Running two providers against one backend is the point of a cross-provider conformance suite: a difference in the results is a difference an application would see when it switches provider. tests/e2e/flagd_container.FlagdContainer would have been the natural thing to reuse and is not importable here -- a package's tests are not part of its distribution -- so tests/tck/testbed.py drives compose directly. What it duplicates is deliberately minimal: compose up, read two mapped ports, poll /readyz. This is a concrete instance of the "no shared containerised-backend helper" gap the TCK's README records. Two capabilities, both on the strength of a line of provider code rather than of a green run: OBJECT and STRICT_NUMERIC_TYPING. The same two the Go and Java OFREP adoptions reached independently, from the same architecture. Every omission is a fact about the provider. OFREPProvider is stateless -- it holds a requests.Session and a rate-limit timestamp, and nothing else survives between evaluations. It does not override initialize, so it inherits AbstractProvider's, which is `pass`, and it never emits: `_on_emit` is not called anywhere in the provider. So EVENTS, STALE and CONFIGURATION_CHANGE have nothing behind them, and UNAVAILABLE_INIT is false in the strong sense -- a provider pointed at a closed port reaches READY, because the SDK's registry dispatches PROVIDER_READY around an initialize that does nothing. events.feature and lifecycle.feature are gated at feature level and skip with their reasons; 24 of the 29 scenarios run. @lifecycle, which lands on the TCK branch this is stacked under, would also be withheld once it is available here: nothing contacts the backend before the first evaluation, so initialisation has no outcome to observe. That capability was split out of EVENTS precisely so a stateless provider can decline it accurately, and this is the case it was split out for. One scenario is marked xfail(strict=True): boolean-flag requested as an Integer. OFREP is untyped on the wire -- the request carries no type and the backend returns the JSON value regardless -- so the whole type check is the provider's, and it is isinstance(value, int), which bool is a subclass of in Python. The value True comes back with reason STATIC and no error code where the specification requires the code default and TYPE_MISMATCH. The SDK client type-checks the same way, so this is the provider-side half of open-feature/python-sdk#619 and fixing one half is not enough. Strict, so the marker fails the suite once it starts passing rather than lingering as a lie. Recorded as a finding: POST /start returns before the backend serves the flag set. The control API specifies that /start reseeds flag state; it does not specify that it returns only once that state is being served, and flagd-testbed's launchpad returns as soon as flagd answers /readyz, which is roughly 40ms before its file sources are in the flag store. The flagd suites never see this because both resolvers block inside initialize until the stream is up or the ruleset has synced, absorbing the window. A stateless provider is the first adopter with no initialisation to hide a backend's warm-up behind, and its first evaluation lands squarely in the gap -- reported, before the fix, as FLAG_NOT_FOUND on every flag. SettledControl closes it by delegating to HttpControl and then polling the public OFREP endpoint until the flag set is actually served. It manipulates nothing and weakens no scenario, but "reseeded" and "serving" should be the same instant in the control API contract, and until they are this belongs in the adoption. 23 passed, 5 skipped, 1 xfailed. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
1 parent e0b9b73 commit 0499aa6

7 files changed

Lines changed: 557 additions & 0 deletions

File tree

‎providers/openfeature-provider-ofrep/pyproject.toml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,12 +29,23 @@ Homepage = "https://github.com/open-feature/python-sdk-contrib"
2929
dev = [
3030
"coverage[toml]>=7.10.0,<8.0.0",
3131
"mypy>=1.18.0,<2.0.0",
32+
# The OpenFeature provider conformance suite. Ships the feature files, the flag
33+
# set and the control-API client, and registers its step definitions through a
34+
# pytest11 entry point, so tests/tck needs no conftest of its own for them.
35+
"openfeature-provider-tck",
3236
"poethepoet>=0.37.0",
3337
"pytest>=9.0.0,<10.0.0",
38+
"pytest-bdd>=8.1.0,<9.0.0",
3439
"requests-mock>=1.12.0,<2.0.0",
40+
# Starts the flagd testbed, which serves the OFREP API on port 8016 alongside
41+
# flagd's own protocols. See tests/tck/testbed.py.
42+
"testcontainers>=4.12.0,<5.0.0",
3543
"types-requests>=2.32.0,<3.0.0",
3644
]
3745

46+
[tool.uv.sources]
47+
openfeature-provider-tck = { workspace = true }
48+
3849
[tool.uv.build-backend]
3950
module-name = "openfeature"
4051
module-root = "src"

‎providers/openfeature-provider-ofrep/tests/tck/__init__.py‎

Whitespace-only changes.
Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
"""Session fixtures for the OFREP conformance suite, and one recorded deviation.
2+
3+
The stack is started once and never restarted, because compose assigns host
4+
ports dynamically and cannot preserve them across a restart: a restarted backend
5+
comes back on a different port, silently invalidating a provider already pointed
6+
at the old one, and the failure reads as a flaky provider rather than a broken
7+
test. Scenario isolation comes from the control API instead -- see the
8+
no-container-restart invariant in the TCK's ``control-api.yaml``.
9+
"""
10+
11+
from __future__ import annotations
12+
13+
import typing
14+
15+
import pytest
16+
17+
from openfeature.contrib.tools.provider_tck import HttpControl
18+
from tests.tck.settled_control import SettledControl
19+
from tests.tck.testbed import FlagdTestbed, running_testbed
20+
21+
22+
@pytest.fixture(scope="session")
23+
def flagd_testbed() -> typing.Iterator[FlagdTestbed]:
24+
"""The testbed stack, up for the whole session."""
25+
yield from running_testbed()
26+
27+
28+
@pytest.fixture(scope="session")
29+
def ofrep_control(flagd_testbed: FlagdTestbed) -> SettledControl:
30+
"""The control API client, pointed at the testbed's launchpad.
31+
32+
The launchpad registers only ``/start``, ``/restart``, ``/stop`` and
33+
``/change`` (flagd-testbed ``launchpad/main.go:29-32``), so ``/reset``
34+
answers 404 and every ``prepare_scenario`` takes the documented ``/start``
35+
fallback. The probe costs one 404 for the whole session.
36+
37+
Wrapped in :class:`SettledControl` because ``/start`` returns before the
38+
backend serves the flag set, and a stateless provider has no initialisation
39+
to hide that window behind. See that module -- it is a finding about the
40+
control API's guarantee, not a convenience.
41+
"""
42+
return SettledControl(
43+
HttpControl(flagd_testbed.get_launchpad_url()),
44+
flagd_testbed.get_ofrep_url(),
45+
)
46+
47+
48+
# ---------------------------------------------------------------------------
49+
# One known deviation, recorded rather than hidden.
50+
#
51+
# A conformance suite that quietly goes green on a scenario it ran and failed is
52+
# as bad as one that goes green on a scenario it skipped. So the single scenario
53+
# this provider cannot satisfy is marked xfail(strict=True), which keeps it in
54+
# the report with its reason attached and fails the suite the moment it starts
55+
# passing -- so the marker is removed when the bug is fixed rather than
56+
# lingering as a lie. Same mechanism, and same bug, as the TCK's own self-test
57+
# (tools/openfeature-provider-tck/tests/conftest.py).
58+
59+
_BOOL_AS_INT = (
60+
"test_requesting_the_wrong_type_returns_the_code_default[boolean-flag-Integer-1]"
61+
)
62+
63+
_REASON = (
64+
"bool satisfies an Integer request. OFREP is an untyped protocol -- the "
65+
"backend returns the JSON value with no knowledge of the requested type -- so "
66+
"the whole type check is the provider's, at ofrep/__init__.py:244-256: "
67+
"FlagType.INTEGER maps to `int` and the check is isinstance(value, int), which "
68+
"bool is a subclass of in Python. boolean-flag requested as an Integer "
69+
"therefore returns True with reason STATIC and no error code, where the "
70+
"specification requires the code default and TYPE_MISMATCH. The Python SDK "
71+
"client type-checks the same way, so fixing only one of the two is not enough. "
72+
"See https://github.com/open-feature/python-sdk/issues/619"
73+
)
74+
75+
76+
def pytest_collection_modifyitems(items: list[pytest.Item]) -> None:
77+
for item in items:
78+
if item.name == _BOOL_AS_INT:
79+
item.add_marker(pytest.mark.xfail(reason=_REASON, strict=True))
Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
"""``HttpControl``, plus a wait for the backend to actually serve the flag set.
2+
3+
**The problem this exists for is worth stating carefully, because it is a
4+
finding rather than a workaround.**
5+
6+
``POST /start`` is specified to reseed flag state to the named configuration's
7+
baseline (normative requirement 2 in the TCK's ``control-api.yaml``). It is not
8+
specified to *return only once that state is being served*, and flagd-testbed's
9+
launchpad does not: it returns as soon as flagd answers ``/readyz``
10+
(``launchpad/pkg/flagd.go``), which flagd does before its file sources have been
11+
loaded into the flag store. Measured against this testbed, the window is short
12+
-- around 40ms -- but it is real and reliably hit:
13+
14+
start:200 {"errorCode":"FLAG_NOT_FOUND","errorDetails":"flag `float-flag` does not exist"}
15+
start:200 {"value":0.5,"key":"float-flag","reason":"STATIC","variant":"half"}
16+
start:200 {"errorCode":"FLAG_NOT_FOUND","errorDetails":"flag `float-flag` does not exist"}
17+
18+
The flagd suites never see it, and that is the interesting part. Both flagd
19+
resolvers block inside ``initialize`` until the evaluation stream is up or the
20+
ruleset has synced, so their initialisation absorbs the window before any
21+
scenario evaluates. OFREP is stateless -- no ``initialize``, no connection, no
22+
warm-up -- so its first evaluation lands directly in the gap and the suite
23+
reports FLAG_NOT_FOUND for every flag, which reads as a catastrophically broken
24+
provider.
25+
26+
**A stateless provider is the first adopter with no initialisation to hide a
27+
backend's warm-up behind**, which makes it the one that discovers whether the
28+
control API's guarantee is strong enough. It is not: "reseeded" and "serving"
29+
need to be the same instant, or every stateless provider reimplements this. That
30+
belongs in the control API contract, and until it is there it belongs here.
31+
32+
**Why this is not cheating.** It manipulates nothing. It is a readiness probe
33+
over the same public OFREP endpoint the provider uses, on a canonical flag,
34+
asserting only that the backend has finished doing what ``/start`` already
35+
promised. No scenario is weakened, no step is bypassed, and no side channel into
36+
the backend is opened -- the normative control path is still ``HttpControl``,
37+
which this delegates to unchanged.
38+
39+
Deliberately not a :class:`ConnectionControl`: it has no ``disconnect`` or
40+
``reconnect``, matching a suite that declares neither ``STALE`` nor
41+
``UNAVAILABLE_INIT``. The two omissions keep each other honest.
42+
"""
43+
44+
from __future__ import annotations
45+
46+
import json
47+
import time
48+
import urllib.error
49+
import urllib.request
50+
51+
from openfeature.contrib.tools.provider_tck import HttpControl
52+
53+
__all__ = ["SettledControl"]
54+
55+
PROBE_FLAG_KEY = "boolean-flag"
56+
"""A canonical flag, used only to ask whether the flag set is being served yet."""
57+
58+
SETTLE_TIMEOUT_SECONDS = 15.0
59+
"""How long to wait for the backend to serve the flag set after ``/start``.
60+
61+
Two orders of magnitude above the ~40ms observed, because the cost of being
62+
generous is nothing -- the loop exits on the first success -- while the cost of
63+
being tight is a suite that fails intermittently on a loaded CI runner and gets
64+
diagnosed as a provider bug.
65+
"""
66+
67+
SETTLE_POLL_SECONDS = 0.02
68+
69+
70+
class SettledControl:
71+
"""Delegates to :class:`HttpControl`, then waits for the flags to appear."""
72+
73+
def __init__(
74+
self,
75+
control: HttpControl,
76+
ofrep_url: str,
77+
*,
78+
timeout: float = SETTLE_TIMEOUT_SECONDS,
79+
) -> None:
80+
self._control = control
81+
self._probe_url = (
82+
f"{ofrep_url.rstrip('/')}/ofrep/v1/evaluate/flags/{PROBE_FLAG_KEY}"
83+
)
84+
self._timeout = timeout
85+
86+
@property
87+
def description(self) -> str:
88+
return f"{self._control.description}, awaited through the OFREP endpoint"
89+
90+
def prepare_scenario(self) -> None:
91+
self._control.prepare_scenario()
92+
self._await_flags()
93+
94+
def change_flag(self) -> None:
95+
self._control.change_flag()
96+
97+
def _await_flags(self) -> None:
98+
"""Block until the probe flag resolves, or fail saying what was seen.
99+
100+
Raising rather than proceeding is deliberate. A scenario allowed to run
101+
against a backend that is not serving its flag set does not report a
102+
harness problem; it reports FLAG_NOT_FOUND as a conformance result,
103+
which is the one outcome a conformance suite must never produce.
104+
"""
105+
deadline = time.monotonic() + self._timeout
106+
last = "no response"
107+
108+
while time.monotonic() < deadline:
109+
status, body = self._probe()
110+
if status == 200:
111+
return
112+
last = f"HTTP {status}: {body}"
113+
time.sleep(SETTLE_POLL_SECONDS)
114+
115+
msg = (
116+
f"the backend did not serve {PROBE_FLAG_KEY!r} within {self._timeout}s of "
117+
f"a successful control-API reseed. Last response from {self._probe_url}: "
118+
f"{last}. This is a problem with the stack under test or its control API, "
119+
f"not with the provider"
120+
)
121+
raise RuntimeError(msg)
122+
123+
def _probe(self) -> tuple[int, str]:
124+
request = urllib.request.Request( # noqa: S310
125+
self._probe_url,
126+
data=json.dumps({}).encode("utf-8"),
127+
headers={"Content-Type": "application/json"},
128+
method="POST",
129+
)
130+
try:
131+
with urllib.request.urlopen(request, timeout=5.0) as response: # noqa: S310
132+
return int(response.status), ""
133+
except urllib.error.HTTPError as err:
134+
return int(err.code), err.read().decode("utf-8", "replace")[:200]
135+
except (urllib.error.URLError, OSError) as err:
136+
return 0, str(err)
Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
"""The OpenFeature provider conformance suite, run against the OFREP provider.
2+
3+
OFREP is the vendor-neutral remote evaluation protocol, so what is under test
4+
here is a pure mapping: one HTTP request per evaluation, and the translation of
5+
its JSON response -- or its error status -- into typed resolution details. There
6+
is no cache, no stream and no local ruleset, so unlike the flagd suites there is
7+
nothing here that a lifecycle could be wrong about.
8+
9+
The backend is flagd, which serves OFREP on port 8016 alongside its own
10+
protocols, driven through the same launchpad control API and seeded with the
11+
same canonical flag set as the flagd conformance suites. Running two providers
12+
against one backend is the point of a cross-provider conformance suite: a
13+
difference in the results is a difference an application would see when it
14+
switches provider.
15+
"""
16+
17+
from __future__ import annotations
18+
19+
import pytest
20+
from pytest_bdd import scenarios
21+
22+
from openfeature.contrib.provider.ofrep import OFREPProvider
23+
from openfeature.contrib.tools.provider_tck import (
24+
Capability,
25+
TckConfig,
26+
features_path,
27+
)
28+
from openfeature.provider import FeatureProvider
29+
from tests.tck.settled_control import SettledControl
30+
from tests.tck.testbed import FlagdTestbed
31+
32+
TIMEOUT_SECONDS = 10.0
33+
"""Bounds a single OFREP request.
34+
35+
Generous, because every scenario is preceded by a control-API ``/start`` that
36+
restarts the flagd process, so the first evaluation of a scenario routinely hits
37+
a backend that came up milliseconds ago. It is the only timing knob this
38+
provider has (``ofrep/__init__.py:52``); everything else the TCK offers -- event
39+
timeouts, ready timeouts -- has nothing to bound, for the reasons below.
40+
"""
41+
42+
# Every capability below is declared on the strength of a line of provider code,
43+
# not on the strength of a green run.
44+
#
45+
# OBJECT
46+
# ofrep/__init__.py:105-113 resolves structured values, and the type check at
47+
# ofrep/__init__.py:248 admits `(dict, list)` for FlagType.OBJECT -- so a JSON
48+
# object comes back as one rather than being rejected or flattened.
49+
#
50+
# STRICT_NUMERIC_TYPING
51+
# ofrep/__init__.py:250 maps FlagType.INTEGER to `int`, and the isinstance
52+
# check at ofrep/__init__.py:255 fails for the float 0.5, raising
53+
# TypeMismatchError. So float-flag's 0.5 is reported as a mismatch rather
54+
# than narrowed to 0. Worth stating plainly that this is the provider's own
55+
# doing: OFREP is untyped on the wire, the request carries no type at all,
56+
# and flagd returns 0.5 whatever was asked for -- so unlike flagd-RPC, where
57+
# the server answers INVALID_ARGUMENT, there is no backend here to catch a
58+
# numeric mismatch. Every type decision in this suite is made at those two
59+
# lines, which is also why the one deviation recorded in conftest.py lives
60+
# there.
61+
#
62+
# Not declared, and why. Each is a fact about the provider, established by
63+
# reading it -- OFREPProvider is stateless: it holds a requests.Session and a
64+
# rate-limit timestamp, and nothing else survives between evaluations.
65+
#
66+
# LIFECYCLE
67+
# Not in the Capability enum on this branch yet (it lands with
68+
# feat/provider-tck), and it would not be declared once it does.
69+
# OFREPProvider does not override `initialize`, so it inherits
70+
# AbstractProvider's, which is `pass` (python-sdk
71+
# openfeature/provider/__init__.py:138-139). Nothing contacts the backend
72+
# before the first evaluation, so initialisation has no outcome to observe.
73+
# lifecycle.feature is gated on @events at feature level on this branch and
74+
# skips for that reason; when it is retagged to @lifecycle it must keep
75+
# skipping, for this one.
76+
#
77+
# EVENTS
78+
# The provider never emits. It extends AbstractProvider, so it inherits
79+
# `attach`, but `_on_emit` is never called anywhere in
80+
# ofrep/__init__.py -- there is no stream, no poll and no background thread
81+
# to notice anything worth emitting about.
82+
#
83+
# The SDK's registry does dispatch PROVIDER_READY around `initialize` for any
84+
# provider (python-sdk openfeature/provider/_registry.py:73-77), so declaring
85+
# EVENTS would make lifecycle.feature's readiness scenario pass without
86+
# demonstrating anything -- a NoOpProvider passes it identically. That is
87+
# exactly the vacuity the @lifecycle capability was split out to end, and
88+
# claiming the capability to collect the pass would be the dishonest use of
89+
# it.
90+
#
91+
# STALE, CONFIGURATION_CHANGE
92+
# Both are event capabilities and follow from EVENTS. There is no connection
93+
# to lose -- every evaluation is an independent HTTP request -- so there is no
94+
# state between them that could go stale, and nothing watches the backend for
95+
# a configuration change. events.feature is gated @events at feature level
96+
# and skips as a whole.
97+
#
98+
# Note that a *change* is nonetheless visible to an application: the next
99+
# evaluation issues a fresh request and returns the new value. What is
100+
# missing is the signal, and the @configuration-change scenario asserts the
101+
# event as well as the behaviour, deliberately -- a provider that changes
102+
# silently is not conformant, it is just not broken.
103+
#
104+
# UNAVAILABLE_INIT
105+
# A provider pointed at a closed port reaches READY, because `initialize`
106+
# does nothing and the registry dispatches PROVIDER_READY unconditionally
107+
# (python-sdk openfeature/provider/_registry.py:73-77). The failure surfaces
108+
# on the first evaluation as GeneralError from ofrep/__init__.py:167, not as
109+
# PROVIDER_ERROR, so the scenario's premise does not hold. `TckConfig` also
110+
# rejects the capability without a `new_unavailable_provider`, and none is
111+
# supplied here for the same reason.
112+
#
113+
# TARGETING, CACHING
114+
# Reserved in the Capability enum; no scenario carries either tag. Declaring
115+
# a capability nothing exercises would be a claim with no evidence behind it.
116+
#
117+
# The result matches the Go and Java OFREP adoptions, which reached the same two
118+
# capabilities from the same architecture, independently.
119+
CAPABILITIES = frozenset(
120+
{
121+
Capability.OBJECT,
122+
Capability.STRICT_NUMERIC_TYPING,
123+
}
124+
)
125+
126+
127+
@pytest.fixture(scope="session")
128+
def tck_config(
129+
flagd_testbed: FlagdTestbed,
130+
ofrep_control: SettledControl,
131+
) -> TckConfig:
132+
"""Wire the provider up to the running testbed.
133+
134+
The port is read here, after the stack is up: compose maps host ports
135+
dynamically, so it does not exist earlier -- and it stays valid for the whole
136+
session because nothing ever restarts a container. Outages, had this suite
137+
any use for them, would be simulated inside the running stack through
138+
``ofrep_control``.
139+
"""
140+
base_url = flagd_testbed.get_ofrep_url()
141+
142+
def new_provider() -> FeatureProvider:
143+
return OFREPProvider(base_url, timeout=TIMEOUT_SECONDS)
144+
145+
return TckConfig(
146+
name="ofrep",
147+
control=ofrep_control,
148+
new_provider=new_provider,
149+
capabilities=CAPABILITIES,
150+
)
151+
152+
153+
scenarios(features_path())

0 commit comments

Comments
 (0)