Skip to content

Commit eaf98a7

Browse files
committed
test(flagd): withhold @reinitialization, and say what was measured
spec@fc99d5ac gates "A provider that was shut down can be initialized again" behind a new @reinitialization tag. Neither resolver declares it, and the reasoning is recorded next to each declaration because the arithmetic that suggested declaring it was wrong in an instructive way. The premise was that the scenario had been untagged at the previous pin, so it ran, and since it was not among the two failures it must have passed. It did not run. lifecycle.feature carries @lifecycle at the feature level, so every scenario in it inherits that tag, and the gate skips a scenario when any capability gating it is undeclared. Neither resolver declares LIFECYCLE, so the scenario is one of the six @lifecycle skips each resolver already reported -- absence from the failure list was absence from the run. Measured rather than reasoned about, by declaring LIFECYCLE and REINITIALIZATION together locally and letting the scenario run: in-process passes -- reuse genuinely works rpc fails -- boolean-flag resolves to the code default because grpc.py:420 raises "Cannot invoke RPC on closed channel!"; shutdown() closes the channel and the second initialize() does not rebuild it So the two resolvers genuinely differ, and RPC exhibits precisely the shape the specification's note on this capability describes: the provider evaluates against a closed connection rather than failing outright. Withheld for both regardless. For RPC because reuse does not work; for in-process because declaring REINITIALIZATION without LIFECYCLE leaves the scenario skipped on @lifecycle and the claim unexamined, which is the vacuous declaration this suite keeps the reserved tags out for. Whether to declare LIFECYCLE is a separate question and is not settled here. No KnownDeviation entry either way: Requirement 2.5.2 says a provider SHOULD revert to its uninitialized state and that "some providers MAY allow reinitialization", so declining reuse is a choice the specification offers rather than a requirement to deviate from. Tallies are unchanged from the previous pin, which is the check that no scenario silently changed status: rpc 29 passed / 2 failed / 9 skipped, in-process 28 / 3 / 9, nine skips per resolver being six @lifecycle and three @numeric-coercion. The failures remain large-integer-flag and huge-integer-flag, absent from flagd-testbed, plus in-process's boolean-flag-as-Float, which is #417. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
1 parent 8f81ab0 commit eaf98a7

3 files changed

Lines changed: 59 additions & 1 deletion

File tree

‎providers/openfeature-provider-flagd/tests/tck/conftest.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
``tests/e2e``.
1515
1616
**The testbed does not yet serve the whole canonical flag set.** The
17-
conformance assets at spec@ba002ce8 ask for three flags that flagd-testbed
17+
conformance assets at spec@fc99d5ac ask for three flags that flagd-testbed
1818
v3.8.0 (``openfeature/test-harness/version.txt``) does not seed:
1919
``large-integer-flag``, ``huge-integer-flag`` and ``integral-float-flag``.
2020
Until open-feature/flagd-testbed catches up, both suites fail these scenarios

‎providers/openfeature-provider-flagd/tests/tck/test_in_process_conformance.py‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,33 @@
7373
# implementation to the lossless rule; when openfeature-flagd-core follows
7474
# it, this is declared again.
7575
#
76+
# REINITIALIZATION
77+
# New at spec@fc99d5ac, which gated the scenario "A provider that was shut
78+
# down can be initialized again" that had been untagged before it. Withheld
79+
# here even though this resolver does support reuse, which is the
80+
# interesting half of the story and was measured rather than assumed:
81+
# declaring LIFECYCLE and REINITIALIZATION together locally makes the
82+
# scenario run, and in-process passes it, while RPC fails it against a
83+
# closed channel. The two resolvers genuinely differ.
84+
#
85+
# It stays withheld because declaring it would be vacuous. The scenario
86+
# lives in lifecycle.feature, which carries @lifecycle at the feature level,
87+
# so it inherits that tag and carries both; the gate skips a scenario when
88+
# any capability gating it is undeclared, and this suite does not declare
89+
# LIFECYCLE. Declaring REINITIALIZATION alone would leave the scenario
90+
# skipped on @lifecycle and the claim unexamined -- the same
91+
# declare-what-nothing-exercises error the reserved tags below are kept out
92+
# for. Declaring LIFECYCLE is a separate question from this one and is not
93+
# settled here.
94+
#
95+
# Requirement 2.5.2 says a provider SHOULD revert to its uninitialized
96+
# state and that "some providers MAY allow reinitialization", so reuse is
97+
# permitted rather than required and withholding needs no KnownDeviation.
98+
#
99+
# Worth recording that this scenario never ran here, at this pin or the one
100+
# before it: it is one of the six @lifecycle skips each resolver reports,
101+
# not a scenario that used to pass.
102+
#
76103
# TARGETING, CACHING
77104
# Reserved in the Capability enum; no scenario carries either tag. Declaring
78105
# a capability nothing exercises would be a claim with no evidence behind it,

‎providers/openfeature-provider-flagd/tests/tck/test_rpc_conformance.py‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,37 @@
8585
# coercion, tracked as open-feature/flagd#1996; this is declared again once
8686
# the testbed ships a flagd that implements it.
8787
#
88+
# REINITIALIZATION
89+
# New at spec@fc99d5ac, which gated the scenario "A provider that was shut
90+
# down can be initialized again" that had been untagged before it. Withheld
91+
# for two independent reasons, either of which is sufficient.
92+
#
93+
# First, RPC genuinely does not support reuse, which was measured rather
94+
# than reasoned about: declaring LIFECYCLE and REINITIALIZATION together
95+
# locally makes the scenario run, and it fails with `boolean-flag` resolving
96+
# to the code default because grpc.py:420 raises "Cannot invoke RPC on
97+
# closed channel!". shutdown() closes the channel and the second initialize()
98+
# does not rebuild it, so the provider evaluates against a closed connection
99+
# rather than failing outright -- exactly the shape the specification's own
100+
# note on this capability describes. Requirement 2.5.2 says a provider
101+
# SHOULD revert to its uninitialized state and that "some providers MAY
102+
# allow reinitialization", so reuse is permitted rather than required and
103+
# declining it is a choice the specification offers. Hence no
104+
# KnownDeviation entry: there is no requirement to deviate from.
105+
#
106+
# Second, and why this cannot be declared even where reuse does work: the
107+
# scenario lives in lifecycle.feature, which carries @lifecycle at the
108+
# feature level, so it inherits that tag and carries both. The gate skips a
109+
# scenario when any capability gating it is undeclared, and neither resolver
110+
# declares LIFECYCLE. Declaring REINITIALIZATION alone would leave the
111+
# scenario skipped on @lifecycle and the claim unexamined -- a vacuous
112+
# declaration of the kind the reserved tags below are kept out for.
113+
#
114+
# Worth recording that this scenario never ran here, at this pin or the one
115+
# before it: it is one of the six @lifecycle skips each resolver reports,
116+
# not a scenario that used to pass. Reading its absence from the failure
117+
# list as evidence of support is the mistake this note exists to prevent.
118+
#
88119
# TARGETING, CACHING
89120
# Reserved in the Capability enum; no scenario carries either tag. Declaring
90121
# a capability nothing exercises would be a claim with no evidence behind it,

0 commit comments

Comments
 (0)