fix(provider-tck): read the canonical count from the assets, not from a test

The base branch followed the conformance assets to open-feature/spec@15fe861,
which takes the canonical set from twenty-nine scenarios to forty. Three
expectations here restated the old number and one declaration claimed a
capability the provider behind InProcessControl does not have.

- test_canonical_set derived nothing: two message assertions and one length
  assertion each named 29. The count now comes from canonical_scenarios() --
  the same parse the guard itself runs -- so adding a canonical scenario does
  not require editing a test that exists to notice the addition. Appendix F
  makes that the rule, and a suite whose own tests have to be edited to follow
  the assets is a suite that will be edited to agree with them.
- test_the_canonical_set_is_read_from_the_packaged_assets would have become
  vacuous if it compared canonical_scenarios() to itself, so it is measured
  against what pytest-bdd actually collected in the unfiltered run, less the
  three scenarios the generated extension contributes. That pins the parse to
  the runner rather than to a literal.
- The generated adoptions in test_report declared NUMERIC_COERCION. The two
  lossless scenarios the new assets add are exactly the ones that catch a
  provider which does not coerce, and the provider behind InProcessControl does
  not: they failed. LARGE_INTEGERS is declared instead -- honest for a Python
  int, and still a third declared capability, which is all those tests needed.
  The declaration is corrected rather than the assertion weakened; finding 3
  in the README is the same call for the in-memory self-tests.

The README's figures follow: the self-test total, the example report line, the
per-step status counts and the partial-run transcript are all re-measured.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Author: aepfli

tools/openfeature-provider-tck/README.md

@@ -363,20 +363,20 @@ at `<dir>/<name>.ndjson`, which is a [Cucumber Messages][messages] stream.
 
 ```console
 $ PROVIDER_TCK_REPORT_DIR=./reports pytest
-provider-tck [in-memory]: report written to reports/in-memory.json with results in in-memory.ndjson (1 failed, 23 passed, 5 skipped)
+provider-tck [in-memory]: report written to reports/in-memory.json with results in in-memory.ndjson (1 failed, 28 passed, 11 skipped)
 
 $ jq -c .results reports/in-memory.json
 {"format":"cucumber-messages","location":"in-memory.ndjson","digest":"sha256:c7e12a…"}
 
 $ jq -r 'select(.testStepFinished) | .testStepFinished.testStepResult.status' \
     reports/in-memory.ndjson | sort | uniq -c
       1 FAILED
-    220 PASSED
-     45 SKIPPED
+    281 PASSED
+     91 SKIPPED
 ```
 
-Statuses are per step, not per scenario. Of the 45 skipped, 42 belong to the five scenarios the
-capability gate stopped — their before-hooks included, which is where the reason is — and three are
+Statuses are per step, not per scenario. Of the 91 skipped, 88 belong to the eleven scenarios the
+capability gate stopped — every step after the before-hook that carries the reason — and three are
 the steps of the failing scenario that were never reached.
 
 It is an environment variable rather than a `TckConfig` field so that emitting a report is a property
@@ -397,11 +397,11 @@ execute all of them writes no report:
 
 ```console
 $ PROVIDER_TCK_REPORT_DIR=./reports pytest -k "unknown_flag_key"
-provider-tck [in-memory]: 28 of 29 canonical scenarios did not run, so this run cannot support a
+provider-tck [in-memory]: 39 of 40 canonical scenarios did not run, so this run cannot support a
 conformance claim and no report is written for it. …
   - features/errors.feature: A float flag is not silently narrowed to an integer
-  - features/errors.feature: Requesting the wrong type returns the code default [key=float-flag requested=Boolean default=false]
-  … and 18 more
+  - features/errors.feature: An integer requested as a float is widened without loss
+  … and 37 more
 ```
 
 A scenario the capability gate skipped **has** run: it was asked, and the report accounts for it
@@ -495,7 +495,7 @@ the field.
 | `test_canonical_set` | the canonical-set guard | a run that executed less than the canonical set fails and publishes nothing |
 
 ```
-155 passed, 21 skipped, 2 xfailed
+177 passed, 21 skipped, 2 xfailed
 ```
 
 No Docker and no network. The conformance suites take under a second; `test_report`,

tools/openfeature-provider-tck/tests/test_canonical_set.py

@@ -21,6 +21,7 @@
 import dataclasses
 import json
 import os
+import re
 import subprocess
 import sys
 import typing
@@ -113,12 +114,26 @@ def pytest_collection_modifyitems(items):
     Then the resolved details value should be "hi"
 """
 
+_EXTENSION_SCENARIOS = 3
+"""How many scenarios ``_VENDOR_FEATURE`` contributes."""
+
+_OUTCOMES = re.compile(r"(\d+) (passed|failed|skipped|xfailed|xpassed)\b")
+"""The per-outcome counts of a ``-q`` summary line, which together are what ran."""
+
 # The canonical scenario the filtered runs below leave out. Named rather than
 # counted, so a change to the canonical assets cannot leave these passing while
 # they select nothing.
 EXCLUDED_SELECTOR = "unknown_flag_key"
 EXCLUDED_SCENARIO = "An unknown flag key returns the code default"
 
+# How many canonical scenarios there are, read from the packaged assets through
+# the same parser the guard itself uses. Appendix F makes that a rule: an
+# expectation about the canonical set that restates a number goes stale the next
+# time the specification adds a scenario, and a conformance suite whose own tests
+# have to be edited to follow the assets is a suite that will be edited to agree
+# with them.
+CANONICAL_COUNT = len(canonical_scenarios())
+
 
 @dataclasses.dataclass(frozen=True)
 class Run:
@@ -212,13 +227,16 @@ def test_the_whole_canonical_set_still_writes_a_report(complete: Run) -> None:
 def test_a_filtered_run_fails_and_writes_no_report(tmp_path: Path) -> None:
     """The Go hazard, at the point it would have produced the document.
 
-    A well-formed report covering one scenario of twenty-nine is worse than no
-    report, because nothing in it says which twenty-eight were never asked.
+    A well-formed report covering one scenario of the canonical set is worse than
+    no report, because nothing in it says which of the rest were never asked.
     """
     run = _run(tmp_path, "-k", EXCLUDED_SELECTOR)
 
     assert run.result.returncode != 0, run.stdout
-    assert "of 29 canonical scenarios did not run" in run.stdout
+    assert (
+        f"{CANONICAL_COUNT - 1} of {CANONICAL_COUNT} canonical scenarios did not run"
+        in run.stdout
+    )
     assert not run.envelopes, "a partial run must publish nothing"
     assert not list(run.reports.glob("*.ndjson"))
     # The message names scenarios rather than only counting them, and says how
@@ -248,26 +266,36 @@ def test_acknowledging_a_partial_run_does_not_make_it_publishable(
 def test_an_extension_cannot_close_a_gap(tmp_path: Path) -> None:
     """Three extension scenarios do not make up for one canonical one.
 
-    The adoption below runs thirty-one scenarios where the canonical set has
-    twenty-nine, and is still one short: an adopter's scenarios are theirs, and
-    counting them towards the specification's set would let any gap be filled by
-    adding a feature file.
+    The adoption below runs more scenarios than the canonical set contains and is
+    still one short: an adopter's scenarios are theirs, and counting them towards
+    the specification's set would let any gap be filled by adding a feature file.
     """
     run = _run(tmp_path, "-k", f"not {EXCLUDED_SELECTOR}", extension=True)
 
     assert run.result.returncode != 0, run.stdout
-    assert "1 of 29 canonical scenarios did not run" in run.stdout
+    assert f"1 of {CANONICAL_COUNT} canonical scenarios did not run" in run.stdout
     assert f"features/errors.feature: {EXCLUDED_SCENARIO}" in run.stdout
     assert not run.envelopes
 
 
 # -- what the canonical set is -----------------------------------------------
 
 
-def test_the_canonical_set_is_read_from_the_packaged_assets() -> None:
-    """One entry per Scenario Outline row, which is what a runner generates."""
+def test_the_canonical_set_is_read_from_the_packaged_assets(complete: Run) -> None:
+    """One entry per Scenario Outline row, which is what a runner generates.
+
+    The number is not written down here. It is read off the unfiltered run --
+    pytest-bdd's own collection of the same packaged feature files, less the
+    three scenarios the adopter's extension contributes -- because an
+    expectation that restated a count would pin this file rather than the
+    assets, and go stale the next time the specification adds a scenario.
+    """
     scenarios = canonical_scenarios()
-    assert len(scenarios) == 29
+
+    summary = [line for line in complete.stdout.splitlines() if line.strip()][-1]
+    collected = sum(int(count) for count, _ in _OUTCOMES.findall(summary))
+
+    assert len(scenarios) == collected - _EXTENSION_SCENARIOS, summary
     assert all(is_canonical_uri(uri) for uri, _, _ in scenarios)
     # An outline contributes rows, not a single templated entry.
     assert any(row for _, _, row in scenarios)

tools/openfeature-provider-tck/tests/test_report.py

@@ -136,8 +136,15 @@ def tck_config():
 scenarios(features_path())
 '''
 
-CAPABILITIES = "{Capability.EVENTS, Capability.OBJECT, Capability.NUMERIC_COERCION}"
-"""What the main generated suite declares: enough to produce a skip and a pass."""
+CAPABILITIES = "{Capability.EVENTS, Capability.OBJECT, Capability.LARGE_INTEGERS}"
+"""What the main generated suite declares: enough to produce a skip and a pass.
+
+``LARGE_INTEGERS`` rather than ``NUMERIC_COERCION`` for the same reason the
+in-memory self-tests declare the one and not the other -- a Python ``int`` is
+unbounded, and the provider behind ``InProcessControl`` hands each variant back
+untouched rather than coercing it. Three declared capabilities are what these
+tests need; which three has to stay an honest claim about the provider.
+"""
 
 NOT_APPLICABLE = '{Capability.STALE: "this provider has no connection to lose"}'
 """One capability the provider cannot have rather than merely does not declare."""
@@ -588,7 +595,7 @@ def narrow_run(tmp_path_factory: pytest.TempPathFactory) -> Run:
         tmp_path_factory,
         file_name="narrow.json",
         name="narrow",
-        capabilities="{Capability.NUMERIC_COERCION}",
+        capabilities="{Capability.LARGE_INTEGERS}",
         not_applicable="{}",
         deviations=False,
     )
@@ -873,7 +880,7 @@ def test_the_declaration_is_an_input_not_a_summary(run: Run) -> None:
     declaration = run.envelope["declaration"]
     assert declaration["declared"] == [
         Capability.EVENTS.tag,
-        Capability.NUMERIC_COERCION.tag,
+        Capability.LARGE_INTEGERS.tag,
         Capability.OBJECT.tag,
     ]
     assert declaration["notApplicable"] == {