Skip to content

Commit df3d21b

Browse files
authored
Add feature to extract lobster test-trace-information from test-name (#505)
lobster-gtest learned to derive the Given/When/Then specification from conventionally named test cases of the form [<Prefix>_]Given<X>[_When<Y>]_(Expect|Then)<Z> with: bmw-software-engineering/lobster#714 Enable that option for the gtest traces collected by the component rule, so tests no longer have to repeat the information as record properties. Record properties still take precedence over the test name. The option does not exist before lobster >1.0.6, and passing an unknown keyword argument to a subrule is an analysis error. Since Starlark cannot introspect a subrule signature, a module extension probes lobster's own rule definition and exposes the result as a load-time constant. Versions without the option keep the previous behaviour, and bumping to a version that provides it enables the derivation without a further change here. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> To reduce the WFT's per line I added some elaborations what is going on. Feel free to ignore them, just to have some kind of documentation also for learning purposes :)
2 parents 04d2734 + 74cae14 commit df3d21b

21 files changed

Lines changed: 382 additions & 42 deletions

‎.github/skills/score-testing/SKILL.md‎

Lines changed: 30 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -85,13 +85,14 @@ TEST(MyUnitTest, ConfigureAndGet) {
8585
| Property | Required | Description |
8686
|----------|----------|-------------|
8787
| `lobster-tracing` | yes | One or more requirement IDs (`Package.RecordId`), linking the test to `CompReq` records. Multiple IDs are separated by whitespace (the docs also describe comma-separated). |
88-
| `given` | no | Initial state / precondition |
88+
| `given` | with coverage lock | Initial state / precondition |
8989
| `when` | no | Action or event under test |
90-
| `then` | no | Expected outcome |
90+
| `then` | with coverage lock | Expected outcome |
9191

9292
- A test **without** `lobster-tracing` has no traceability and is excluded from coverage tracking.
9393
- The referenced IDs must resolve to `CompReq` records exposed through the component's
9494
`requirements` targets.
95+
- A test case in a coverage lock needs a Given and a Then; When is always optional.
9596

9697
```starlark
9798
cc_test(
@@ -108,6 +109,29 @@ unit(
108109
)
109110
```
110111

112+
### Given/When/Then from the test name
113+
114+
A test named `[<Prefix>_]Given<X>[_When<Y>]_(Expect|Then)<Z>` needs no `given`/`when`/`then`
115+
properties; its name provides them. `<Prefix>` is alphanumeric; `<X>`, `<Y>`, `<Z>` are CamelCase and
116+
become lowercase words separated by spaces:
117+
118+
```cpp
119+
// Given "empty stack", When "pop called", Then "error"
120+
TEST(Stack, GivenEmptyStack_WhenPopCalled_ExpectError) {
121+
::testing::Test::RecordProperty("lobster-tracing", "MinimalExample.FEAT_001");
122+
...
123+
}
124+
```
125+
126+
- `_When<Y>` is optional; `GivenEmptyStack_ExpectError` has no When.
127+
- A test that sets any of the three properties to a non-blank value ignores its name completely, so
128+
missing parts are not filled in from the name. If the name also matches, lobster prints a warning.
129+
Set all needed parts, or none.
130+
- Needs a `lobster` version that provides `--derive-gwt-from-test-name`. It is not available in
131+
lobster 1.0.6 and earlier, where only the properties are used.
132+
- After a `lobster` upgrade that provides the option, the lock entries of pattern-named tests without
133+
properties become stale; run `bazel run //<pkg>:<component>.update` and review the diff.
134+
111135
---
112136

113137
## Test-Case Coverage (`test_case_coverage.lock.yaml`)
@@ -140,7 +164,8 @@ requirements:
140164
141165
- `requirements[].id` — matches the `lobster-tracing` value; extracted from the requirements targets.
142166
- `test_cases[].uid` — `//bazel_package/SuiteName:TestName`, a package-scoped gtest tag.
143-
- `given` / `when` / `then` — the GWT fields from `RecordProperty`; **any** change makes the lock stale.
167+
- `given` / `when` / `then` — the GWT fields from `RecordProperty` or the test name; **any** change
168+
makes the lock stale. `given` and `then` must be non-blank; `when` may be empty and is stored as `''`.
144169

145170
### Two workflows keep the lock in sync
146171

@@ -159,8 +184,8 @@ controls how strict it is:
159184

160185
| `maturity` | Behavior |
161186
|------------|----------|
162-
| `"development"` | `--allow-check-failures`: lock-drift **and** missing-GWT-annotation errors are downgraded to warnings; the `.lobster` artifact is always produced. |
163-
| `"release"` | A drift or a missing GWT annotation fails the Bazel build action directly. |
187+
| `"development"` | `--allow-check-failures`: lock-drift **and** missing-Given/Then errors are downgraded to warnings; the `.lobster` artifact is always produced. |
188+
| `"release"` | A drift or a missing Given/Then fails the Bazel build action directly. |
164189

165190
Switch back to `"release"` before certification.
166191

‎MODULE.bazel‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,15 @@ bazel_dep(name = "flatbuffers", version = "25.9.23")
3838
bazel_dep(name = "download_utils", version = "1.2.2")
3939
bazel_dep(name = "trlc", version = "3.0.1")
4040
bazel_dep(name = "lobster", version = "1.0.6")
41+
42+
# Probes which optional features the resolved lobster module provides, so that
43+
# the rules can keep working with versions that predate them.
44+
lobster_capabilities = use_extension(
45+
"//bazel/rules/rules_score/private:lobster_capabilities.bzl",
46+
"lobster_capabilities",
47+
)
48+
use_repo(lobster_capabilities, "lobster_capabilities")
49+
4150
bazel_dep(name = "rules_distroless", version = "0.8.0")
4251
bazel_dep(name = "googletest", version = "1.17.0.bcr.2")
4352

‎bazel/rules/rules_score/docs/requirements/tool_requirements.trlc‎

Lines changed: 47 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -277,16 +277,58 @@ section "Tool Requirements" {
277277

278278
ToolQualification.ToolRequirement Parse_GWT_Annotations {
279279
description = '''
280-
The tool shall parse GTest XML result files, extracting
281-
RecordProperty values for the keys lobster-tracing, given, when,
282-
and then on a per-TEST, per-TEST_F, and per-TEST_P basis. A test case
283-
without a given, when, or then property shall be included with an
284-
empty string for each missing field.
280+
The tool shall read the requirement references and the given,
281+
when, and then parts of every test case from the gtest.lobster
282+
file that lobster-gtest produces from the GTest XML result
283+
files. A test case without a given, when, or then part shall be
284+
included with an empty string for each missing part.
285285
'''
286286
derived_from = [Detect_Specification_Drift, Detect_Coverage_Set_Changes]
287287
satisfied_by = Tools.TestCaseCoverage
288288
}
289289

290+
ToolQualification.ToolRequirement Enable_GWT_Derivation {
291+
description = '''
292+
The component rule shall pass the option
293+
--derive-gwt-from-test-name to lobster-gtest when the resolved
294+
lobster version provides that option, and shall omit it
295+
otherwise.
296+
'''
297+
derived_from = [Detect_Specification_Drift]
298+
satisfied_by = Tools.Bazel
299+
}
300+
301+
ToolQualification.ToolRequirement Derive_GWT_From_Test_Name {
302+
description = '''
303+
With the option --derive-gwt-from-test-name, lobster-gtest
304+
shall derive the given, when, and then parts of a test case
305+
from its test name, if the name matches
306+
[<Prefix>_]Given<X>[_When<Y>]_(Expect|Then)<Z>. <Prefix> is
307+
alphanumeric, and <X>, <Y>, and <Z> are in CamelCase. Each
308+
derived part is its CamelCase text converted to lowercase
309+
words separated by spaces. A test case whose name has no When
310+
part shall have an empty when field. A test case that sets a
311+
non-blank given, when, or then RecordProperty shall take all
312+
three parts from its RecordProperty values only, and its
313+
name shall be ignored.
314+
'''
315+
derived_from = [Detect_Specification_Drift]
316+
satisfied_by = Tools.Lobster
317+
}
318+
319+
ToolQualification.ToolRequirement Require_Given_And_Then {
320+
description = '''
321+
The tool shall report every linked test case whose given or
322+
then field is empty or blank as an error naming the requirement
323+
identifier, the test uid, and the missing fields. An empty
324+
when field shall not be reported. The error shall fail the
325+
test action, and shall prevent the .update command from
326+
writing the lock file, unless check failures are allowed.
327+
'''
328+
derived_from = [Declare_Complete_Coverage, Detect_Specification_Drift]
329+
satisfied_by = Tools.TestCaseCoverage
330+
}
331+
290332
ToolQualification.ToolRequirement Multi_Requirement_Linking {
291333
description = '''
292334
A test case that carries lobster-tracing values matching more than

‎bazel/rules/rules_score/docs/tool_reference/test_case_coverage.rst‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -57,10 +57,10 @@ Two phases
5757

5858
* ``maturity = "development"`` — ``--allow-check-failures`` is passed so
5959
the ``.lobster`` artifact is always produced even when coverage drifts.
60-
Both lock-drift failures **and** missing-GWT-annotation errors are
60+
Both lock-drift failures **and** missing Given/Then errors are
6161
downgraded to warnings;
6262
* ``maturity = "release"`` — no ``--allow-check-failures``; a drift or
63-
missing GWT annotation fails the Bazel build action directly. The
63+
a missing Given or Then fails the Bazel build action directly. The
6464
``.lobster`` artifact is physically written before the action exits, but
6565
Bazel marks the action as failed.
6666

@@ -133,5 +133,10 @@ reviewing ``git diff``.
133133
* - ``test_cases[].uid``
134134
- ``//bazel_package/SuiteName:TestName`` — package-scoped gtest tag
135135
* - ``test_cases[].given`` / ``when`` / ``then``
136-
- Individual GWT fields from ``RecordProperty`` annotations; any change
137-
makes the lock stale
136+
- Individual GWT fields from ``RecordProperty`` annotations or the test
137+
name; any change makes the lock stale. ``given`` and ``then`` must be
138+
non-blank. ``when`` may be empty and is then stored as ``''``
139+
140+
Upgrading ``lobster`` to a version that derives the GWT fields from test names
141+
changes the text of tests that follow the naming pattern and set no properties.
142+
Run ``.update`` and review the diff, see :doc:`../user_guide/validation`.

‎bazel/rules/rules_score/docs/user_guide/tutorial/validation.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,10 @@ They are picked up the same way as ``lobster-tracing`` — as plain
4545
traceability report. Test cases without them are still traced, but show up
4646
without a Given-When-Then specification.
4747

48+
Instead of the properties, a test can be named
49+
``[<Prefix>_]Given<X>[_When<Y>]_(Expect|Then)<Z>`` so that its name provides the
50+
specification. See :doc:`../validation` for the rules.
51+
4852
BUILD
4953
------
5054

‎bazel/rules/rules_score/docs/user_guide/validation.rst‎

Lines changed: 45 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,15 +54,51 @@ Call ``RecordProperty`` inside the test body next to the respective code blocks:
5454
- yes
5555
- Comma-separated requirement IDs; links the test to one or more ``CompReq`` records
5656
* - ``given``
57-
- no
57+
- with coverage lock
5858
- Initial state / precondition
5959
* - ``when``
6060
- no
6161
- Action or event under test
6262
* - ``then``
63-
- no
63+
- with coverage lock
6464
- Expected outcome
6565

66+
A test case that is part of a coverage lock (see `Stating Coverage for test cases`_)
67+
needs a ``given`` and a ``then``. The ``when`` is always optional.
68+
69+
Deriving Given/When/Then from the Test Name
70+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
71+
72+
A test named ``[<Prefix>_]Given<X>[_When<Y>]_(Expect|Then)<Z>`` does not need the
73+
``given``/``when``/``then`` properties. Its name provides them. ``<Prefix>`` is
74+
alphanumeric, and ``<X>``, ``<Y>`` and ``<Z>`` are in CamelCase. Each of them becomes
75+
lowercase words separated by spaces:
76+
77+
.. code-block:: cpp
78+
79+
TEST(Stack, GivenEmptyStack_WhenPopCalled_ExpectError) {
80+
::testing::Test::RecordProperty("lobster-tracing",
81+
"SampleComponent.REQ_COMP_001");
82+
...
83+
}
84+
85+
This test has the Given ``empty stack``, the When ``pop called`` and the Then
86+
``error``.
87+
88+
* The ``_When<Y>`` part is optional. ``GivenEmptyStack_ExpectError`` has no When.
89+
* For a value-parameterized test, the segments of the name are separated by ``/``
90+
and the last segment that matches is used.
91+
* If a test sets any of ``given``, ``when`` or ``then`` to a non-blank value, its
92+
name is ignored completely, so parts missing from the properties are not filled in
93+
from the name. If the name also matches the pattern, lobster prints a warning. Set
94+
all parts that the test needs, or none.
95+
* Deriving requires a ``lobster`` version that provides the option
96+
``--derive-gwt-from-test-name``. It is not available in lobster 1.0.6 and earlier,
97+
where only the properties are used.
98+
* After a ``lobster`` upgrade that provides the option, the derived text of tests that
99+
follow the pattern and set none of the properties changes. Their entries in the lock
100+
become stale, so run ``bazel run //<pkg>:<name>.update`` and review the diff.
101+
66102
A ``dependable_element`` report renders every annotated test on its **Unit
67103
Test** page at ``<dependable-element>_index/traceability_report/unit_test.html``.
68104
The entry shows its Given/When/Then text, links to the traced requirements, and
@@ -78,6 +114,13 @@ Coverage is declared through a committed ``test_case_coverage.lock.yaml`` file t
78114
per requirement, every test case (uid + given/when/then) that covers it.
79115
Committing the file is stating the coverage claim.
80116

117+
Every test case in the lock needs a Given and a Then, set through properties or
118+
derived from the test name. The When is optional and is stored as an empty string
119+
when absent. A test case without a Given or a Then is reported as an error. With
120+
``maturity = "release"`` on the ``dependable_element`` the error fails the build, with
121+
``"development"`` it is downgraded to a warning (see
122+
:doc:`../tool_reference/test_case_coverage`).
123+
81124
In Bazel the yaml file can be linked to the ``component`` macro via the ``test_case_coverage_lock`` attribute:
82125

83126
.. code-block:: starlark

‎bazel/rules/rules_score/private/component.bzl‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ with associated requirements and tests.
2020
"""
2121

2222
load("@lobster//:lobster.bzl", "subrule_lobster_gtest", "subrule_lobster_html_report", "subrule_lobster_report")
23+
load("@lobster_capabilities//:capabilities.bzl", "SUPPORTS_DERIVE_GWT_FROM_TEST_NAME")
2324
load("//bazel/rules/rules_score:providers.bzl", "AssumedSystemRequirementsInfo", "CertifiedScope", "ComponentInfo", "ComponentRequirementsInfo", "ComponentTestCaseCoverageInfo", "FeatureRequirementsInfo", "SphinxSourcesInfo", "UnitInfo")
2425
load("//bazel/rules/rules_score/private:lobster_config.bzl", "format_lobster_sources")
2526

@@ -246,7 +247,15 @@ def _component_impl(ctx):
246247
collected_architecture.append(component[ComponentInfo].architecture)
247248

248249
# Convert unit test XML files to a single gtest.lobster file
249-
gtest_lobster_file, provider = subrule_lobster_gtest(depset(transitive = collected_unit_test_files).to_list())
250+
# Given/When/Then is derived from conventionally named test cases, so that
251+
# tests do not have to repeat the information as record properties. Older
252+
# lobster versions do not offer the option and keep relying on the record
253+
# properties alone.
254+
derive_gwt = {"derive_gwt_from_test_name": True} if SUPPORTS_DERIVE_GWT_FROM_TEST_NAME else {}
255+
gtest_lobster_file, provider = subrule_lobster_gtest(
256+
depset(transitive = collected_unit_test_files).to_list(),
257+
**derive_gwt
258+
)
250259

251260
# -------------------------------------------------------------------------
252261
# Architecture Lobster: generate architecture.lobster for this component
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# *******************************************************************************
2+
# Copyright (c) 2026 Contributors to the Eclipse Foundation
3+
#
4+
# See the NOTICE file(s) distributed with this work for additional
5+
# information regarding copyright ownership.
6+
#
7+
# This program and the accompanying materials are made available under the
8+
# terms of the Apache License Version 2.0 which is available at
9+
# https://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# SPDX-License-Identifier: Apache-2.0
12+
# *******************************************************************************
13+
"""Detect optional features of the resolved ``lobster`` module.
14+
15+
``subrule_lobster_gtest`` gained the keyword argument
16+
``derive_gwt_from_test_name`` after the 1.0.6 release. Passing an unknown
17+
keyword argument to a subrule is an analysis error, and Starlark offers no way
18+
to introspect a subrule's signature, so the capability is probed by reading
19+
lobster's own rule definition.
20+
21+
The result is exposed as a load-time constant through a generated repository,
22+
which lets the call sites choose the matching invocation. Pinning a lobster
23+
version that predates the feature therefore degrades to the previous behaviour
24+
instead of breaking the build, and bumping to a version that provides it
25+
enables the feature without any further change.
26+
"""
27+
28+
_DERIVE_GWT_FROM_TEST_NAME = "derive_gwt_from_test_name"
29+
30+
# Definition of the gtest rule inside the lobster module. A restructuring of
31+
# lobster makes the probe report "unsupported", which is the safe answer.
32+
_LOBSTER_GTEST_BZL = Label("@lobster//bazel/private:lobster_gtest.bzl")
33+
34+
_CAPABILITIES_TEMPLATE = """\
35+
\"\"\"Generated by //bazel/rules/rules_score/private:lobster_capabilities.bzl.
36+
37+
Do not edit. Describes the features of the lobster module that the current
38+
dependency resolution selected.
39+
\"\"\"
40+
41+
# Whether subrule_lobster_gtest accepts the keyword argument
42+
# "derive_gwt_from_test_name".
43+
SUPPORTS_DERIVE_GWT_FROM_TEST_NAME = {derive_gwt_from_test_name}
44+
"""
45+
46+
def _lobster_capabilities_repo_impl(repository_ctx):
47+
definition = repository_ctx.read(repository_ctx.path(_LOBSTER_GTEST_BZL))
48+
repository_ctx.file("BUILD.bazel", "")
49+
repository_ctx.file(
50+
"capabilities.bzl",
51+
_CAPABILITIES_TEMPLATE.format(
52+
derive_gwt_from_test_name = repr(_DERIVE_GWT_FROM_TEST_NAME in definition),
53+
),
54+
)
55+
56+
_lobster_capabilities_repo = repository_rule(
57+
implementation = _lobster_capabilities_repo_impl,
58+
doc = "Generates a .bzl file holding the probed lobster feature flags.",
59+
)
60+
61+
def _lobster_capabilities_impl(_module_ctx):
62+
_lobster_capabilities_repo(name = "lobster_capabilities")
63+
64+
lobster_capabilities = module_extension(
65+
implementation = _lobster_capabilities_impl,
66+
doc = "Provides the @lobster_capabilities repository.",
67+
)

‎bazel/rules/rules_score/src/test_case_coverage/check_lock.py‎

Lines changed: 6 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -92,29 +92,21 @@ def compare_lock_files(
9292

9393

9494
def validate_specs(computed: LockFile) -> tuple[bool, list[str]]:
95-
"""Fail if any test case in *computed* is missing a GWT spec.
95+
"""Fail if any test case in *computed* lacks a ``given`` or ``then`` part.
9696
97-
A missing spec means the developer did not add ``given``/``when``/``then``
98-
``RecordProperty`` annotations to the test. Coverage without a spec
99-
cannot be meaningfully reviewed or approved.
97+
``when`` is optional, because a test name derived by lobster-gtest may omit it.
98+
Coverage without ``given`` and ``then`` cannot be meaningfully reviewed or approved.
10099
101100
Returns (ok, issue_lines).
102101
"""
103102
issues: list[str] = []
104103
for req in computed.requirements:
105104
for tc in req.test_cases:
106-
missing = [
107-
name
108-
for name, val in (
109-
("given", tc.given),
110-
("when", tc.when),
111-
("then", tc.then),
112-
)
113-
if not val.strip()
114-
]
105+
missing = [name for name, val in (("given", tc.given), ("then", tc.then)) if not val.strip()]
115106
if missing:
116107
issues.append(
117108
f" ! [{req.id}] test missing GWT fields ({', '.join(missing)}): {tc.uid}\n"
118-
f" Add RecordProperty('given'/'when'/'then') annotations."
109+
f" Set RecordProperty 'given' and 'then' ('when' is optional). Alternatively, name the test "
110+
f"Given<X>[_When<Y>]_(Expect|Then)<Z> if lobster provides --derive-gwt-from-test-name."
119111
)
120112
return len(issues) == 0, issues

0 commit comments

Comments
 (0)