Skip to content

Add feature to extract lobster test-trace-information from test-name - #505

Merged
hoe-jo merged 3 commits into
eclipse-score:mainfrom
Ehrja:feat_add_lobster_trace_extraction_from_regular_pattern_in_test_name
Oct 26, 2026
Merged

hoe-jo merged 3 commits into
eclipse-score:mainfrom
Ehrja:feat_add_lobster_trace_extraction_from_regular_pattern_in_test_name

Conversation

@Ehrja

@Ehrja Ehrja commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

lobster-gtest learned to derive the Given/When/Then specification from
conventionally named test cases of the form

[<Prefix>_]Given<X>[_When<Y>]_(Expect|Then)<Z>

with: bmw-software-engineering/lobster#714

Enable that option for the gtest traces collected by the component rule,
so tests no longer have to repeat the information as record properties.
Record properties still take precedence over the test name.

The option does not exist before lobster >1.0.6, and passing an unknown
keyword argument to a subrule is an analysis error. Since Starlark cannot
introspect a subrule signature, a module extension probes lobster's own
rule definition and exposes the result as a load-time constant. Versions
without the option keep the previous behaviour, and bumping to a version
that provides it enables the derivation without a further change here.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

To reduce the WFT's per line I added some elaborations what is going on.
Feel free to ignore them, just to have some kind of documentation also for learning
purposes :)

@Ehrja
Ehrja marked this pull request as ready for review October 8, 2026 12:02
@hoe-jo

hoe-jo commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

PR should be splitted into a part in the lobster repo: bmw-software-engineering/lobster#714
and into the part which remains in this repo

@Ehrja
Ehrja force-pushed the feat_add_lobster_trace_extraction_from_regular_pattern_in_test_name branch from eafb640 to 68ddfc8 Compare October 14, 2026 16:38
@Ehrja
Ehrja marked this pull request as draft October 14, 2026 16:38
@Ehrja

Ehrja commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor Author

PR should be splitted into a part in the lobster repo: bmw-software-engineering/lobster#714 and into the part which remains in this repo

I've reworked it such that it uses your function-extension from linked PR in lobster repo. However I've changed it in a way that it is not requireing the lobster update before you can rereview/merge this PR because it contains a shim which probes lobster whether the required extension is available and if it is the case use it, otherwise take the previous logic. By doing so client codes are not obliged to update lobster at the same time like this repo.

In the mid-term given that the lobster-change is accepted this shim can be removed again. I modelled it intenionally as "fade in/out" behavior 😅

This would be the behavior/failure modes:

Situation Probe Effect
lobster has the attribute True --derive-gwt-from-test-name passed
lobster ≤ 1.0.6 False argument omitted, previous behaviour
lobster restructured, file moved error repo rule fails: label no longer resolves
attribute renamed upstream False silently degrades to no-op

The last two are the important ones.
3. row: If lobster ever moves bazel/private/lobster_gtest.bzl, repository_ctx.path fails outright rather than returning False.
4. row: The failure is safe but silent if the attribute is renamed.

"""

load("@lobster//:lobster.bzl", "subrule_lobster_gtest", "subrule_lobster_html_report", "subrule_lobster_report")
load("@lobster_capabilities//:capabilities.bzl", "SUPPORTS_DERIVE_GWT_FROM_TEST_NAME")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bazel evaluates in strictly ordered phases, and the problem is that the decision must be made earlier than the information is available by normal means:

  1. Module resolution MODULE.bazel -> version selection -> module extensions run
  2. Loading .bzl files evaluated, load() resolved, BUILD files evaluated
  3. Analysis rule/subrule implementations run, actions registered
  4. Execution actions run

subrule_lobster_gtest(files, derive_gwt_from_test_name = True) executes in phase 3, and an unknown keyword argument there is a hard analysis error, since Starlark has no try/except. Furthermore, there is no introspection API: like python dir() on a subrule object gives you nothing about its implementation's signature, and Bazel exposes no way to list a rule's or subrule's attributes from Starlark.

So the capability has to be known as a constant by phase 2, which means it must be produced in phase 1. That is exactly what a repository rule driven by a module extension does.


# Definition of the gtest rule inside the lobster module. A restructuring of
# lobster makes the probe report "unsupported", which is the safe answer.
_LOBSTER_GTEST_BZL = Label("@lobster//bazel/private:lobster_gtest.bzl")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Label() is resolved at load time in the context of the file that defines it. Because this .bzl lives in score-tooling, the apparent name @lobster is resolved through score-tooling's repo mapping — which works precisely because score-tooling has bazel_dep(name = "lobster", ...). The canonical name it maps to (lobster+ (at least in bazel 8.x)) never has to be spelled out.
This also relies on bazel/private/BUILD existing in lobster, so that bazel/private is a real package and the label is well-formed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One could argue that it is bad that we "depend" on a private part of a third-party, on the otherhand it enables some kind of gracefull roll-out. So I could argue that it is okay for a certain amount of time but has to be removed in the mid-term (after lobster accepted the PR and merged it)

"""

def _lobster_capabilities_repo_impl(repository_ctx):
definition = repository_ctx.read(repository_ctx.path(_LOBSTER_GTEST_BZL))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things happen here:

  • repository_ctx.path(Label) forces lobster's repository to be fetched (if it isn't already) and returns the on-disk path.
    Repo rules bypass target visibility - bazel/private being "private" by convention is irrelevant, this is a filesystem read.
  • Crucially, it registers a dependency on that file. If lobster changes like a version bump or a different override, Bazel re-runs this repo rule rather than serving a stale answer.

Then DERIVE_GWT_FROM_TEST_NAME in definition is a plain substring test over the source text.
Crude, but it tests the capability rather than a version number, so a backport or an unexpected release numbering still works

Comment on lines +48 to +54
repository_ctx.file("BUILD.bazel", "")
repository_ctx.file(
"capabilities.bzl",
_CAPABILITIES_TEMPLATE.format(
derive_gwt_from_test_name = repr(_DERIVE_GWT_FROM_TEST_NAME in definition),
),
)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The empty BUILD.bazel is mandatory: without it the repo root isn't a package and @lobster_capabilities//:capabilities.bzl would not be loadable.
repr(True) yields the string True, which is valid Starlark.

The generated file is literally:
SUPPORTS_DERIVE_GWT_FROM_TEST_NAME = True

Comment on lines +56 to +67
_lobster_capabilities_repo = repository_rule(
implementation = _lobster_capabilities_repo_impl,
doc = "Generates a .bzl file holding the probed lobster feature flags.",
)

def _lobster_capabilities_impl(_module_ctx):
_lobster_capabilities_repo(name = "lobster_capabilities")

lobster_capabilities = module_extension(
implementation = _lobster_capabilities_impl,
doc = "Provides the @lobster_capabilities repository.",
)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A repository_rule cannot be invoked from MODULE.bazel directly under bzlmod; it must be created by a module_extension.
That's all lobster_capabilities_impl does.
The module_ctx parameter is unused (leading underscore marks that for buildifier) because the extension takes no tags; it always creates exactly one repo.

Comment thread MODULE.bazel

# Probes which optional features the resolved lobster module provides, so that
# the rules can keep working with versions that predate them.
lobster_capabilities = use_extension(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ehrja
Ehrja force-pushed the feat_add_lobster_trace_extraction_from_regular_pattern_in_test_name branch 2 times, most recently from e50da2e to fc1e805 Compare October 21, 2026 06:45
lobster-gtest learned to derive the Given/When/Then specification from
conventionally named test cases of the form

    [<Prefix>_]Given<X>[_When<Y>]_(Expect|Then)<Z>

Enable that option for the gtest traces collected by the component rule,
so tests no longer have to repeat the information as record properties.
Record properties still take precedence over the test name.

The option does not exist before lobster 1.0.7, and passing an unknown
keyword argument to a subrule is an analysis error. Since Starlark cannot
introspect a subrule signature, a module extension probes lobster's own
rule definition and exposes the result as a load-time constant. Versions
without the option keep the previous behaviour, and bumping to a version
that provides it enables the derivation without a further change here.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Ehrja
Ehrja force-pushed the feat_add_lobster_trace_extraction_from_regular_pattern_in_test_name branch from fc1e805 to 949b4cb Compare October 21, 2026 10:39
- add tests
- update docu
- make when optional
@hoe-jo
hoe-jo marked this pull request as ready for review October 22, 2026 12:27
hoe-jo
hoe-jo previously approved these changes Oct 22, 2026
…ar_pattern_in_test_name

Signed-off-by: Jan Ehrensperger <jan.ehrensperger@bmw.de>
@hoe-jo
hoe-jo added this pull request to the merge queue Oct 26, 2026
Merged via the queue into eclipse-score:main with commit df3d21b Oct 26, 2026
14 checks passed
@Ehrja
Ehrja deleted the feat_add_lobster_trace_extraction_from_regular_pattern_in_test_name branch October 26, 2026 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants