Support test automation exclusion annotations in JUnit4

[calvinbayer]

What Does This Do

Add public @NoAutoRetry and @PreventFailureSuppression annotations and JUnit4 support. Protected tests run despite Datadog disabling, preserve blocking failures under quarantine, and can exclude ATR, EFD, and Attempt to Fix while retaining backend assignment tags.

Motivation

Let developers opt individual critical tests out of Datadog automation. Framework skips, TIA, parameterized source declarations, and explicit framework retries keep their normal semantics. Tag-based frameworks can use public TestAutomationExclusions constants.

Additional Notes

First PR in a per-framework Java stack; shared policy/API/reporting live here. No refactor commit was needed.

Validation: ./gradlew spotlessApply; focused existing ExecutionStrategyTest; new AutomationExclusionsTest; JUnit4 compile and instrumented JUnit4Test.test automation exclusions* all passed. The instrumented regression proves one execution, blocking failure, both annotation tags, retained disabled/quarantined/attempt-to-fix tags, and no attempt-to-fix completion tag.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	56.36	56.55	$\color{green}{\blacktriangledown}$ -0.19	56.55	$\color{green}{\blacktriangledown}$ -0.19	150/390
agentEvpProxy	55.89	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	38.81	37.15	$\color{red}{\blacktriangle}$ +1.66	37.15	$\color{red}{\blacktriangle}$ +1.66	74/207
agentless	37.46	37.15	$\color{red}{\blacktriangle}$ +0.31	36.42	$\color{red}{\blacktriangle}$ +1.04	73/179
agentlessCodeCoverage	46.94	46.30	$\color{red}{\blacktriangle}$ +0.64	45.38	$\color{red}{\blacktriangle}$ +1.56	71/176
agentlessLineCoverage	57.81	56.55	$\color{red}{\blacktriangle}$ +1.26	56.55	$\color{red}{\blacktriangle}$ +1.26	70/175

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	94.22	89.58	$\color{red}{\blacktriangle}$ +4.64	89.58	$\color{red}{\blacktriangle}$ +4.64	74/208
agentless	83.63	82.69	$\color{red}{\blacktriangle}$ +0.94	82.69	$\color{red}{\blacktriangle}$ +0.94	74/179
agentlessCodeCoverage	99.99	97.04	$\color{red}{\blacktriangle}$ +2.95	97.04	$\color{red}{\blacktriangle}$ +2.95	72/177
agentlessLineCoverage	114.09	113.88	$\color{red}{\blacktriangle}$ +0.21	113.88	$\color{red}{\blacktriangle}$ +0.21	70/175

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	21.41	22.54	$\color{green}{\blacktriangledown}$ -1.13	22.54	$\color{green}{\blacktriangledown}$ -1.13	92/226
agentless	19.25	19.20	$\color{red}{\blacktriangle}$ +0.05	19.59	$\color{green}{\blacktriangledown}$ -0.34	87/189
agentlessCodeCoverage	19.77	20.39	$\color{green}{\blacktriangledown}$ -0.62	20.39	$\color{green}{\blacktriangledown}$ -0.62	87/189
agentlessLineCoverage	26.39	27.52	$\color{green}{\blacktriangledown}$ -1.13	27.52	$\color{green}{\blacktriangledown}$ -1.13	56/130

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	8.17	10.54	$\color{green}{\blacktriangledown}$ -2.37	10.13	$\color{green}{\blacktriangledown}$ -1.96	74/205
agentless	7.41	9.92	$\color{green}{\blacktriangledown}$ -2.51	9.92	$\color{green}{\blacktriangledown}$ -2.51	73/175
agentlessCodeCoverage	14.83	17.38	$\color{green}{\blacktriangledown}$ -2.55	17.73	$\color{green}{\blacktriangledown}$ -2.90	63/158

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	8.04	9.92	$\color{green}{\blacktriangledown}$ -1.88	9.92	$\color{green}{\blacktriangledown}$ -1.88	74/179

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	13.87	13.40	$\color{red}{\blacktriangle}$ +0.47	13.40	$\color{red}{\blacktriangle}$ +0.47	75/210
agentless	12.53	12.37	$\color{red}{\blacktriangle}$ +0.16	12.37	$\color{red}{\blacktriangle}$ +0.16	75/180
agentlessCodeCoverage	15.56	15.72	$\color{green}{\blacktriangledown}$ -0.16	15.41	$\color{red}{\blacktriangle}$ +0.15	74/179
agentlessLineCoverage	17.12	17.38	$\color{green}{\blacktriangledown}$ -0.26	17.38	$\color{green}{\blacktriangledown}$ -0.26	69/174

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	97.09	95.12	$\color{red}{\blacktriangle}$ +1.97	95.12	$\color{red}{\blacktriangle}$ +1.97	90/224
agentless	93.04	91.39	$\color{red}{\blacktriangle}$ +1.65	89.58	$\color{red}{\blacktriangle}$ +3.46	90/196
agentlessCodeCoverage	102.44	101.00	$\color{red}{\blacktriangle}$ +1.44	99.00	$\color{red}{\blacktriangle}$ +3.44	90/196
agentlessLineCoverage	102.63	103.04	$\color{green}{\blacktriangledown}$ -0.41	101.00	$\color{red}{\blacktriangle}$ +1.63	99/211

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	19.41	19.20	$\color{red}{\blacktriangle}$ +0.21	19.59	$\color{green}{\blacktriangledown}$ -0.18	77/210
agentless	18.96	18.82	$\color{red}{\blacktriangle}$ +0.14	19.20	$\color{green}{\blacktriangledown}$ -0.24	77/186
agentlessCodeCoverage	21.67	21.65	$\color{red}{\blacktriangle}$ +0.02	22.54	$\color{green}{\blacktriangledown}$ -0.87	79/190
agentlessLineCoverage	38.12	37.91	$\color{red}{\blacktriangle}$ +0.21	38.67	$\color{green}{\blacktriangledown}$ -0.55	74/182

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.94 s	13.85 s	[-0.1%; +1.4%] (no difference)
startup:insecure-bank:tracing:Agent	12.79 s	12.91 s	[-1.6%; -0.2%] (maybe better)
startup:petclinic:appsec:Agent	17.10 s	16.87 s	[+0.5%; +2.1%] (maybe worse)
startup:petclinic:iast:Agent	16.95 s	16.98 s	[-1.0%; +0.7%] (no difference)
startup:petclinic:profiling:Agent	16.45 s	16.97 s	[-4.1%; -2.1%] (significantly better)
startup:petclinic:sca:Agent	16.98 s	16.79 s	[+0.3%; +2.0%] (maybe worse)
startup:petclinic:tracing:Agent	16.05 s	16.19 s	[-2.0%; +0.2%] (no difference)

Commit: 8028762a · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	16.05	16.36	$\color{green}{\blacktriangledown}$ -0.31	16.36	$\color{green}{\blacktriangledown}$ -0.31	75/207
agentless	10.45	9.92	$\color{red}{\blacktriangle}$ +0.53	9.92	$\color{red}{\blacktriangle}$ +0.53	75/179
agentlessCodeCoverage	13.72	13.94	$\color{green}{\blacktriangledown}$ -0.22	13.67	$\color{red}{\blacktriangle}$ +0.05	75/181
agentlessLineCoverage	23.76	22.99	$\color{red}{\blacktriangle}$ +0.77	22.54	$\color{red}{\blacktriangle}$ +1.22	69/173

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	31.34	16.36	$\color{red}{\blacktriangle}$ +14.98	14.23	$\color{red}{\blacktriangle}$ +17.11	75/207
agentless	30.91	12.62	$\color{red}{\blacktriangle}$ +18.29	12.87	$\color{red}{\blacktriangle}$ +18.04	75/181
agentlessCodeCoverage	99.31	84.36	$\color{red}{\blacktriangle}$ +14.95	81.05	$\color{red}{\blacktriangle}$ +18.26	75/181
agentlessLineCoverage	100.83	120.92	$\color{green}{\blacktriangledown}$ -20.09	123.36	$\color{green}{\blacktriangledown}$ -22.53	72/179

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-19T22:13:14.139699Z	193ac29	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[datadog-datadog-prod-us1-2]

Protected JUnit4 tests forced to run by TIA’s unskippable category can lose their forced-run tag and metric because reporting ignores the new exclusions.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 193ac29}

[calvinbayer]

@autotest Review

[datadog-datadog-prod-us1-2]

More details

Source and test inspection support that protected JUnit4 tests retain blocking failures and backend assignment tags while preserving framework-owned skips and retries.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 8028762}