Support test automation exclusions in TestNG

[calvinbayer]

What Does This Do

Support test automation exclusions in TestNG. Wire test declaration metadata into Datadog policy selection and reporting, preserving assigned Test Management properties.

Motivation

Allow individual tests to prevent Datadog quarantine/disabling and exclude Datadog ATR, EFD and Attempt to Fix. Developer selection and framework retries retain their existing semantics.

Additional Notes

Stacked on codex/junit5-automation-exclusions; review this PR against its base for the framework-specific change. Shared public API and policy are introduced by the first JUnit4 PR. No refactor commit was needed.

Validation: Module spotlessApply and instrumented TestNG6Test/TestNG7Test exclusion regressions passed. Combined annotations keep failure blocking and execute once; native retry analyzers execute twice without Datadog retry tags. TestNG6 retains its legacy failed command status after a recovered native retry.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	58.14	56.55	$\color{red}{\blacktriangle}$ +1.59	56.55	$\color{red}{\blacktriangle}$ +1.59	163/403
agentEvpProxy	56.66	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	38.62	37.15	$\color{red}{\blacktriangle}$ +1.47	37.15	$\color{red}{\blacktriangle}$ +1.47	82/215
agentless	37.77	37.15	$\color{red}{\blacktriangle}$ +0.62	36.42	$\color{red}{\blacktriangle}$ +1.35	82/188
agentlessCodeCoverage	46.61	46.30	$\color{red}{\blacktriangle}$ +0.31	45.38	$\color{red}{\blacktriangle}$ +1.23	82/187
agentlessLineCoverage	57.40	56.55	$\color{red}{\blacktriangle}$ +0.85	56.55	$\color{red}{\blacktriangle}$ +0.85	81/186

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	90.35	89.58	$\color{red}{\blacktriangle}$ +0.77	89.58	$\color{red}{\blacktriangle}$ +0.77	82/216
agentless	83.12	82.69	$\color{red}{\blacktriangle}$ +0.43	82.69	$\color{red}{\blacktriangle}$ +0.43	82/187
agentlessCodeCoverage	98.07	97.04	$\color{red}{\blacktriangle}$ +1.03	97.04	$\color{red}{\blacktriangle}$ +1.03	82/187
agentlessLineCoverage	115.31	113.88	$\color{red}{\blacktriangle}$ +1.43	113.88	$\color{red}{\blacktriangle}$ +1.43	81/186

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	6.77	10.54	$\color{green}{\blacktriangledown}$ -3.77	10.13	$\color{green}{\blacktriangledown}$ -3.36	83/214
agentless	9.11	9.92	$\color{green}{\blacktriangledown}$ -0.81	9.92	$\color{green}{\blacktriangledown}$ -0.81	82/184
agentlessCodeCoverage	14.67	17.38	$\color{green}{\blacktriangledown}$ -2.71	17.73	$\color{green}{\blacktriangledown}$ -3.06	69/164

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	23.12	22.54	$\color{red}{\blacktriangle}$ +0.58	22.54	$\color{red}{\blacktriangle}$ +0.58	100/234
agentless	20.65	19.59	$\color{red}{\blacktriangle}$ +1.06	19.59	$\color{red}{\blacktriangle}$ +1.06	96/198
agentlessCodeCoverage	21.54	20.39	$\color{red}{\blacktriangle}$ +1.15	20.39	$\color{red}{\blacktriangle}$ +1.15	95/197
agentlessLineCoverage	28.41	27.52	$\color{red}{\blacktriangle}$ +0.89	27.52	$\color{red}{\blacktriangle}$ +0.89	59/133

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	9.94	9.73	$\color{red}{\blacktriangle}$ +0.21	9.92	$\color{red}{\blacktriangle}$ +0.02	80/185

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	14.58	13.40	$\color{red}{\blacktriangle}$ +1.18	13.40	$\color{red}{\blacktriangle}$ +1.18	82/217
agentless	12.97	12.37	$\color{red}{\blacktriangle}$ +0.60	12.37	$\color{red}{\blacktriangle}$ +0.60	82/187
agentlessCodeCoverage	16.69	15.72	$\color{red}{\blacktriangle}$ +0.97	15.41	$\color{red}{\blacktriangle}$ +1.28	82/187
agentlessLineCoverage	18.52	17.38	$\color{red}{\blacktriangle}$ +1.14	17.38	$\color{red}{\blacktriangle}$ +1.14	79/184

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	98.29	97.04	$\color{red}{\blacktriangle}$ +1.25	95.12	$\color{red}{\blacktriangle}$ +3.17	97/231
agentless	93.67	91.39	$\color{red}{\blacktriangle}$ +2.28	91.39	$\color{red}{\blacktriangle}$ +2.28	97/203
agentlessCodeCoverage	102.85	101.00	$\color{red}{\blacktriangle}$ +1.85	99.00	$\color{red}{\blacktriangle}$ +3.85	97/203
agentlessLineCoverage	102.92	103.04	$\color{green}{\blacktriangledown}$ -0.12	101.00	$\color{red}{\blacktriangle}$ +1.92	107/219

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	19.38	19.20	$\color{red}{\blacktriangle}$ +0.18	19.59	$\color{green}{\blacktriangledown}$ -0.21	84/217
agentless	18.60	18.45	$\color{red}{\blacktriangle}$ +0.15	19.20	$\color{green}{\blacktriangledown}$ -0.60	84/193
agentlessCodeCoverage	22.27	21.65	$\color{red}{\blacktriangle}$ +0.62	22.54	$\color{green}{\blacktriangledown}$ -0.27	86/197
agentlessLineCoverage	38.55	37.91	$\color{red}{\blacktriangle}$ +0.64	38.67	$\color{green}{\blacktriangledown}$ -0.12	78/186

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[datadog-datadog-us1-prod]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 59.90% (-0.00%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d308a2b | Docs | Give us feedback!}

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	16.46	16.36	$\color{red}{\blacktriangle}$ +0.10	16.36	$\color{red}{\blacktriangle}$ +0.10	82/214
agentless	10.05	9.92	$\color{red}{\blacktriangle}$ +0.13	9.92	$\color{red}{\blacktriangle}$ +0.13	82/186
agentlessCodeCoverage	14.10	13.94	$\color{red}{\blacktriangle}$ +0.16	13.67	$\color{red}{\blacktriangle}$ +0.43	82/188
agentlessLineCoverage	23.46	22.99	$\color{red}{\blacktriangle}$ +0.47	22.54	$\color{red}{\blacktriangle}$ +0.92	79/183

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	14.70	14.81	$\color{green}{\blacktriangledown}$ -0.11	14.23	$\color{red}{\blacktriangle}$ +0.47	82/214
agentless	14.20	12.62	$\color{red}{\blacktriangle}$ +1.58	12.87	$\color{red}{\blacktriangle}$ +1.33	82/188
agentlessCodeCoverage	40.78	79.45	$\color{green}{\blacktriangledown}$ -38.67	79.45	$\color{green}{\blacktriangledown}$ -38.67	82/188
agentlessLineCoverage	88.89	120.92	$\color{green}{\blacktriangledown}$ -32.03	123.36	$\color{green}{\blacktriangledown}$ -34.47	79/186

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-19T22:21:34.901907Z	a9fcfd4	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.82 s	14.66 s	[+0.4%; +1.9%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.52 s	13.61 s	[-1.5%; +0.1%] (no difference)
startup:petclinic:appsec:Agent	17.13 s	16.93 s	[+0.2%; +2.1%] (maybe worse)
startup:petclinic:iast:Agent	16.29 s	16.91 s	[-7.8%; +0.4%] (no difference)
startup:petclinic:profiling:Agent	16.64 s	16.72 s	[-1.6%; +0.6%] (no difference)
startup:petclinic:sca:Agent	17.07 s	16.88 s	[+0.4%; +2.0%] (maybe worse)
startup:petclinic:tracing:Agent	16.13 s	16.02 s	[-0.3%; +1.6%] (no difference)

Commit: d308a2b5 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a9fcfd4104

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[datadog-datadog-us1-prod]

The two most critical issues are: (1) skip exclusion decisions read only raw Java annotation groups and miss groups injected by IAnnotationTransformer, causing tests marked with datadog_prevent_failure_suppression to be incorrectly skipped; and (2) the native retry-analyzer preservation logic fails to recover class-level analyzer declarations and bypasses TestNG's object factory when constructing analyzer instances, silently dropping native retries for excluded tests.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit a9fcfd4}

[datadog-datadog-us1-prod]

Two distinct correctness issues are present: one where explicit user overrides to disable retries are silently ignored in favor of Datadog's RetryAnalyzer, and one where a missing interface in older TestNG runtimes throws an unguarded ClassNotFoundException before the assumption-skip can fire.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit ab0824f}

[calvinbayer]

@autotest Review

[datadog-datadog-us1-prod]

More details

Static review found no reportable defect in effective-group policy selection, native retry preservation, or Test Management reporting. Runtime behavior remains unverified because Gradle tests were not run.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit d308a2b}