[samsullivan1]

Purpose

Wire the three tensorflow-training framework test suites — tensorflow-training/unit, tensorflow-training/single_gpu, tensorflow-training/sagemaker — into the content-addressed test-skip cache: CI skips a suite when a prior PASS exists for the same image content + test code, and records a PASS once the suite genuinely runs and passes. Fail-open throughout — any cache miss, degraded lookup, or infra failure just runs the suite.

• tensorflow-training.tests-{unit,single-gpu,sagemaker}.yml — each reusable gains optional image-content-hash / suite-code-hash inputs and a record-test-pass job, gated to match each suite's topology:
  • sagemaker runs a preflight job and gates sagemaker-test at the job level on the image probe, so an absent image leaves it skipped (never success) → the record keys on sagemaker-test.result == 'success'.
  • unit and single_gpu run as a single job whose steps self-skip on the image-exists check, so the job can report success even when the image is absent. To avoid recording a false PASS, that job exposes an image-exists output and the record additionally requires needs.<job>.outputs.image-exists == 'true'.
• tensorflow-training.pipeline.yml — add all three suites to the check job, gate unit-test / single-gpu-test / sagemaker-tests on their skips[...] result (fail-open via skips || '{}', gated only on build failure — never on the check job), thread the hashes, and preserve the existing gates (single-gpu-test keeps device-type == 'gpu', sagemaker-tests keeps customer-type == 'sagemaker').
• .github/config/test-suites.yml — author tensorflow-training/unit → test/tensorflow/unit/**, tensorflow-training/single_gpu → test/tensorflow/single_gpu/**, tensorflow-training/sagemaker → test/tensorflow/integration/training/**. test/tensorflow/ is shared with tensorflow-inference, so the globs are scoped to the training-specific subpaths (disjoint from test/tensorflow/integration/inference/**). The sagemaker glob captures the suite's test modules plus its conftest.py, resources/ training scripts, and in-dir requirements.txt. Generic shared harness (root test/conftest.py, test/test_utils/**, root test/requirements.txt) is excluded as an accepted cache-hit tradeoff.

Test Plan

• python -m pytest test/db/test_config_matches_workflows.py test/db/test_hash_suite_code.py — reconciliation (invoked suites exist in config; skip accessors are configured and checked) + suite-code hashing resolves to real files.
• Verified the edited workflow files parse as YAML.
• Independent fresh-eyes review of the diff, with specific scrutiny of the per-suite record-gate topology (the self-skipping unit/single_gpu jobs vs. the job-level-gated sagemaker) and code_paths disjointness from tensorflow-inference.

Test Result

• Reconciliation + suite-hash tests: 13 passed.
• All edited *.yml parse as YAML; every tensorflow-training/* accessor is present in the check-job suites list and the gated jobs' if:.
• Fresh-eyes review: 0 findings (unit/single-gpu image-exists gate confirmed correct; sagemaker job-level gate confirmed safe; paths confirmed disjoint from inference).

Live CI validation — record → hit (clean HIT)

Validated via test PR #6644 (no-op Dockerfile edit → CPU build; single-gpu/sagemaker disabled). Run 32877989279:

• Run 1 (record): sanity + telemetry + tensorflow-training/unit recorded PASS rows → sanity, telemetry, unit.
• Run 2 (cache HIT): check hit → all three skipped with no re-record: sanity, telemetry, unit.

---

Toggle if you are merging into master Branch

By default, docker image builds and tests are disabled. Two ways to run builds and tests:

1. Using dlc_developer_config.toml
2. Using this PR description (currently only supported for PyTorch, TensorFlow, vllm, and base images)

How to use the helper utility for updating dlc_developer_config.toml

Assuming your remote is called origin (you can find out more with git remote -v)...

• Run default builds and tests for a particular buildspec - also commits and pushes changes to remote; Example:

python src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -cp origin

• Enable specific tests for a buildspec or set of buildspecs - also commits and pushes changes to remote; Example:

python src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -t sanity_tests -cp origin

• Restore TOML file when ready to merge

python src/prepare_dlc_dev_environment.py -rcp origin

NOTE: If you are creating a PR for a new framework version, please ensure success of the local, standard, rc, and efa sagemaker tests by updating the dlc_developer_config.toml file:

• sagemaker_remote_tests = true
• sagemaker_efa_tests = true
• sagemaker_rc_tests = true
• sagemaker_local_tests = true

How to use PR description

Use the code block below to uncomment commands and run the PR CodeBuild jobs. There are two commands available:

• # /buildspec <buildspec_path>
  • e.g.: # /buildspec pytorch/training/buildspec.yml
  • If this line is commented out, dlc_developer_config.toml will be used.
• # /tests <test_list>
  • e.g.: # /tests sanity security ec2
  • If this line is commented out, it will run the default set of tests (same as the defaults in dlc_developer_config.toml): sanity, security, ec2, ecs, eks, sagemaker, sagemaker-local.

# /buildspec <buildspec_path>
# /tests <test_list>

Toggle if you are merging into main Branch

PR Checklist

• I ran pre-commit run --all-files locally before creating this PR. (Read DEVELOPMENT.md for details).