[lucasjia-aws]

Summary

The scheduled CI Health V2 slow-tests-v2 job (run against master-v2) hit the 180-minute CodeBuild timeout in 9 of the last 16 runs, including https://github.com/aws/sagemaker-python-sdk/actions/runs/37570212277. When that happens the GitHub job only reports "The security token included in the request is expired", because the workflow's 3-hour AWS session expires at the same moment. This PR fixes the test-side causes of the timeouts and one recurring failure. No SDK source changes. Follow-up to #6386.

Root causes and fixes

JumpStart gated training tests block for hours: test_gated_model_training_v1/v2 train on ml.g5.12xlarge with estimator.fit() and max_run=259200. When capacity is short the job sits in Pending without failing, so @x_fail_if_ice never fires (e.g. 524 and 449 minutes of Pending on 2026-10-07, 1348 and 1521 minutes on 2026-10-02). These tests, and test_jumpstart_hub_gated_estimator_with_eula, now use the existing fit_estimator_with_capacity_xfail, which stops a job that has not been granted capacity after 30 minutes and raises a CapacityError that x_fail_if_ice turns into an xfail. The helper gains **fit_kwargs so the hub test can still pass accept_eula=True.

test_run_explainability_monitor reruns until the build times out: On 2026-09-29, 10-03 and 10-04 every explainability monitoring execution failed with "The primary container for production variant AllTraffic did not pass the ping health check" on the Clarify shadow endpoint. The shadow endpoint re-downloads the model from session-default-prefix/integ-test-data/xgboost/model/xgb_model.tar.gz, and that key is shared by tests uploading two different files: tests/data/xgboost_model (the model the xgboost:1 image serves) and tests/data/xgboost_abalone / the identical experiments transform model (a newer XGBoost pickle). When an abalone upload from a concurrently running suite lands between the clarify endpoint deploy and the shadow endpoint creation, the shadow endpoint cannot load the model. On 2026-10-03 the last model created from that key before the shadow endpoint was test_model_package's abalone model; on good days it was an xgboost_model test. With reruns=5 and each attempt waiting for the next hourly scheduled execution, one bad day costs about six hours. The clarify fixture now uploads under a per-run key, the 13 abalone uploads move to integ-test-data/xgboost_abalone/model, and the two test_run_*_monitor tests drop to reruns=1.

Feature processor tests fail when two start Spark at once: Local feature_processor runs start a JVM whose spark-submit --packages resolves hadoop-aws/hadoop-common into the shared ~/.ivy2 cache. When two xdist workers do this at the same moment on a cold cache the downloads collide ("Downloaded file size (0) doesn't match expected Content Length") and the JVM exits with "Java gateway process exited before sending its port number". This failed test_feature_processor_transform_with_customized_data_source on 2026-10-06 and test_feature_processor_transform_online_only_store_ingestion on 2026-10-07. A local_spark_lock fixture (a host-wide FileLock) now runs the three local Spark tests one at a time; each takes under about 70 seconds.

Not included

test_lda failed on 2026-10-07 because canaries-v2-release, which runs the tests of the latest released v2 tag, still contains the account-wide endpoint cleanup removed in #6386 and deleted the test's endpoint mid-deploy. That resolves with the next v2 release.

Testing

black (24.3.0) and flake8 (7.1.2) pass on all changed files, and pytest --collect-only collects every changed module with the new fixtures resolving. Offline checks covered fit_estimator_with_capacity_xfail forwarding accept_eula, its unchanged behavior without extra kwargs, and its stop-and-raise path on capacity timeout, plus the file lock serializing tests across separate xdist worker processes. The integ tests themselves were not run locally because they need the CI account; they will be exercised by the next CI Health V2 run.