Repository navigation
Commit b981113
authored
fix(ci,train): run fast-integ-tests in CodeBuild and give shallow RLVR cases a reward signal (#6207)
* fix(train): give shallow RLVR cases the reward signal RLVR requires
RLVRTrainer.train() refuses to submit unless custom_reward_function was passed
or hyperparameters.preset_reward_function is set. TestRLVRTrainerSubmission
inherits the shared cases from RecipeTrainerCases, which pass neither -- they
are about recipe rendering and dataset handling, not reward configuration -- so
14 of the class's 17 tests failed: 12 raising the ValueError, and the two
negative cases failing with "rejected, but not for the expected reason" because
the reward error preempted the S3 validation error they assert on.
Set the preset in a build() override rather than repeating it in each test, and
skip it when the test supplies its own custom_reward_function so the three
reward-function variants still exercise exactly what they name. "prime_code" is
one of the values the recipe's preset_reward_function enum accepts ('', gsm8k,
prime_code, prime_math) and is what the deep suite pairs with an ordinary
training dataset on this same model.
This was not a regression from a later change to sagemaker-train. The guard
landed in #6181 on 2026-08-14, five days before the shallow suite merged
(#6176), and rlvr_trainer.py is unchanged since. The suite had simply never run
in CI: the fast-integ-tests job could not check out fork PR code, and because
pull_request_target runs the base branch's workflow it could not have run on
#6176 itself either.
Verified against us-west-2 in the SDK test account: 14 passed in 94s, each
submitting and immediately stopping a real training job.
---
X-AI-Prompt: Fix the failing shallow sagemaker-train RLVR integ tests, which were being rejected at submission for a missing reward signal
X-AI-Tool: claude-code
* ci: run fast-integ-tests in CodeBuild instead of on the runner
Replaces the runner-based shallow suite with a CodeBuild invocation, so the
suite gates fork PRs -- which is nearly all of them.
The job stopped working when actions/checkout began refusing to place fork PR
code in a pull_request_target job. That refusal is correct: the runner holds the
base repo's GITHUB_TOKEN and assumes CI_AWS_ROLE_ARN, so a fork could edit
conftest.py and read those credentials out. On a public repo, overriding it with
allow-unsafe-pr-checkout would be a live credential-exfiltration path.
Guarding the job to same-repo PRs would stop the failure, but 59 of the last 60
merged PRs here are from forks, so that leaves ~2% coverage. This is the real
fix: start CodeBuild with source-version-override, exactly as the
codestyle-doc-tests, unit-tests and integ-tests jobs already do. The build never
sees the runner's token, secrets or default-branch cache, so no same-repo guard
is needed.
Its own project rather than folding into sagemaker-train-integ-tests, so a
shallow failure stays distinguishable from a deep-suite failure and runs
concurrently with it rather than queueing behind it.
Dropped the upload-artifact step: the JUnit XML no longer exists on the runner,
and results are in the CodeBuild logs.
Tradeoff recorded in both the workflow comment and the suite README: the pytest
selection now lives in createCIShallowIntegBuildSpec in
SageMakerMLFPySDKInfraCDK, so changing how the suite is invoked is no longer
reviewable in a PR to this repo. Adding a test file under shallow/ is still
picked up automatically.
The project sagemaker-python-sdk-ci-sagemaker-train-fast-integ-tests is
deployed, so the job resolves on merge.
---
X-AI-Prompt: Instead of the GitHub runner, run the sagemaker-train shallow integ suite in CodeBuild like the other CI workflows, so fork PRs are gated after actions/checkout began refusing fork PR code in pull_request_target
X-AI-Tool: claude-code1 parent 3792482 commit b981113
3 files changed
Lines changed: 73 additions & 82 deletions
File tree
- .github/workflows
- sagemaker-train/tests/integ/train/shallow
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
246 | 246 | | |
247 | 247 | | |
248 | 248 | | |
249 | | - | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
254 | | - | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
255 | 260 | | |
256 | | - | |
257 | | - | |
258 | | - | |
259 | | - | |
260 | | - | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
261 | 279 | | |
262 | 280 | | |
263 | 281 | | |
| |||
272 | 290 | | |
273 | 291 | | |
274 | 292 | | |
| 293 | + | |
| 294 | + | |
275 | 295 | | |
276 | 296 | | |
277 | | - | |
278 | | - | |
279 | | - | |
280 | | - | |
281 | | - | |
282 | | - | |
283 | | - | |
284 | | - | |
285 | | - | |
286 | | - | |
287 | | - | |
288 | 297 | | |
289 | 298 | | |
290 | 299 | | |
291 | 300 | | |
292 | 301 | | |
293 | 302 | | |
294 | 303 | | |
295 | | - | |
296 | | - | |
297 | | - | |
298 | | - | |
299 | | - | |
300 | | - | |
301 | | - | |
302 | 304 | | |
303 | | - | |
304 | | - | |
305 | | - | |
306 | | - | |
307 | | - | |
308 | | - | |
309 | | - | |
310 | | - | |
311 | | - | |
312 | | - | |
313 | | - | |
314 | | - | |
315 | | - | |
316 | | - | |
317 | | - | |
318 | | - | |
319 | | - | |
320 | | - | |
321 | | - | |
322 | | - | |
323 | | - | |
324 | | - | |
325 | | - | |
326 | | - | |
327 | | - | |
328 | | - | |
329 | | - | |
330 | | - | |
331 | | - | |
332 | | - | |
333 | | - | |
334 | | - | |
335 | | - | |
336 | | - | |
337 | | - | |
338 | | - | |
339 | | - | |
340 | | - | |
341 | | - | |
342 | | - | |
343 | | - | |
344 | | - | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | | - | |
352 | | - | |
| 305 | + | |
353 | 306 | | |
354 | | - | |
355 | | - | |
356 | | - | |
| 307 | + | |
| 308 | + | |
357 | 309 | | |
358 | 310 | | |
359 | 311 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
11 | 28 | | |
12 | 29 | | |
13 | 30 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
36 | 40 | | |
37 | 41 | | |
38 | 42 | | |
39 | 43 | | |
40 | 44 | | |
41 | 45 | | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
42 | 64 | | |
43 | 65 | | |
44 | 66 | | |
| |||
0 commit comments