Repository navigation
Commit 898a72a
* feat: add args field to SourceCode for command-based training (#5226)
* fix: stop the generated train script from defeating shlex.quote on args
The `args` shell-quoting introduced with SourceCode.args did not actually hold,
because EXECUTE_BASE_COMMANDS interpolated the already-quoted command inside a
double-quoted assignment:
CMD="{base_command}"
eval $CMD
Single quotes are inert inside double quotes, so bash expanded the contents at
assignment time and the quoting was gone before `eval` ever ran. Verified
against real bash, an arg of:
$HOME arrived as /Users/... (expected literal $HOME)
$(id -u) arrived as 503 (expected literal $(id -u))
`id -u` arrived as 503 (expected literal `id -u`)
a"b aborted the script with `syntax error: unexpected end of file`
it's aborted the script likewise
a\tb, a\nb collapsed to a b
So the docstring promise that values "may contain spaces or special characters"
held only for whitespace, which is the one case the original tests covered --
that is why all nine of them passed against a broken implementation.
The command is now carried through a quoted heredoc, which performs no
expansion whatsoever, and `eval "$CMD"` parses it exactly once:
CMD=$(cat <<'SAGEMAKER_BASE_COMMAND_EOF'
{base_command}
SAGEMAKER_BASE_COMMAND_EOF
)
eval "$CMD"
All 15 metacharacter cases now round-trip verbatim. `eval` is deliberately
retained so `command` keeps its existing semantics: env vars in the command
still expand and shell operators like `&&` still chain. Only the spurious
*second* expansion -- the one that defeated the quoting -- is removed.
Tests now assert on behavior instead of on the text of the generated script:
they execute the generated command block under bash against a stub that reports
its argv, which is the only way to pin a quoting contract. Two graded negative
controls confirm they do the work:
- reverting to CMD="..." -> 23 of 28 fail
- keeping the heredoc but unquoting
the delimiter (expansion only) -> exactly the 4 expansion cases fail
The five assertions that string-matched `CMD="..."` are rewritten against a
helper that tolerates either delimiter form, so a future regression fails on
semantics rather than on a parse error in the test.
No regressions: 2282 passed in the sagemaker-train unit suite.
The unused copy in sagemaker-core/src/sagemaker/core/modules/templates.py has
the same flaw but is imported by nothing, so it is left out of scope here.
---------
Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>
1 parent bb94da4 commit 898a72a
5 files changed
Lines changed: 392 additions & 4 deletions
File tree
- sagemaker-core/src/sagemaker/core
- modules
- training
- sagemaker-train
- src/sagemaker/train
- tests/unit/train
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
107 | 114 | | |
108 | 115 | | |
109 | 116 | | |
| |||
113 | 120 | | |
114 | 121 | | |
115 | 122 | | |
| 123 | + | |
116 | 124 | | |
117 | 125 | | |
118 | 126 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
107 | 107 | | |
108 | 108 | | |
109 | 109 | | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
110 | 117 | | |
111 | 118 | | |
112 | 119 | | |
| |||
116 | 123 | | |
117 | 124 | | |
118 | 125 | | |
| 126 | + | |
119 | 127 | | |
120 | 128 | | |
121 | 129 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
1332 | 1333 | | |
1333 | 1334 | | |
1334 | 1335 | | |
1335 | | - | |
1336 | | - | |
| 1336 | + | |
| 1337 | + | |
| 1338 | + | |
| 1339 | + | |
| 1340 | + | |
| 1341 | + | |
| 1342 | + | |
| 1343 | + | |
| 1344 | + | |
| 1345 | + | |
| 1346 | + | |
| 1347 | + | |
| 1348 | + | |
| 1349 | + | |
1337 | 1350 | | |
1338 | 1351 | | |
1339 | 1352 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
17 | 23 | | |
18 | | - | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
19 | 28 | | |
20 | | - | |
| 29 | + | |
21 | 30 | | |
22 | 31 | | |
23 | 32 | | |
| |||
0 commit comments