Skip to content

Commit 898a72a

Browse files
mohamedzeidan2021Mohamed Zeidan
andauthored
feat: add args field to SourceCode for command-based training (#5226) (#6356)
* feat: add args field to SourceCode for command-based training (#5226) * fix: stop the generated train script from defeating shlex.quote on args The `args` shell-quoting introduced with SourceCode.args did not actually hold, because EXECUTE_BASE_COMMANDS interpolated the already-quoted command inside a double-quoted assignment: CMD="{base_command}" eval $CMD Single quotes are inert inside double quotes, so bash expanded the contents at assignment time and the quoting was gone before `eval` ever ran. Verified against real bash, an arg of: $HOME arrived as /Users/... (expected literal $HOME) $(id -u) arrived as 503 (expected literal $(id -u)) `id -u` arrived as 503 (expected literal `id -u`) a"b aborted the script with `syntax error: unexpected end of file` it's aborted the script likewise a\tb, a\nb collapsed to a b So the docstring promise that values "may contain spaces or special characters" held only for whitespace, which is the one case the original tests covered -- that is why all nine of them passed against a broken implementation. The command is now carried through a quoted heredoc, which performs no expansion whatsoever, and `eval "$CMD"` parses it exactly once: CMD=$(cat <<'SAGEMAKER_BASE_COMMAND_EOF' {base_command} SAGEMAKER_BASE_COMMAND_EOF ) eval "$CMD" All 15 metacharacter cases now round-trip verbatim. `eval` is deliberately retained so `command` keeps its existing semantics: env vars in the command still expand and shell operators like `&&` still chain. Only the spurious *second* expansion -- the one that defeated the quoting -- is removed. Tests now assert on behavior instead of on the text of the generated script: they execute the generated command block under bash against a stub that reports its argv, which is the only way to pin a quoting contract. Two graded negative controls confirm they do the work: - reverting to CMD="..." -> 23 of 28 fail - keeping the heredoc but unquoting the delimiter (expansion only) -> exactly the 4 expansion cases fail The five assertions that string-matched `CMD="..."` are rewritten against a helper that tolerates either delimiter form, so a future regression fails on semantics rather than on a parse error in the test. No regressions: 2282 passed in the sagemaker-train unit suite. The unused copy in sagemaker-core/src/sagemaker/core/modules/templates.py has the same flaw but is imported by nothing, so it is left out of scope here. --------- Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>
1 parent bb94da4 commit 898a72a

5 files changed

Lines changed: 392 additions & 4 deletions

File tree

‎sagemaker-core/src/sagemaker/core/modules/configs.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,13 @@ class SourceCode(BaseConfig):
104104
command (Optional[str]):
105105
The command(s) to execute in the training job container. Example: "python my_script.py".
106106
If not specified, entry_script must be provided.
107+
args (Optional[List[Union[str, int, float]]]):
108+
A list of arguments to append to ``command`` when it is executed in the training job
109+
container. Example: ``["--epochs", 25, "--learning_rate", 0.001]``. Each argument is
110+
shell-quoted before being appended, so values may contain spaces or special characters.
111+
Only applicable when ``command`` is provided; top-level ``hyperparameters`` are not
112+
passed as CLI arguments in ``command`` mode -- they are available inside the container
113+
via the ``SM_HPS`` environment variable.
107114
ignore_patterns: (Optional[List[str]]) :
108115
The ignore patterns to ignore specific files/folders when uploading to S3. If not specified,
109116
default to: ['.env', '.git', '__pycache__', '.DS_Store', '.cache', '.ipynb_checkpoints'].
@@ -113,6 +120,7 @@ class SourceCode(BaseConfig):
113120
requirements: Optional[str] = None
114121
entry_script: Optional[str] = None
115122
command: Optional[str] = None
123+
args: Optional[List[Union[str, int, float]]] = None
116124
ignore_patterns: Optional[List[str]] = [
117125
".env",
118126
".git",

‎sagemaker-core/src/sagemaker/core/training/configs.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,13 @@ class SourceCode(BaseConfig):
107107
command (Optional[StrPipeVar]):
108108
The command(s) to execute in the training job container. Example: "python my_script.py".
109109
If not specified, entry_script must be provided.
110+
args (Optional[List[Union[str, int, float]]]):
111+
A list of arguments to append to ``command`` when it is executed in the training job
112+
container. Example: ``["--epochs", 25, "--learning_rate", 0.001]``. Each argument is
113+
shell-quoted before being appended, so values may contain spaces or special characters.
114+
Only applicable when ``command`` is provided; top-level ``hyperparameters`` are not
115+
passed as CLI arguments in ``command`` mode -- they are available inside the container
116+
via the ``SM_HPS`` environment variable.
110117
ignore_patterns: (Optional[List[str]]) :
111118
The ignore patterns to ignore specific files/folders when uploading to S3. If not specified,
112119
default to: ['.env', '.git', '__pycache__', '.DS_Store', '.cache', '.ipynb_checkpoints'].
@@ -116,6 +123,7 @@ class SourceCode(BaseConfig):
116123
requirements: Optional[StrPipeVar] = None
117124
entry_script: Optional[StrPipeVar] = None
118125
command: Optional[StrPipeVar] = None
126+
args: Optional[List[Union[str, int, float]]] = None
119127
ignore_patterns: Optional[List[str]] = [
120128
".env",
121129
".git",

‎sagemaker-train/src/sagemaker/train/model_trainer.py‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
import os
1919
import json
2020
import re
21+
import shlex
2122
import shutil
2223
from tempfile import TemporaryDirectory
2324
from typing import Optional, List, Union, Dict, Any, ClassVar
@@ -1332,8 +1333,20 @@ def _prepare_train_script(
13321333
"Both 'command' and 'entry_script' are provided in the SourceCode. "
13331334
"Defaulting to 'command'."
13341335
)
1335-
base_command = source_code.command.split()
1336-
base_command = " ".join(base_command)
1336+
command_parts = source_code.command.split()
1337+
if source_code.args:
1338+
# Shell-quote each arg so values with spaces or special characters are
1339+
# passed through as single, literal arguments to the executed command.
1340+
command_parts.extend(shlex.quote(str(arg)) for arg in source_code.args)
1341+
base_command = " ".join(command_parts)
1342+
if self.hyperparameters:
1343+
logger.warning(
1344+
"Hyperparameters are set but are not passed as command-line arguments "
1345+
"when 'command' is used in the SourceCode. They are available inside the "
1346+
"training container via the 'SM_HPS' environment variable, e.g. "
1347+
"`hps = json.loads(os.environ['SM_HPS'])`. To pass arguments to your "
1348+
"'command', use the SourceCode 'args' field."
1349+
)
13371350

13381351
install_requirements = ""
13391352
if source_code.requirements:

‎sagemaker-train/src/sagemaker/train/templates.py‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,19 @@
1414

1515
from __future__ import absolute_import
1616

17+
# The command is carried in through a quoted heredoc ('SAGEMAKER_BASE_COMMAND_EOF'), which
18+
# performs no expansion at all, so the shell quoting applied by
19+
# ModelTrainer._prepare_train_script survives into CMD verbatim. Assigning with
20+
# CMD="{base_command}" instead would expand $VAR, $(...) and backticks at assignment time and
21+
# let an embedded double quote terminate the string early, which defeats that quoting and
22+
# corrupts SourceCode.args. `eval "$CMD"` then parses the command exactly once.
1723
EXECUTE_BASE_COMMANDS = """
18-
CMD="{base_command}"
24+
CMD=$(cat <<'SAGEMAKER_BASE_COMMAND_EOF'
25+
{base_command}
26+
SAGEMAKER_BASE_COMMAND_EOF
27+
)
1928
echo "Executing command: $CMD"
20-
eval $CMD
29+
eval "$CMD"
2130
"""
2231

2332
EXECUTE_BASIC_SCRIPT_DRIVER = """

0 commit comments

Comments
 (0)