Skip to content

Commit 9d44b40

Browse files
authored
feat(train): Add inherited list_supported_models to BaseTrainer (#6187)
* feat(train): Add inherited list_supported_models to BaseTrainer Expose a list_supported_models capability on all fine-tuning trainers (SFT, RLVR, RLAIF, DPO, CPT) mirroring MultiTurnRLTrainer. Rather than duplicating the method per trainer, add a single inherited classmethod on BaseTrainer that resolves cls._customization_technique and delegates to the existing recipe_utils._list_hub_models_by_recipe primitive. RLAIFTrainer previously used the technique inline; add its _customization_technique class attribute so the inherited method resolves. MultiTurnRLTrainer keeps its own override (CreateJob flow, non-enum technique). Add unit tests covering per-trainer technique resolution, the base delegation path, and the missing-technique guard. --- X-AI-Prompt: Add list_supported_models to SFT/RLVR/RLAIF/DPO trainers like MTRL; refactored to a shared BaseTrainer method X-AI-Tool: Kiro * test(train): Add parametrized integ test for list_supported_models Add an integration test that queries SageMakerPublicHub for each fine-tuning trainer (SFT, RLVR, RLAIF, DPO, CPT) and asserts the inherited list_supported_models returns a non-empty, sorted list of model names. This closes the contract gap unit tests cannot cover: that each trainer's _customization_technique string matches the live hub recipe keywords (@recipe:finetuning_{technique}_...). Mirrors the existing MTRL integ test. --- X-AI-Prompt: Add a parametrized integ test for list_supported_models across all fine-tuning trainers X-AI-Tool: Kiro * fix(train): Match suffix-less recipe keywords in model listing _list_hub_models_by_recipe built the search keyword as "@recipe:{type}_{technique}_" (trailing underscore) and matched by prefix, assuming every recipe keyword carries a "_{strategy}" component. Techniques with no strategy — CPT is tagged as the bare "@recipe:finetuning_cpt" — never matched, so CPTTrainer.list_supported_models() always returned an empty list despite the hub having CPT-tagged models. Match the bare base keyword OR the "{base}_{strategy}" form. The "{base}_" guard prevents a shorter technique from matching a longer one that shares its prefix (e.g. "rl" must not match "rlvr"). Rework the integ test into a hub-content-agnostic oracle: independently scan the active hub and assert list_supported_models returns exactly the tagged set per technique (0 or many), instead of a brittle non-empty assertion that broke under the private test hub which carries no CPT models. Add unit coverage for the bare-keyword match and the prefix-collision guard. Verified against SageMakerPublicHub: SFT=36, RLVR=33, RLAIF=31, DPO=32, CPT=5 models, each matching the independent oracle. --- X-AI-Prompt: Integ test surfaced CPT list returning empty; fixed suffix-less recipe keyword matching and made the integ test hub-agnostic X-AI-Tool: Kiro
1 parent 7198091 commit 9d44b40

10 files changed

Lines changed: 317 additions & 8 deletions

‎sagemaker-train/src/sagemaker/train/base_trainer.py‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,37 @@ class BaseTrainer(ABC):
102102
training_image: Optional[str] = None
103103
latest_training_job: Optional[TrainingJob] = None
104104

105+
@classmethod
106+
@_telemetry_emitter(
107+
feature=Feature.MODEL_CUSTOMIZATION,
108+
func_name="BaseTrainer.list_supported_models",
109+
)
110+
def list_supported_models(cls, session=None) -> List[str]:
111+
"""Return the models that support this trainer's fine-tuning technique.
112+
113+
Queries SageMakerPublicHub for all models whose ``RecipeCollection``
114+
contains a FineTuning recipe for this trainer's customization technique
115+
(``cls._customization_technique``, e.g. ``"SFT"``, ``"DPO"``,
116+
``"RLVR"``, ``"RLAIF"``, ``"CPT"``).
117+
118+
Args:
119+
session: Optional boto3 session.
120+
121+
Returns:
122+
Sorted list of hub content model names supporting the technique.
123+
"""
124+
from sagemaker.train.common_utils.recipe_utils import _list_hub_models_by_recipe
125+
126+
technique = getattr(cls, "_customization_technique", None)
127+
if not technique:
128+
raise NotImplementedError(
129+
f"{cls.__name__} does not define a customization technique and "
130+
"cannot list supported models."
131+
)
132+
return _list_hub_models_by_recipe(
133+
recipe_type="FineTuning", technique=technique, session=session
134+
)
135+
105136
def __init__(
106137
self,
107138
sagemaker_session: Optional[Session] = None,

‎sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py‎

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -506,20 +506,25 @@ def resolve_recipe(
506506

507507

508508
def _build_recipe_keyword(recipe_type: str, technique: str) -> str:
509-
"""Build the ``@recipe:`` search keyword for a recipe type and technique.
509+
"""Build the base ``@recipe:`` search keyword for a recipe type and technique.
510510
511-
The hub tags recipes as ``@recipe:{type}_{technique}_{strategy}`` (all
512-
lowercase). We match on the ``@recipe:{type}_{technique}_`` prefix so
513-
the strategy component (e.g. ``lora``) is ignored.
511+
The hub tags recipes as either ``@recipe:{type}_{technique}_{strategy}``
512+
(e.g. ``@recipe:finetuning_sft_lora``) or, for techniques with no strategy
513+
component, the bare ``@recipe:{type}_{technique}`` (e.g.
514+
``@recipe:finetuning_cpt``) — all lowercase. This returns the base
515+
``@recipe:{type}_{technique}`` form (no trailing underscore); callers match
516+
it exactly OR as a ``{base}_`` prefix so the optional strategy component is
517+
ignored without matching an unrelated technique that merely shares a prefix
518+
(e.g. base ``..._rl`` must not match ``..._rlvr_...``).
514519
515520
Args:
516521
recipe_type: ``"FineTuning"`` or ``"Evaluation"``.
517522
technique: Technique value, e.g. ``"MTRL"`` or ``"MTRLEvaluation"``.
518523
519524
Returns:
520-
Lowercase keyword prefix string, e.g. ``"@recipe:finetuning_mtrl_"``.
525+
Lowercase base keyword string, e.g. ``"@recipe:finetuning_mtrl"``.
521526
"""
522-
return f"@recipe:{recipe_type}_{technique}_".lower()
527+
return f"@recipe:{recipe_type}_{technique}".lower()
523528

524529

525530
def _list_hub_models_by_recipe(
@@ -552,7 +557,7 @@ def _list_hub_models_by_recipe(
552557
f"recipe_type must be 'FineTuning' or 'Evaluation', got: {recipe_type!r}"
553558
)
554559

555-
keyword_prefix = _build_recipe_keyword(recipe_type, technique)
560+
keyword_base = _build_recipe_keyword(recipe_type, technique)
556561

557562
region = (getattr(session, "region_name", None) or
558563
getattr(getattr(session, "boto_session", None), "region_name", None) or
@@ -577,7 +582,14 @@ def _list_hub_models_by_recipe(
577582
if not content_name:
578583
continue
579584
keywords = summary.get("HubContentSearchKeywords", [])
580-
if any(kw.lower().startswith(keyword_prefix) for kw in keywords):
585+
# Match the bare base keyword (techniques with no strategy component,
586+
# e.g. "@recipe:finetuning_cpt") OR the "{base}_{strategy}" form
587+
# (e.g. "@recipe:finetuning_sft_lora"). The "{base}_" guard prevents
588+
# matching an unrelated technique that merely shares a prefix.
589+
if any(
590+
(kwl := kw.lower()) == keyword_base or kwl.startswith(keyword_base + "_")
591+
for kw in keywords
592+
):
581593
matched_models.append(content_name)
582594

583595
next_token = response.get("NextToken")

‎sagemaker-train/src/sagemaker/train/rlaif_trainer.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,8 @@ class RLAIFTrainer(BaseTrainer):
129129
and 'job_name_prefix'. If not specified, no notifications are sent.
130130
"""
131131

132+
_customization_technique = CustomizationTechnique.RLAIF.value
133+
132134
def __init__(
133135
self,
134136
model: Union[str, ModelPackage],
Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
#
2+
# Licensed under the Apache License, Version 2.0 (the "License"). You
3+
# may not use this file except in compliance with the License. A copy of
4+
# the License is located at
5+
#
6+
# http://aws.amazon.com/apache2.0/
7+
#
8+
# or in the "license" file accompanying this file. This file is
9+
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
10+
# ANY KIND, either express or implied. See the License for the specific
11+
# language governing permissions and limitations under the License.
12+
"""Integration tests for ``BaseTrainer.list_supported_models`` across the
13+
fine-tuning trainers (SFT / RLVR / RLAIF / DPO / CPT).
14+
15+
These tests run against real SageMaker services in prod us-west-2 and query the
16+
active SageMaker hub (the integ harness pins a private ``SAGEMAKER_HUB_NAME`` in
17+
``conftest.py``; falls back to ``SageMakerPublicHub``).
18+
19+
They verify the contract that unit tests (which mock the hub) cannot: that each
20+
trainer's ``_customization_technique`` string matches how the live hub tags its
21+
FineTuning recipes. Rather than assert a hard non-empty count -- which is brittle
22+
because the pinned hub may not carry every technique -- the test independently
23+
scans the hub once and asserts ``list_supported_models`` returns exactly the set
24+
of models tagged for that technique (whether that is zero or many). A helper
25+
regression that (for example) required a ``_{strategy}`` suffix -- and so dropped
26+
suffix-less techniques like CPT (``@recipe:finetuning_cpt``) -- would surface
27+
here as a mismatch against the oracle.
28+
"""
29+
from __future__ import annotations
30+
31+
import collections
32+
import os
33+
34+
import boto3
35+
import pytest
36+
from sagemaker.core.helper.session_helper import Session
37+
from sagemaker.train.sft_trainer import SFTTrainer
38+
from sagemaker.train.rlvr_trainer import RLVRTrainer
39+
from sagemaker.train.rlaif_trainer import RLAIFTrainer
40+
from sagemaker.train.dpo_trainer import DPOTrainer
41+
from sagemaker.train.cpt_trainer import CPTTrainer
42+
43+
_REGION = "us-west-2"
44+
_FINETUNING_PREFIX = "@recipe:finetuning_"
45+
46+
# (trainer class, expected customization technique string)
47+
_TRAINER_CASES = [
48+
pytest.param(SFTTrainer, "SFT", id="SFT"),
49+
pytest.param(RLVRTrainer, "RLVR", id="RLVR"),
50+
pytest.param(RLAIFTrainer, "RLAIF", id="RLAIF"),
51+
pytest.param(DPOTrainer, "DPO", id="DPO"),
52+
pytest.param(CPTTrainer, "CPT", id="CPT"),
53+
]
54+
55+
56+
@pytest.fixture(scope="module")
57+
def sagemaker_session():
58+
boto_session = boto3.Session(region_name=_REGION)
59+
yield Session(boto_session=boto_session)
60+
61+
62+
@pytest.fixture(scope="module")
63+
def hub_finetuning_models(sagemaker_session):
64+
"""Independent oracle: scan the active hub once and map technique token ->
65+
sorted list of model names tagged with a matching FineTuning recipe.
66+
67+
Built independently of the SDK helper (groups by the token immediately after
68+
``@recipe:finetuning_``), so it can catch a regression in that helper rather
69+
than merely re-deriving it.
70+
"""
71+
client = sagemaker_session.boto_session.client("sagemaker", region_name=_REGION)
72+
hub_name = os.environ.get("SAGEMAKER_HUB_NAME", "SageMakerPublicHub")
73+
mapping: dict[str, set] = collections.defaultdict(set)
74+
next_token = None
75+
while True:
76+
kwargs = {"HubName": hub_name, "HubContentType": "Model"}
77+
if next_token:
78+
kwargs["NextToken"] = next_token
79+
response = client.list_hub_contents(**kwargs)
80+
for summary in response.get("HubContentSummaries", []):
81+
name = summary.get("HubContentName")
82+
if not name:
83+
continue
84+
for keyword in summary.get("HubContentSearchKeywords", []):
85+
kwl = keyword.lower()
86+
if kwl.startswith(_FINETUNING_PREFIX):
87+
token = kwl[len(_FINETUNING_PREFIX):].split("_")[0]
88+
mapping[token].add(name)
89+
next_token = response.get("NextToken")
90+
if not next_token:
91+
break
92+
return {tech: sorted(names) for tech, names in mapping.items()}
93+
94+
95+
class TestTrainerListSupportedModels:
96+
"""List supported models per fine-tuning technique (requires API access)."""
97+
98+
@pytest.mark.parametrize("trainer_cls,expected_technique", _TRAINER_CASES)
99+
def test_list_supported_models(
100+
self, trainer_cls, expected_technique, sagemaker_session, hub_finetuning_models
101+
):
102+
"""Each trainer resolves its technique and returns exactly the hub models
103+
tagged for it."""
104+
# Sanity: the class attribute the inherited method keys off is set.
105+
assert trainer_cls._customization_technique == expected_technique
106+
107+
result = trainer_cls.list_supported_models(
108+
session=sagemaker_session.boto_session
109+
)
110+
111+
# Structural contract: a sorted, de-duplicated list of non-empty strings.
112+
assert isinstance(result, list)
113+
assert all(isinstance(name, str) and name for name in result)
114+
assert result == sorted(result)
115+
assert len(set(result)) == len(result)
116+
117+
# Correctness contract: exactly the models the active hub tags for this
118+
# technique (may legitimately be empty if the pinned hub carries none).
119+
expected = hub_finetuning_models.get(expected_technique.lower(), [])
120+
assert result == expected
121+
122+
def test_public_hub_has_models_for_core_techniques(self, hub_finetuning_models):
123+
"""Guard against a silent all-empty hub / broken scan: only meaningful
124+
against the public hub, where these techniques are known to be tagged.
125+
Skipped when a private test hub is pinned."""
126+
if os.environ.get("SAGEMAKER_HUB_NAME", "SageMakerPublicHub") != "SageMakerPublicHub":
127+
pytest.skip("private hub pinned; model population is environment-specific")
128+
for technique in ("sft", "dpo", "rlvr", "rlaif", "cpt"):
129+
assert hub_finetuning_models.get(technique), (
130+
f"public hub returned no models for technique '{technique}'"
131+
)

‎sagemaker-train/tests/unit/train/test_base_trainer_compute.py‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -300,3 +300,33 @@ def test_model_source_passed_as_override_parameter(
300300
start_cmd = mock_subprocess.run.call_args_list[-1].args[0]
301301
overrides = json.loads(start_cmd[start_cmd.index("--override-parameters") + 1])
302302
assert overrides["recipes.run.model_name_or_path"] == "s3://bucket/checkpoint/step_10"
303+
304+
305+
class TestBaseTrainerListSupportedModels:
306+
"""The inherited ``list_supported_models`` classmethod on ``BaseTrainer``."""
307+
308+
def test_delegates_with_class_technique(self):
309+
class _TechTrainer(BaseTrainer):
310+
_customization_technique = "SFT"
311+
312+
def train(self, *args, **kwargs): # pragma: no cover - abstract impl
313+
return None
314+
315+
with patch(
316+
"sagemaker.train.common_utils.recipe_utils._list_hub_models_by_recipe"
317+
) as mock_list:
318+
mock_list.return_value = ["meta-llama/Llama-3"]
319+
result = _TechTrainer.list_supported_models()
320+
321+
assert result == ["meta-llama/Llama-3"]
322+
mock_list.assert_called_once_with(
323+
recipe_type="FineTuning", technique="SFT", session=None
324+
)
325+
326+
def test_raises_when_technique_missing(self):
327+
class _NoTechTrainer(BaseTrainer):
328+
def train(self, *args, **kwargs): # pragma: no cover - abstract impl
329+
return None
330+
331+
with pytest.raises(NotImplementedError, match="customization technique"):
332+
_NoTechTrainer.list_supported_models()

‎sagemaker-train/tests/unit/train/test_dpo_trainer.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -738,3 +738,15 @@ def test_dry_run_returns_none_without_submitting(
738738
mock_create.assert_not_called()
739739
mock_role.assert_called_once()
740740
mock_validate_hp.assert_called_once()
741+
742+
743+
class TestDPOTrainerListSupportedModels:
744+
745+
@patch("sagemaker.train.common_utils.recipe_utils._list_hub_models_by_recipe")
746+
def test_list_supported_models(self, mock_list):
747+
mock_list.return_value = ["meta-llama/Llama-3"]
748+
result = DPOTrainer.list_supported_models()
749+
assert result == ["meta-llama/Llama-3"]
750+
mock_list.assert_called_once_with(
751+
recipe_type="FineTuning", technique="DPO", session=None
752+
)

‎sagemaker-train/tests/unit/train/test_multi_turn_rl_trainer.py‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -666,6 +666,51 @@ def test_invalid_recipe_type_raises(self):
666666
with pytest.raises(ValueError, match="recipe_type must be"):
667667
_list_hub_models_by_recipe(recipe_type="Invalid", technique="MTRL")
668668

669+
@patch("sagemaker.train.common_utils.recipe_utils.boto3.Session")
670+
def test_finds_models_with_bare_keyword_no_strategy(self, mock_session_cls):
671+
"""Techniques whose recipes carry no strategy suffix are tagged with the
672+
bare ``@recipe:finetuning_{technique}`` keyword (e.g. CPT). The matcher
673+
must find these, not just ``{base}_{strategy}`` forms."""
674+
mock_client = MagicMock()
675+
mock_session_cls.return_value.client.return_value = mock_client
676+
677+
mock_client.list_hub_contents.return_value = {
678+
"HubContentSummaries": [
679+
{
680+
"HubContentName": "model-cpt-bare",
681+
"HubContentSearchKeywords": ["@recipe:finetuning_cpt"],
682+
},
683+
{
684+
"HubContentName": "model-cpt-suffixed",
685+
"HubContentSearchKeywords": ["@recipe:finetuning_cpt_full"],
686+
},
687+
],
688+
}
689+
690+
from sagemaker.train.common_utils.recipe_utils import _list_hub_models_by_recipe
691+
result = _list_hub_models_by_recipe(recipe_type="FineTuning", technique="CPT")
692+
assert result == ["model-cpt-bare", "model-cpt-suffixed"]
693+
694+
@patch("sagemaker.train.common_utils.recipe_utils.boto3.Session")
695+
def test_does_not_match_technique_sharing_a_prefix(self, mock_session_cls):
696+
"""A shorter technique must not match a longer one that merely shares its
697+
prefix (e.g. ``rl`` must not match ``rlvr``)."""
698+
mock_client = MagicMock()
699+
mock_session_cls.return_value.client.return_value = mock_client
700+
701+
mock_client.list_hub_contents.return_value = {
702+
"HubContentSummaries": [
703+
{
704+
"HubContentName": "model-rlvr",
705+
"HubContentSearchKeywords": ["@recipe:finetuning_rlvr_lora"],
706+
},
707+
],
708+
}
709+
710+
from sagemaker.train.common_utils.recipe_utils import _list_hub_models_by_recipe
711+
result = _list_hub_models_by_recipe(recipe_type="FineTuning", technique="rl")
712+
assert result == []
713+
669714

670715
class TestListAgentRuntimes:
671716
@patch("sagemaker.train.multi_turn_rl_trainer.boto3.Session")

‎sagemaker-train/tests/unit/train/test_rlaif_trainer.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -805,3 +805,15 @@ def test_train_passes_sequence_length_to_serverless_config(
805805
mock_serverless_config.assert_called_once()
806806
call_kwargs = mock_serverless_config.call_args[1]
807807
assert call_kwargs["sequence_length"] == "64K"
808+
809+
810+
class TestRLAIFTrainerListSupportedModels:
811+
812+
@patch("sagemaker.train.common_utils.recipe_utils._list_hub_models_by_recipe")
813+
def test_list_supported_models(self, mock_list):
814+
mock_list.return_value = ["meta-llama/Llama-3"]
815+
result = RLAIFTrainer.list_supported_models()
816+
assert result == ["meta-llama/Llama-3"]
817+
mock_list.assert_called_once_with(
818+
recipe_type="FineTuning", technique="RLAIF", session=None
819+
)

‎sagemaker-train/tests/unit/train/test_rlvr_trainer.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -763,3 +763,15 @@ def test_dry_run_returns_none_without_submitting(
763763
mock_create.assert_not_called()
764764
mock_role.assert_called_once()
765765
mock_validate_hp.assert_called_once()
766+
767+
768+
class TestRLVRTrainerListSupportedModels:
769+
770+
@patch("sagemaker.train.common_utils.recipe_utils._list_hub_models_by_recipe")
771+
def test_list_supported_models(self, mock_list):
772+
mock_list.return_value = ["meta-llama/Llama-3"]
773+
result = RLVRTrainer.list_supported_models()
774+
assert result == ["meta-llama/Llama-3"]
775+
mock_list.assert_called_once_with(
776+
recipe_type="FineTuning", technique="RLVR", session=None
777+
)

‎sagemaker-train/tests/unit/train/test_sft_trainer.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1556,3 +1556,25 @@ def test_dry_run_raises_on_role_validation_failure(
15561556

15571557
with pytest.raises(ValueError, match="Missing permissions"):
15581558
trainer.train(dry_run=True)
1559+
1560+
1561+
class TestSFTTrainerListSupportedModels:
1562+
1563+
@patch("sagemaker.train.common_utils.recipe_utils._list_hub_models_by_recipe")
1564+
def test_list_supported_models(self, mock_list):
1565+
mock_list.return_value = ["meta-llama/Llama-3", "Qwen/Qwen3-32B"]
1566+
result = SFTTrainer.list_supported_models()
1567+
assert isinstance(result, list)
1568+
assert "Qwen/Qwen3-32B" in result
1569+
mock_list.assert_called_once_with(
1570+
recipe_type="FineTuning", technique="SFT", session=None
1571+
)
1572+
1573+
@patch("sagemaker.train.common_utils.recipe_utils._list_hub_models_by_recipe")
1574+
def test_list_supported_models_passes_session(self, mock_list):
1575+
mock_list.return_value = []
1576+
session = Mock()
1577+
SFTTrainer.list_supported_models(session=session)
1578+
mock_list.assert_called_once_with(
1579+
recipe_type="FineTuning", technique="SFT", session=session
1580+
)

0 commit comments

Comments
 (0)