Skip to content

Commit e698fc6

Browse files
e-davidsonEli Davidson
andauthored
fix(serve): propagate additional_model_data_sources for JumpStart models (#6151)
* fix(serve): propagate additional_model_data_sources for JumpStart models ModelBuilder._build_for_jumpstart() copied image_uri, env, enable_network_isolation, model_reference_arn, and model_data from the resolved JumpStart init_kwargs, but never propagated additional_model_data_sources. Any JumpStart model whose spec declares an extra S3 data channel (e.g. an EAGLE speculative decoding draft model, a LoRA adapter, or a tokenizer override) therefore had that channel silently dropped from the CreateModel call, and the container failed at runtime trying to read artifacts from a path that was never mounted. Propagate additional_model_data_sources from init_kwargs onto the builder so _prepare_container_def_base attaches it to the CreateModel request. Each source is filtered to only the keys the CreateModel API accepts (ChannelName, S3DataSource); JumpStart specs may include extra metadata keys such as HostingEulaKey that the API rejects. Adds unit tests for both the propagation+filtering path and the no-op case when the spec declares no additional data sources. * fix(serve): enforce accept_eula on gated additional model data sources A source carrying a truthy HostingEulaKey is gated and requires license acceptance, exactly like the base model. container_def folds accept_eula into the base model's ModelAccessConfig but passes additional sources through untouched, so the acceptance is translated onto each gated source here: fold ModelAccessConfig={AcceptEula: True} into its S3DataSource when accept_eula is set, raise the same error the base model raises when it is not. Ungated sources just drop the JumpStart-internal HostingEulaKey the CreateModel API rejects. Test coverage mirrors the v2 helper's case inventory: multiple public sources, ungated ignores acceptance, mixed public+gated, gated accept, gated missing/false acceptance raises, plus a raw-spec-JSON end-to-end test driving the real JumpStartModelSpecs parsing and factory shaping. * test(serve): drive additional-source EULA tests through the user flow Replace the mock-harness additional_model_data_sources tests with user-flow tests that exercise the documented user contract: construct a real ModelBuilder, set accept_eula, and call build(). Only the spec fetch boundary (JumpStartModelsAccessor.get_model_specs / _get_manifest) and the AWS boundary (_prepare_for_mode / _create_model, mock session) are mocked, mirroring how master-v2 tested JumpStartModel. Raw spec JSON fixtures live in data/jumpstart_specs.json as a base spec plus overlay variants (public model without/with public sources, gated sources, mixed), reusable across tests. Everything between the two boundaries runs for real: JumpStart detection, JumpStartModelSpecs parsing, the get_init_kwargs factory shaping, and the EULA translation in _build_for_jumpstart. * test(serve): use real captured JumpStart specs as test fixtures Replace the synthetic base-spec-plus-variants fixture with real, unmodified specs captured from the production JumpStart content bucket, mirroring master-v2's PROTOTYPICAL_MODEL_SPECS_DICT pattern: the spec accessor routes by model id into one captured spec file per model, and each test calls the SDK exactly like a customer does. Fixture models (tests/unit/servers/data/jumpstart_specs/): - pytorch-ic-mobilenet-v2 v3.0.25: public, no additional data sources - openai-reasoning-gpt-oss-20b v3.38.0: ungated EAGLE speculative decoding source in its default (lmi-optimized) config -- the model that surfaced the propagation bug - meta-textgeneration-llama-3-1-70b v2.16.2: gated draft_model source (hosting_eula_key) in its lmi-optimized config, selected in tests via the public set_deployment_config API All patching is centralized in setUp (spec fetch, manifest, and the AWS boundary), so test bodies contain only public API calls: ModelBuilder(model=<real id>, instance_type=...), accept_eula, and build(). Real inference-config resolution now runs in every test. * test(serve): assert the CreateModel request shape via the single-call API Restructure the additional-model-data-source tests on three axes: 1. Test names describe the case under test (gated source with accepted EULA sends ModelAccessConfig, unselected config sources do not leak, etc.), not the fixture model. 2. Each test drives the documented single-call customer API: ModelBuilder.from_jumpstart_config(JumpStartConfig(model_id, accept_eula, inference_config_name), compute=Compute(...)) followed by build(). No attribute pokes and no separate config-selection call. 3. Assertions target the CreateModel request itself: _create_model is no longer patched, so the real _create_sagemaker_model and _prepare_container_def_base run, and every test inspects the container_defs captured from sagemaker_session.create_model. This covers the full downstream impact: AdditionalModelDataSources presence/absence/shape, ModelAccessConfig on both the gated additional source and the primary ModelDataSource, and create_model never being called when the EULA gate raises. * test(serve): derive expected sources from the spec fixtures Replace the hand-copied expected-source class constants with a helper that reads the captured spec fixture and applies the expected transformation declaratively (PascalCase key mapping, content-bucket resolution, HostingEulaKey absence, EULA folding). The fixture stays the single source of truth for data values; only the behavior under test remains spelled out in the test, avoiding both duplication and tautological assertions. * refactor(core): apply accept_eula to additional sources in container_def Move the additional-model-data-source EULA handling out of the JumpStart-specific block in _build_for_jumpstart and into _apply_accept_eula_to_additional_model_data_sources next to container_def, the single place the primary model's accept_eula already becomes ModelAccessConfig. Both the primary model and the additional sources are now driven by the same accept_eula flag in adjacent code, and every container-assembly call site (including the deployment-config path that never passes through _build_for_jumpstart) gets the gated-source enforcement. _build_for_jumpstart is reduced to pure propagation. No behavior change on the JumpStart path: all 63 serve tests pass unchanged, plus 103 core session_helper tests. * refactor(serve): inline additional-source EULA fold in _build_for_jumpstart Scope the EULA handling down to mirror the primary model exactly: strip the JumpStart-internal HostingEulaKey (not part of the CreateModel API shape) and fold accept_eula into each source's ModelAccessConfig when set. No client-side gatedness check or raise: the control plane determines gatedness from the artifact's bucket and rejects CreateModel with its own EULA validation error, for the primary model and additional sources alike. This reverts the sagemaker-core half of the previous commit: session_helper.py returns to upstream and the whole change lives in the JumpStart builder that produces the spec-derived sources. Tests assert the new contract on the CreateModel request: acceptance folded when accept_eula is set (True or False), no ModelAccessConfig when unset. * test(serve): state expected request values inline, trim to key assertions Remove the expected-value helpers whose conditional re-implemented the accept_eula fold rule, making the tests a parallel implementation of the code under test. Expectations are now hand-written literals stated in each test body: one exact-shape pin for the propagation case, and targeted assertions (ModelAccessConfig value or absence, HostingEulaKey absence, resolved bucket URI) for the EULA cases. Mutation coverage is unchanged: the naive-propagation mutant still fails the same 5 tests. --------- Co-authored-by: Eli Davidson <elleedee@amazon.com>
1 parent 3c6d362 commit e698fc6

6 files changed

Lines changed: 7090 additions & 0 deletions

File tree

‎sagemaker-serve/src/sagemaker/serve/model_builder_servers.py‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1014,6 +1014,39 @@ def _build_for_jumpstart(self) -> Model:
10141014
if getattr(init_kwargs, "model_reference_arn", None):
10151015
self.model_reference_arn = init_kwargs.model_reference_arn
10161016

1017+
# Propagate additional model data sources resolved from the JumpStart
1018+
# spec (e.g. speculative decoding draft models like EAGLE). The JumpStart
1019+
# factory (_add_additional_model_data_sources_to_kwargs) already returns
1020+
# these in CreateModel API shape via camel_case_to_pascal_case(...),
1021+
# except for two adjustments applied here, mirroring how ``container_def``
1022+
# treats the primary model's ``ModelDataSource``:
1023+
# - the JumpStart-internal ``HostingEulaKey`` is removed (it is not part
1024+
# of the CreateModel API shape, request validation rejects it), and
1025+
# - when ``accept_eula`` is set, it is folded into each source's
1026+
# ``S3DataSource.ModelAccessConfig``.
1027+
# Gated-source EULA enforcement is server-side, same as for the primary
1028+
# model: the control plane determines gatedness from the artifact's
1029+
# bucket and rejects CreateModel with an EULA validation error when a
1030+
# gated source lacks ``ModelAccessConfig`` with ``AcceptEula`` true.
1031+
# Without this propagation, sources declared in the spec are dropped
1032+
# from the CreateModel call and the container fails to find the
1033+
# referenced artifacts at runtime.
1034+
additional_model_data_sources = getattr(
1035+
init_kwargs, "additional_model_data_sources", None
1036+
)
1037+
if isinstance(additional_model_data_sources, list) and additional_model_data_sources:
1038+
accept_eula = getattr(self, "accept_eula", None)
1039+
prepared_sources = []
1040+
for source in additional_model_data_sources:
1041+
prepared_source = dict(source)
1042+
prepared_source.pop("HostingEulaKey", None)
1043+
if accept_eula is not None:
1044+
s3_data_source = dict(prepared_source.get("S3DataSource", {}))
1045+
s3_data_source["ModelAccessConfig"] = {"AcceptEula": accept_eula}
1046+
prepared_source["S3DataSource"] = s3_data_source
1047+
prepared_sources.append(prepared_source)
1048+
self.additional_model_data_sources = prepared_sources
1049+
10171050
# Handle model artifacts for fine-tuned models
10181051
if hasattr(init_kwargs, "model_data") and init_kwargs.model_data:
10191052
if (
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
[
2+
{
3+
"model_id": "pytorch-ic-mobilenet-v2",
4+
"version": "3.0.25",
5+
"min_version": "2.189.0",
6+
"spec_key": "community_models/pytorch-ic-mobilenet-v2/specs_v3.0.25.json",
7+
"deprecated": false,
8+
"provider": "pytorch",
9+
"search_keywords": [
10+
"MobileNet V2",
11+
"Vision",
12+
"ResNet 18"
13+
]
14+
},
15+
{
16+
"model_id": "openai-reasoning-gpt-oss-20b",
17+
"version": "3.38.0",
18+
"min_version": "2.244.2",
19+
"spec_key": "community_models/openai-reasoning-gpt-oss-20b/specs_v3.38.0.json",
20+
"deprecated": false,
21+
"provider": "openai",
22+
"search_keywords": [
23+
"Text Generation",
24+
"Reasoning"
25+
]
26+
},
27+
{
28+
"model_id": "meta-textgeneration-llama-3-1-70b",
29+
"version": "2.16.2",
30+
"min_version": "2.232.1",
31+
"spec_key": "community_models/meta-textgeneration-llama-3-1-70b/specs_v2.16.2.json",
32+
"deprecated": false,
33+
"provider": "meta",
34+
"search_keywords": [
35+
"Text Generation"
36+
]
37+
}
38+
]

0 commit comments

Comments
 (0)