Repository navigation
Commit cd42fee
Sagar Dubey
fix(tgi): honor S3 model_path as weight source for TGI builds
ModelBuilder with ModelServer.TGI silently ignored an S3 weight source
(model_path="s3://..." or s3_model_data_url="s3://..."): it created a
literal local "s3:/..." directory and set HF_MODEL_ID to the HF repo id,
so the container always downloaded weights from huggingface.co.
_build_for_tgi now detects an S3 weight source, skips the local mkdir for
it, attaches the S3 prefix as an uncompressed ModelDataSource, and sets
HF_MODEL_ID=/opt/ml/model and HF_HUB_OFFLINE=1 (via setdefault, preserving
any user-supplied HF_MODEL_ID). Genuine local paths, HF-Hub downloads,
JumpStart, and all non-TGI servers are unchanged.
Also fixes two defects surfaced during real deployment:
- HF_HUB_OFFLINE was reset to "0" at the end of the build; it now stays
"1" for the S3-mounted path so TGI loads from /opt/ml/model.
- The uncompressed ModelDataSource S3Uri could end in "//" when the input
prefix already had a trailing slash; normalized to exactly one slash so
S3Prefix matching finds the weight objects.
Adds unit and regression tests for all of the above.1 parent db63167 commit cd42fee
4 files changed
Lines changed: 641 additions & 4 deletions
File tree
- sagemaker-serve
- src/sagemaker/serve
- model_server/tgi
- tests/unit
- model_server
- servers
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| 28 | + | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
| |||
205 | 206 | | |
206 | 207 | | |
207 | 208 | | |
208 | | - | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
209 | 224 | | |
210 | 225 | | |
211 | 226 | | |
212 | | - | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
213 | 233 | | |
214 | 234 | | |
215 | 235 | | |
| |||
252 | 272 | | |
253 | 273 | | |
254 | 274 | | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
255 | 280 | | |
256 | 281 | | |
257 | 282 | | |
| |||
284 | 309 | | |
285 | 310 | | |
286 | 311 | | |
287 | | - | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
288 | 316 | | |
289 | 317 | | |
290 | 318 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
131 | 131 | | |
132 | 132 | | |
133 | 133 | | |
134 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
135 | 139 | | |
136 | 140 | | |
137 | 141 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
127 | 127 | | |
128 | 128 | | |
129 | 129 | | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
130 | 157 | | |
131 | 158 | | |
132 | 159 | | |
| |||
0 commit comments