Repository navigation
Commit a248126
authored
fix(core,mlops): honor caller region in feature_store ingest_dataframe and stop telemetry from blocking SDK calls (#6197)
* feat(mlops): allow specifying region in feature_store ingest_dataframe
ingest_dataframe() had no way to say which AWS region the FeatureGroup
lives in, so both the DescribeFeatureGroup call and the record writes
fell back to whatever region boto3 resolved, with no caller control.
Add an optional `region` argument and thread it through the whole
ingestion path:
- feature_utils.ingest_dataframe -> CoreFeatureGroup.get(region=...)
and IngestionManagerPandas(region=...)
- IngestionManagerPandas gains a `region` field, forwarded to
put_record(region=...) and batch_write_record(region=...) in the
single-thread, multi-thread, multi-process, and BatchWriteRecord paths
`region` is appended last and defaults to None, so existing calls and
their behavior are unchanged.
* fix(core): stop telemetry from blocking SDK calls
Telemetry could add unbounded latency to any decorated SDK call. A Feature
Store ingest was reported taking ~47 minutes from a private VPC while the
underlying PutRecord completed server-side in 348ms; all of the remaining time
was spent in the two telemetry emissions that follow the call.
Three defects, all in the emission path:
1. `_requests_helper` passed the timeout positionally. `requests.get` takes
`params` as its second positional argument, so the value was appended to the
query string and the request had no timeout at all. From a VPC with no route
to the telemetry endpoint the GET hung until a network device dropped the
flow. Now passed as `timeout=`, verified bounded at ~2s against a
black-holed address. The same one-line defect existed in
sagemaker-serve's telemetry_logger and is fixed there too.
2. `_get_default_sagemaker_session` hardcoded us-west-2. Module-level functions
such as `ingest_dataframe` have no session of their own, so the decorator
synthesizes one, which pointed both the STS `get_caller_identity` call and
the telemetry GET at a region the caller may have no route to. The region is
now resolved by boto3 from the caller's environment, with the default kept
only as a last resort since `Session` requires a region.
3. Emission was synchronous. `_send_telemetry_request` now dispatches to a
daemon thread and returns immediately, so neither the STS call nor the GET
can sit in the caller's critical path. Daemon threads are killed at
interpreter exit, so a pending send cannot delay shutdown either. In-flight
sends are capped and excess events dropped rather than growing threads
without bound, and nothing can escape the thread. The existing body moved to
`_send_telemetry_request_sync`.
Telemetry request failures now log at debug instead of logging a full
traceback at error level; a best-effort metric should not look like an error.
Note: events queued at process exit may now be lost. That is the intended
trade-off for never blocking the caller.
* fix(core): never drop telemetry events when sending asynchronously
Removes the in-flight cap added alongside the async send. Telemetry events
are data we cannot silently lose, so every event now gets its own daemon
thread rather than being dropped once eight sends are outstanding.
Feature Store ingestion is the special case that motivated moving the send
off the caller's thread at all: it is decorated at more than one level
(ingest_dataframe and IngestionManagerPandas.run), so a single user call
emits several events, and sending them serially turned an ingest the
service finished in under a second into a multi-minute wait.1 parent 4f2cdfd commit a248126
9 files changed
Lines changed: 582 additions & 25 deletions
File tree
- sagemaker-core
- src/sagemaker/core/telemetry
- tests/unit/telemetry
- sagemaker-mlops
- src/sagemaker/mlops/feature_store
- tests/unit/sagemaker/mlops/feature_store
- sagemaker-serve
- src/sagemaker/serve/utils
- tests/unit
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| |||
66 | 67 | | |
67 | 68 | | |
68 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
69 | 87 | | |
70 | 88 | | |
71 | 89 | | |
| |||
422 | 440 | | |
423 | 441 | | |
424 | 442 | | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
425 | 478 | | |
426 | 479 | | |
427 | 480 | | |
| |||
449 | 502 | | |
450 | 503 | | |
451 | 504 | | |
452 | | - | |
| 505 | + | |
453 | 506 | | |
454 | 507 | | |
455 | 508 | | |
| |||
482 | 535 | | |
483 | 536 | | |
484 | 537 | | |
485 | | - | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
486 | 544 | | |
487 | 545 | | |
488 | | - | |
| 546 | + | |
489 | 547 | | |
490 | | - | |
| 548 | + | |
491 | 549 | | |
492 | 550 | | |
493 | 551 | | |
| |||
511 | 569 | | |
512 | 570 | | |
513 | 571 | | |
514 | | - | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
515 | 582 | | |
516 | | - | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
517 | 590 | | |
518 | 591 | | |
519 | 592 | | |
0 commit comments