Skip to content

Commit a248126

Browse files
authored
fix(core,mlops): honor caller region in feature_store ingest_dataframe and stop telemetry from blocking SDK calls (#6197)
* feat(mlops): allow specifying region in feature_store ingest_dataframe ingest_dataframe() had no way to say which AWS region the FeatureGroup lives in, so both the DescribeFeatureGroup call and the record writes fell back to whatever region boto3 resolved, with no caller control. Add an optional `region` argument and thread it through the whole ingestion path: - feature_utils.ingest_dataframe -> CoreFeatureGroup.get(region=...) and IngestionManagerPandas(region=...) - IngestionManagerPandas gains a `region` field, forwarded to put_record(region=...) and batch_write_record(region=...) in the single-thread, multi-thread, multi-process, and BatchWriteRecord paths `region` is appended last and defaults to None, so existing calls and their behavior are unchanged. * fix(core): stop telemetry from blocking SDK calls Telemetry could add unbounded latency to any decorated SDK call. A Feature Store ingest was reported taking ~47 minutes from a private VPC while the underlying PutRecord completed server-side in 348ms; all of the remaining time was spent in the two telemetry emissions that follow the call. Three defects, all in the emission path: 1. `_requests_helper` passed the timeout positionally. `requests.get` takes `params` as its second positional argument, so the value was appended to the query string and the request had no timeout at all. From a VPC with no route to the telemetry endpoint the GET hung until a network device dropped the flow. Now passed as `timeout=`, verified bounded at ~2s against a black-holed address. The same one-line defect existed in sagemaker-serve's telemetry_logger and is fixed there too. 2. `_get_default_sagemaker_session` hardcoded us-west-2. Module-level functions such as `ingest_dataframe` have no session of their own, so the decorator synthesizes one, which pointed both the STS `get_caller_identity` call and the telemetry GET at a region the caller may have no route to. The region is now resolved by boto3 from the caller's environment, with the default kept only as a last resort since `Session` requires a region. 3. Emission was synchronous. `_send_telemetry_request` now dispatches to a daemon thread and returns immediately, so neither the STS call nor the GET can sit in the caller's critical path. Daemon threads are killed at interpreter exit, so a pending send cannot delay shutdown either. In-flight sends are capped and excess events dropped rather than growing threads without bound, and nothing can escape the thread. The existing body moved to `_send_telemetry_request_sync`. Telemetry request failures now log at debug instead of logging a full traceback at error level; a best-effort metric should not look like an error. Note: events queued at process exit may now be lost. That is the intended trade-off for never blocking the caller. * fix(core): never drop telemetry events when sending asynchronously Removes the in-flight cap added alongside the async send. Telemetry events are data we cannot silently lose, so every event now gets its own daemon thread rather than being dropped once eight sends are outstanding. Feature Store ingestion is the special case that motivated moving the send off the caller's thread at all: it is decorated at more than one level (ingest_dataframe and IngestionManagerPandas.run), so a single user call emits several events, and sending them serially turned an ingest the service finished in under a second into a multi-minute wait.
1 parent 4f2cdfd commit a248126

9 files changed

Lines changed: 582 additions & 25 deletions

File tree

‎sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py‎

Lines changed: 79 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
import os
1717
import platform
1818
import sys
19+
import threading
1920
from time import perf_counter
2021
from typing import List
2122
import functools
@@ -66,6 +67,23 @@
6667
)
6768
_telemetry_msg_shown = False
6869

70+
# Seconds to wait for the telemetry endpoint before giving up on an event.
71+
TELEMETRY_REQUEST_TIMEOUT = 2
72+
73+
# Telemetry must never sit in the caller's critical path, so every event is sent
74+
# from a daemon thread. A slow or unreachable telemetry endpoint (for example from
75+
# inside a VPC with no route to it) can no longer stall the SDK call the customer
76+
# actually made, and because the threads are daemons a pending send cannot delay
77+
# interpreter shutdown either. No event is ever dropped: one thread is started per
78+
# event.
79+
#
80+
# Special case worth calling out: Feature Store ingestion is decorated at more than
81+
# one level (``ingest_dataframe`` and ``IngestionManagerPandas.run``), so a single
82+
# user call can emit several events. Sending them serially on the caller's thread is
83+
# what turned an ingest that the service finished in under a second into a
84+
# multi-minute wait, which is why the send is moved off that thread here rather than
85+
# only having its timeout tightened.
86+
6987
FEATURE_TO_CODE = {
7088
str(Feature.SDK_DEFAULTS): 11,
7189
str(Feature.LOCAL_MODE): 12,
@@ -422,6 +440,41 @@ def _send_telemetry_request(
422440
failure_reason: str = None,
423441
failure_type: str = None,
424442
extra_info: str = None,
443+
) -> threading.Thread:
444+
"""Schedule a telemetry event to be sent on a background daemon thread.
445+
446+
Every event is still sent; this only moves the send off the caller's thread so
447+
that telemetry never adds latency to the SDK call that triggered it.
448+
449+
Returns:
450+
threading.Thread: The thread doing the send. Callers generally ignore this;
451+
tests can join on it.
452+
"""
453+
454+
def _run():
455+
try:
456+
_send_telemetry_request_sync(
457+
status, feature_list, session, failure_reason, failure_type, extra_info
458+
)
459+
except Exception: # pylint: disable=W0703
460+
# Nothing can be raised out of a fire-and-forget thread: there is no
461+
# caller to catch it, and even the logging call inside
462+
# _send_telemetry_request_sync can fail once the interpreter starts
463+
# tearing down. Telemetry is best-effort, so drop the event silently.
464+
pass
465+
466+
thread = threading.Thread(target=_run, name="sagemaker-telemetry", daemon=True)
467+
thread.start()
468+
return thread
469+
470+
471+
def _send_telemetry_request_sync(
472+
status: int,
473+
feature_list: List[int],
474+
session: Session,
475+
failure_reason: str = None,
476+
failure_type: str = None,
477+
extra_info: str = None,
425478
) -> None:
426479
"""Make GET request to an empty object in S3 bucket"""
427480
try:
@@ -449,7 +502,7 @@ def _send_telemetry_request(
449502
)
450503
# Send the telemetry request
451504
logger.debug("Sending telemetry request to [%s]", url)
452-
_requests_helper(url, 2)
505+
_requests_helper(url, TELEMETRY_REQUEST_TIMEOUT)
453506
logger.debug("SageMaker Python SDK telemetry successfully emitted.")
454507
except Exception: # pylint: disable=W0703
455508
logger.debug("SageMaker Python SDK telemetry not emitted!")
@@ -482,12 +535,17 @@ def _construct_url(
482535

483536

484537
def _requests_helper(url, timeout):
485-
"""Make a GET request to the given URL"""
538+
"""Make a GET request to the given URL
539+
540+
``timeout`` must be passed by keyword. ``requests.get`` takes ``params`` as
541+
its second positional argument, so passing it positionally would append the
542+
value to the query string and leave the request with no timeout at all.
543+
"""
486544
response = None
487545
try:
488-
response = requests.get(url, timeout)
546+
response = requests.get(url, timeout=timeout)
489547
except requests.exceptions.RequestException as e:
490-
logger.exception("Request exception: %s", str(e))
548+
logger.debug("Request exception: %s", str(e))
491549
return response
492550

493551

@@ -511,9 +569,24 @@ def _get_region_or_default(session):
511569

512570

513571
def _get_default_sagemaker_session():
514-
"""Return the default sagemaker session"""
572+
"""Return the default sagemaker session
573+
574+
The region is resolved by boto3 from the caller's own environment
575+
(``AWS_REGION``, ``AWS_DEFAULT_REGION``, or the active profile in
576+
``~/.aws/config``). ``DEFAULT_AWS_REGION`` is only used as a last resort,
577+
because ``Session`` requires a region. Hardcoding the default meant that
578+
callers with no session of their own (module-level functions such as
579+
``ingest_dataframe``) had their telemetry pointed at a region they may have
580+
no network route to.
581+
"""
515582

516-
boto_session = boto3.Session(region_name=DEFAULT_AWS_REGION)
583+
boto_session = boto3.Session()
584+
if not boto_session.region_name:
585+
logger.debug(
586+
"No region resolved from the local AWS configuration. Falling back to %s.",
587+
DEFAULT_AWS_REGION,
588+
)
589+
boto_session = boto3.Session(region_name=DEFAULT_AWS_REGION)
517590
sagemaker_session = Session(boto_session=boto_session)
518591

519592
return sagemaker_session

0 commit comments

Comments
 (0)