Skip to content

fix(processing): respect session region when waiting for processing jobs - #6369

Merged
rsareddy0329 merged 1 commit into
aws:masterfrom
rsareddy0329:fix/processing-wait-region
Oct 5, 2026
Merged

rsareddy0329 merged 1 commit into
aws:masterfrom
rsareddy0329:fix/processing-wait-region

Conversation

@rsareddy0329

Copy link
Copy Markdown
Contributor

Issue #, if available: Relates to #5796

Description of changes:

Processor.run / FrameworkProcessor.run waited via self.latest_job.wait(). The
generated ProcessingJob.wait() → refresh() resolves a default, global SageMaker
client (Base.get_sagemaker_client()), not the region of the sagemaker_session that
created the job. With a non-default-region session, the wait/describe and log tailing hit
the wrong region and fail — the reported bug (#5796). (V2 is unaffected: it waits through
self.sagemaker_session.logs_for_processing_job / wait_for_processing_job.)

This routes the wait path through the session:

  • logs=True → logs_for_processing_job(sagemaker_session, ...), which already resolves
    the logs client from the session. It's also fixed to describe the job via
    sagemaker_session.sagemaker_client (region-aware) instead of
    ProcessingJob.get(...).refresh() — the chained .refresh() re-described through the
    global client (wrong region) and read PascalCase keys (ProcessingJobStatus) off a
    snake_case resource __dict__.
  • logs=False → new _wait_for_processing_job(sagemaker_session, job_name), which polls
    describe_processing_job through the session's region-aware client and raises on failure.

Testing:

  • test_run_with_wait (updated): wait=True, logs=False goes through _wait_for_processing_job
    with the processor's session (no longer latest_job.wait()).
  • test_run_with_wait_and_logs_uses_session: logs=True routes through logs_for_processing_job
    with the session.
  • test_wait_for_processing_job_uses_session_region / _raises_on_failure: the helper describes
    via sagemaker_session.sagemaker_client and raises on a Failed status.
  • Full test_processing.py (125 passed) and model_monitor suites (82 passed, uses
    logs_for_processing_job) green. flake8 clean on changed files.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Processor.run / FrameworkProcessor.run waited via self.latest_job.wait(),
whose generated refresh() resolves a default, global SageMaker client
(Base.get_sagemaker_client()) rather than the region of the sagemaker_session
that created the job. With a non-default-region session the wait/describe
and log tailing hit the wrong region and fail (issue aws#5796).

Route the wait path through the session instead:
- logs=True -> logs_for_processing_job(sagemaker_session, ...), which already
  resolves clients from sagemaker_session; also fix it to describe via
  sagemaker_session.sagemaker_client (region-aware) instead of
  ProcessingJob.get(...).refresh() (the chained refresh used the global client
  and read PascalCase keys off a snake_case resource __dict__).
- logs=False -> new _wait_for_processing_job(sagemaker_session, job_name),
  which polls describe_processing_job through the session's region-aware client
  and raises on failure.

Add unit tests covering both wait paths and _wait_for_processing_job.

Relates-to aws#5796
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Claude finished @rsareddy0329's task in 1m 39s —— View job


Code Review — fix(processing): respect session region when waiting for processing jobs

  • Read PR diff and changed files
  • Trace helpers (_wait_until, _check_job_status, logs_for_processing_job) and callers
  • Assess correctness, tests, and conventions
  • Post review

Summary

This is a solid, well-targeted fix. The root cause is correctly diagnosed: latest_job.wait() → refresh() resolves the global Base.get_sagemaker_client() rather than the region of the sagemaker_session that created the job, so a non-default-region session hits the wrong region (#5796). Routing the wait path through sagemaker_session.sagemaker_client is the right fix and matches how v2 behaved.

I also want to call out a second, real bug this incidentally fixes: the old logs_for_processing_job read description["ProcessingJobStatus"] off ProcessingJob.get(...).refresh().__dict__, which is a snake_case resource dict — that key would never exist, so the logs path would have raised KeyError. Switching to describe_processing_job (PascalCase service response) makes _get_initial_job_state/_check_job_status work correctly. Good catch.

The split of latest_job.wait(logs=logs) into logs=True → logs_for_processing_job(..., wait=True) and logs=False → _wait_for_processing_job(...) preserves the prior semantics (log tailing + terminal-status check vs. plain poll + status check), and _describe_if_terminal returning None until a terminal state correctly drives _wait_until.

No correctness or security concerns found. The changes are backward compatible (logs_for_processing_job signature unchanged; _wait_for_processing_job is new) and well covered by the four added/updated tests.

Minor suggestions (non-blocking nits)

  1. Docstring raise type is inaccurate. _wait_for_processing_job (and the pre-existing logs_for_processing_job) document Raises: ValueError, but _check_job_status actually raises exceptions.UnexpectedStatusException / exceptions.CapacityError (common_utils.py:2445). Worth correcting the new docstring to reflect the real exception type.

  2. Test assertion is broad. test_wait_for_processing_job_raises_on_failure uses pytest.raises(Exception), which would pass on any exception (including an unrelated KeyError). Asserting the specific type (UnexpectedStatusException) would make the test more meaningful.

  3. Duplicated block. The identical 11-line if logs: ... else: ... wait block now appears in both run methods (processing.py:363 and :941). This mirrors the pre-existing duplication, so it's fine to leave, but extracting a small self._wait_for_latest_job(wait, logs) helper would reduce drift risk if this logic changes again.

  4. Uncovered terminal state. "Stopped" is handled by _check_job_status (warns, doesn't raise) but isn't exercised by a test; optional to add.

Nice work — the fix is correct and the tests target the regressed behavior precisely.
· fix/processing-wait-region

@rsareddy0329
rsareddy0329 merged commit 193df0f into aws:master Oct 5, 2026
26 of 28 checks passed
AkshayShah03 added a commit to AkshayShah03/sagemaker-python-sdk that referenced this pull request Nov 9, 2026
…_for_transform_job

logs_for_transform_job described the job with
TransformJob.get(transform_job_name=..., session=sagemaker_session).__dict__.
TransformJob.get expects a boto3 Session, so passing the SageMaker Session
failed pydantic validation ('Input should be an instance of Session'), and the
resource's snake_case __dict__ does not have the PascalCase keys
(TransformResources, TransformJobStatus) that _logs_init and
_check_job_status read.

Describe the job through sagemaker_session.sagemaker_client, as
logs_for_processing_job does (aws#6369). aws#6509 already fixed the session used
by transform()/wait()/stop()/attach(); this covers the log tailing path.

This branch was previously deployed

1 inactive deployment
auto-approve — 8cfef215 Deployed Oct 1, 2026 by rsareddy0329 via wait-for-approval #1601
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants