Repository navigation
fix(processing): respect session region when waiting for processing jobs - #6369
Conversation
Processor.run / FrameworkProcessor.run waited via self.latest_job.wait(), whose generated refresh() resolves a default, global SageMaker client (Base.get_sagemaker_client()) rather than the region of the sagemaker_session that created the job. With a non-default-region session the wait/describe and log tailing hit the wrong region and fail (issue aws#5796). Route the wait path through the session instead: - logs=True -> logs_for_processing_job(sagemaker_session, ...), which already resolves clients from sagemaker_session; also fix it to describe via sagemaker_session.sagemaker_client (region-aware) instead of ProcessingJob.get(...).refresh() (the chained refresh used the global client and read PascalCase keys off a snake_case resource __dict__). - logs=False -> new _wait_for_processing_job(sagemaker_session, job_name), which polls describe_processing_job through the session's region-aware client and raises on failure. Add unit tests covering both wait paths and _wait_for_processing_job. Relates-to aws#5796
|
Claude finished @rsareddy0329's task in 1m 39s —— View job Code Review — fix(processing): respect session region when waiting for processing jobs
SummaryThis is a solid, well-targeted fix. The root cause is correctly diagnosed: I also want to call out a second, real bug this incidentally fixes: the old The split of No correctness or security concerns found. The changes are backward compatible ( Minor suggestions (non-blocking nits)
Nice work — the fix is correct and the tests target the regressed behavior precisely. |
…_for_transform_job
logs_for_transform_job described the job with
TransformJob.get(transform_job_name=..., session=sagemaker_session).__dict__.
TransformJob.get expects a boto3 Session, so passing the SageMaker Session
failed pydantic validation ('Input should be an instance of Session'), and the
resource's snake_case __dict__ does not have the PascalCase keys
(TransformResources, TransformJobStatus) that _logs_init and
_check_job_status read.
Describe the job through sagemaker_session.sagemaker_client, as
logs_for_processing_job does (aws#6369). aws#6509 already fixed the session used
by transform()/wait()/stop()/attach(); this covers the log tailing path.
Issue #, if available: Relates to #5796
Description of changes:
Processor.run/FrameworkProcessor.runwaited viaself.latest_job.wait(). Thegenerated
ProcessingJob.wait()→refresh()resolves a default, global SageMakerclient (
Base.get_sagemaker_client()), not the region of thesagemaker_sessionthatcreated the job. With a non-default-region session, the wait/describe and log tailing hit
the wrong region and fail — the reported bug (#5796). (V2 is unaffected: it waits through
self.sagemaker_session.logs_for_processing_job/wait_for_processing_job.)This routes the wait path through the session:
logs=True→logs_for_processing_job(sagemaker_session, ...), which already resolvesthe logs client from the session. It's also fixed to describe the job via
sagemaker_session.sagemaker_client(region-aware) instead ofProcessingJob.get(...).refresh()— the chained.refresh()re-described through theglobal client (wrong region) and read PascalCase keys (
ProcessingJobStatus) off asnake_case resource
__dict__.logs=False→ new_wait_for_processing_job(sagemaker_session, job_name), which pollsdescribe_processing_jobthrough the session's region-aware client and raises on failure.Testing:
test_run_with_wait(updated):wait=True, logs=Falsegoes through_wait_for_processing_jobwith the processor's session (no longer
latest_job.wait()).test_run_with_wait_and_logs_uses_session:logs=Trueroutes throughlogs_for_processing_jobwith the session.
test_wait_for_processing_job_uses_session_region/_raises_on_failure: the helper describesvia
sagemaker_session.sagemaker_clientand raises on a Failed status.test_processing.py(125 passed) andmodel_monitorsuites (82 passed, useslogs_for_processing_job) green.flake8clean on changed files.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.