fix: sign tracked-run uploads for as long as the PUT may take - #1537
Open
EngHabu wants to merge 1 commit into
Open
fix: sign tracked-run uploads for as long as the PUT may take#1537EngHabu wants to merge 1 commit into
EngHabu wants to merge 1 commit into
Conversation
`upload_tracked_run_artifact` minted its pre-signed URL with a hardcoded 60s lifetime, then handed it to `_put_signed_url_with_retry` -- the same helper file uploads use, which allows a full `FLYTE_UPLOAD_TIMEOUT` (600s by default) per attempt and then retries with backoff on top. A single attempt could therefore run ten times longer than the signature it was using, and the upload dies as `RuntimeSystemError: Upload failed` with the object store reporting a plain authorization failure. This is the defect #1363 fixed for file uploads (FLYTE-SDK-5F, 4B, 6C, 7G, 7H, 7K); that PR derived the lifetime from the upload timeout plus a margin, but only wired it into `flyte.remote._data`. This call site was missed, so `flyte run --tracked` still signs inputs.pb / outputs.pb / report.html for 60s. Reuse the derived constant instead of a second literal. That also brings the `FLYTE_UPLOAD_EXPIRES_IN` escape hatch and the control-plane `upload.maxExpiresIn` clamp to this path, and makes the expiry error message -- which quotes the requested lifetime -- truthful here. Note this path is not visible in Sentry: tracked-run reporting is best-effort, so the failure is logged and swallowed unless `--tracked-strict` is set, and it then surfaces as `TrackedRunReportingError`, which is not a `RuntimeSystemError` and so is never captured. It was found by auditing the call sites of the function #1363 touched, not from an issue. The regression test pins the derived constant rather than a literal, and asserts the signature outlives one PUT attempt -- what keeps two paths sharing one PUT helper from drifting apart again. Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
upload_tracked_run_artifactminted its pre-signed upload URL with a hardcoded 60s lifetime:…and then handed that URL to
flyte.remote._data._put_signed_url_with_retry— the same helper file uploads use, which allows a fullFLYTE_UPLOAD_TIMEOUT(600s by default) per attempt, and then retries with backoff on top of that.So a single PUT attempt could run ten times longer than the signature authorizing it. When it does, the object store answers with a plain authorization failure and the upload dies as
RuntimeSystemError: Upload failed for inputs.pb.Why this call site
This is exactly the defect #1363 fixed for file uploads (FLYTE-SDK-5F, 4B, 6C, 7G, 7H, 7K — all now resolved). That PR derived the lifetime from the upload timeout plus a margin, clamped to the control plane's
upload.maxExpiresIn:…but only wired it into
flyte/remote/_data.py._persistence/_remote_upload.pyshares the PUT helper while keeping its own literal, soflyte run --trackedstill signsinputs.pb/outputs.pb/report.htmlfor 60s.Reusing the derived constant also:
FLYTE_UPLOAD_EXPIRES_INescape hatch to this path (it silently did nothing here), and_data's 900s while the URL actually lived 60s.The import is function-local, matching this module's existing cold-path import discipline (it already imports
get_extra_headers_for_protocolthe same way).Not a Sentry issue — found by auditing #1363's call sites
Worth stating plainly so the diff isn't read as fixing reported events: this path cannot appear in Sentry. Tracked-run reporting is best-effort — failures are logged and swallowed unless
--tracked-strictis set, and they then surface asTrackedRunReportingError, a plainRuntimeErrorthat is not aRuntimeSystemErrorand so is never captured bycli/_run.py.It was found by grepping the other callers of the function #1363 touched, rather than from the issue corpus. The user-visible effect is silently degraded tracked-run reporting (missing inputs/outputs/report artifacts), not a crash report.
Test
Both assertions fail on clean
mainwith the exact production numbers (assert 60.0 == 900.0) and pass here.The regression test pins the derived constant rather than a literal, and separately asserts the signature outlives one PUT attempt:
That second assertion is the point: it is what stops two paths sharing one PUT helper from drifting apart again. The pre-existing
== 60assertion intest_upload_tracked_run_artifact_successis updated for the same reason.tests/persistence/— 115 passed. ruff format + check, mypy,ty, andcheck_docstring_style.pyall clean.