Skip to content

fix: sign tracked-run uploads for as long as the PUT may take - #1537

Open
EngHabu wants to merge 1 commit into
mainfrom
fix/sentry-tracked-upload-expiry
Open

fix: sign tracked-run uploads for as long as the PUT may take#1537
EngHabu wants to merge 1 commit into
mainfrom
fix/sentry-tracked-upload-expiry

Conversation

@EngHabu

@EngHabu EngHabu commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

What

upload_tracked_run_artifact minted its pre-signed upload URL with a hardcoded 60s lifetime:

_UPLOAD_EXPIRES_IN = timedelta(seconds=60)
...
req.expires_in.FromTimedelta(_UPLOAD_EXPIRES_IN)

…and then handed that URL to flyte.remote._data._put_signed_url_with_retry — the same helper file uploads use, which allows a full FLYTE_UPLOAD_TIMEOUT (600s by default) per attempt, and then retries with backoff on top of that.

So a single PUT attempt could run ten times longer than the signature authorizing it. When it does, the object store answers with a plain authorization failure and the upload dies as RuntimeSystemError: Upload failed for inputs.pb.

Why this call site

This is exactly the defect #1363 fixed for file uploads (FLYTE-SDK-5F, 4B, 6C, 7G, 7H, 7K — all now resolved). That PR derived the lifetime from the upload timeout plus a margin, clamped to the control plane's upload.maxExpiresIn:

_UPLOAD_EXPIRES_IN_SECONDS = float(os.environ.get(
    "FLYTE_UPLOAD_EXPIRES_IN",
    min(_UPLOAD_TIMEOUT_SECONDS + _UPLOAD_EXPIRES_MARGIN_SECONDS, _UPLOAD_EXPIRES_CAP_SECONDS),
))

…but only wired it into flyte/remote/_data.py. _persistence/_remote_upload.py shares the PUT helper while keeping its own literal, so flyte run --tracked still signs inputs.pb / outputs.pb / report.html for 60s.

Reusing the derived constant also:

  • brings the FLYTE_UPLOAD_EXPIRES_IN escape hatch to this path (it silently did nothing here), and
  • makes the expiry error message truthful — it quotes the requested lifetime, and on this path it was quoting _data's 900s while the URL actually lived 60s.

The import is function-local, matching this module's existing cold-path import discipline (it already imports get_extra_headers_for_protocol the same way).

Not a Sentry issue — found by auditing #1363's call sites

Worth stating plainly so the diff isn't read as fixing reported events: this path cannot appear in Sentry. Tracked-run reporting is best-effort — failures are logged and swallowed unless --tracked-strict is set, and they then surface as TrackedRunReportingError, a plain RuntimeError that is not a RuntimeSystemError and so is never captured by cli/_run.py.

It was found by grepping the other callers of the function #1363 touched, rather than from the issue corpus. The user-visible effect is silently degraded tracked-run reporting (missing inputs/outputs/report artifacts), not a crash report.

Test

Both assertions fail on clean main with the exact production numbers (assert 60.0 == 900.0) and pass here.

The regression test pins the derived constant rather than a literal, and separately asserts the signature outlives one PUT attempt:

assert signed_for == data_mod._UPLOAD_EXPIRES_IN.total_seconds()
assert signed_for >= data_mod._UPLOAD_TIMEOUT_SECONDS

That second assertion is the point: it is what stops two paths sharing one PUT helper from drifting apart again. The pre-existing == 60 assertion in test_upload_tracked_run_artifact_success is updated for the same reason.

tests/persistence/ — 115 passed. ruff format + check, mypy, ty, and check_docstring_style.py all clean.

`upload_tracked_run_artifact` minted its pre-signed URL with a hardcoded
60s lifetime, then handed it to `_put_signed_url_with_retry` -- the same
helper file uploads use, which allows a full `FLYTE_UPLOAD_TIMEOUT`
(600s by default) per attempt and then retries with backoff on top. A
single attempt could therefore run ten times longer than the signature
it was using, and the upload dies as `RuntimeSystemError: Upload failed`
with the object store reporting a plain authorization failure.

This is the defect #1363 fixed for file uploads (FLYTE-SDK-5F, 4B, 6C,
7G, 7H, 7K); that PR derived the lifetime from the upload timeout plus a
margin, but only wired it into `flyte.remote._data`. This call site was
missed, so `flyte run --tracked` still signs inputs.pb / outputs.pb /
report.html for 60s.

Reuse the derived constant instead of a second literal. That also brings
the `FLYTE_UPLOAD_EXPIRES_IN` escape hatch and the control-plane
`upload.maxExpiresIn` clamp to this path, and makes the expiry error
message -- which quotes the requested lifetime -- truthful here.

Note this path is not visible in Sentry: tracked-run reporting is
best-effort, so the failure is logged and swallowed unless
`--tracked-strict` is set, and it then surfaces as
`TrackedRunReportingError`, which is not a `RuntimeSystemError` and so is
never captured. It was found by auditing the call sites of the function
#1363 touched, not from an issue.

The regression test pins the derived constant rather than a literal, and
asserts the signature outlives one PUT attempt -- what keeps two paths
sharing one PUT helper from drifting apart again.

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@EngHabu EngHabu added the sentry-fix Fix for an issue surfaced by Sentry label Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sentry-fix Fix for an issue surfaced by Sentry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant