Skip to content

Commit 3f864b9

Browse files
kumare3claude
andauthored
feat: populate RunSpec.related_to on rerun and in-container flyte.run (#1279)
## Why The backend added `RunSpec.related_to` (field 15, `flyteidl2.common.RunIdentifier`): an optional, set-at-creation pointer to the single parent run this run was derived from (re-run, recover, clone, or explicit provenance). This PR makes the SDK populate it automatically at CreateRun time. ## What Two implicit provenance sources, both resolved by a new `_Runner._resolve_related_to` helper and stamped in `_apply_overrides`: 1. **Rerun** — every rerun surface (`flyte rerun`, `flyte run --rerun-from`, `flyte.rerun`, `with_runcontext().rerun()`) converges on `_Runner.rerun`; the new run points at the run being rerun. The source id comes from the fetched `action_details.pb2.id.run`, with a fallback to the `run_name` argument when the fetched id is empty. 2. **In-container `flyte.run`** — when a run is created from inside a running remote task (`TaskContext.is_in_cluster()`), the new run points at the invoking run (`flyte.ctx().action`). Local and hybrid executions are excluded; in a rerun the explicit source wins over the ambient task context. ### Gating & scope rules - **Descriptor-gated, silent skip**: the published flyteidl2 (2.0.26) doesn't ship the field yet. Unlike `recover` — explicitly user-requested, so its absence raises — implicit provenance silently skips with a debug log, so reruns keep working on older builds. - **Same-scope, complete-id only**: per the proto contract `related_to` is scoped to the new run's org/project/domain, and all four `RunIdentifier` fields are server-required (buf.validate `min_len=1`). The resolver stamps only when the source scope (empty fields inherited from the init config) equals the new run's target scope and the id is complete — a provenance pointer must never fail run creation. - **Rerun base-copy path**: the field is unconditionally cleared before stamping, so a rerun-of-rerun overwrites the inherited grandparent pointer, and a cross-scope rerun (e.g. `with_runcontext(project="other").rerun(...)`) clears it rather than propagating a wrong-scope pointer. ### Tests - New `tests/flyte/test_related_to.py`: resolver rules (ctx detection, mode gating, explicit-source precedence, scope mismatch, empty-org, cfg-scope leniency) — runs on any flyteidl2 build. - `test_run_runspec_chars.py` / `test_rerun.py`: wire-level tests gated on `RunSpec.DESCRIPTOR.fields_by_name` — they skip on 2.0.26 and were verified to pass against the generated protos that ship the field (stamping, overwrite, clear, name fallback). Two inverse-gated tests pin the silent no-op on old builds and flag when the gate becomes obsolete. - `_init_for_testing` gains an `org=` parameter (stamping requires a non-empty org); existing tests init without org and are unaffected — the RunSpec snapshot oracle stays byte-for-byte identical. ### Notes for reviewers - **Backend caveat**: if the server stamps `related_to` itself, a rerun performed with an *old* SDK fetches the base spec carrying field 15 as an unknown field, which `CopyFrom` preserves and `ClearField` cannot touch — the backend should treat client-sent `related_to` at create time as advisory. - Follow-ups deliberately out of scope: explicit `with_runcontext(related_to=...)`, provenance for trigger-created runs. --------- Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> Co-authored-by: Ketan Umare <kumare3@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 9045a5c commit 3f864b9

36 files changed

Lines changed: 399 additions & 93 deletions

File tree

plugins/anthropic/uv.lock

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

plugins/bigquery/uv.lock

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

plugins/codegen/uv.lock

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

plugins/dask/uv.lock

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

plugins/databricks/uv.lock

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

plugins/echo/uv.lock

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

plugins/gemini/uv.lock

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

plugins/hitl/uv.lock

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

plugins/huggingface/uv.lock

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

plugins/hydra/uv.lock

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)