Commit 3f864b9
feat: populate RunSpec.related_to on rerun and in-container flyte.run (#1279)
## Why
The backend added `RunSpec.related_to` (field 15,
`flyteidl2.common.RunIdentifier`): an optional, set-at-creation pointer
to the single parent run this run was derived from (re-run, recover,
clone, or explicit provenance). This PR makes the SDK populate it
automatically at CreateRun time.
## What
Two implicit provenance sources, both resolved by a new
`_Runner._resolve_related_to` helper and stamped in `_apply_overrides`:
1. **Rerun** — every rerun surface (`flyte rerun`, `flyte run
--rerun-from`, `flyte.rerun`, `with_runcontext().rerun()`) converges on
`_Runner.rerun`; the new run points at the run being rerun. The source
id comes from the fetched `action_details.pb2.id.run`, with a fallback
to the `run_name` argument when the fetched id is empty.
2. **In-container `flyte.run`** — when a run is created from inside a
running remote task (`TaskContext.is_in_cluster()`), the new run points
at the invoking run (`flyte.ctx().action`). Local and hybrid executions
are excluded; in a rerun the explicit source wins over the ambient task
context.
### Gating & scope rules
- **Descriptor-gated, silent skip**: the published flyteidl2 (2.0.26)
doesn't ship the field yet. Unlike `recover` — explicitly
user-requested, so its absence raises — implicit provenance silently
skips with a debug log, so reruns keep working on older builds.
- **Same-scope, complete-id only**: per the proto contract `related_to`
is scoped to the new run's org/project/domain, and all four
`RunIdentifier` fields are server-required (buf.validate `min_len=1`).
The resolver stamps only when the source scope (empty fields inherited
from the init config) equals the new run's target scope and the id is
complete — a provenance pointer must never fail run creation.
- **Rerun base-copy path**: the field is unconditionally cleared before
stamping, so a rerun-of-rerun overwrites the inherited grandparent
pointer, and a cross-scope rerun (e.g.
`with_runcontext(project="other").rerun(...)`) clears it rather than
propagating a wrong-scope pointer.
### Tests
- New `tests/flyte/test_related_to.py`: resolver rules (ctx detection,
mode gating, explicit-source precedence, scope mismatch, empty-org,
cfg-scope leniency) — runs on any flyteidl2 build.
- `test_run_runspec_chars.py` / `test_rerun.py`: wire-level tests gated
on `RunSpec.DESCRIPTOR.fields_by_name` — they skip on 2.0.26 and were
verified to pass against the generated protos that ship the field
(stamping, overwrite, clear, name fallback). Two inverse-gated tests pin
the silent no-op on old builds and flag when the gate becomes obsolete.
- `_init_for_testing` gains an `org=` parameter (stamping requires a
non-empty org); existing tests init without org and are unaffected — the
RunSpec snapshot oracle stays byte-for-byte identical.
### Notes for reviewers
- **Backend caveat**: if the server stamps `related_to` itself, a rerun
performed with an *old* SDK fetches the base spec carrying field 15 as
an unknown field, which `CopyFrom` preserves and `ClearField` cannot
touch — the backend should treat client-sent `related_to` at create time
as advisory.
- Follow-ups deliberately out of scope: explicit
`with_runcontext(related_to=...)`, provenance for trigger-created runs.
---------
Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com>
Co-authored-by: Ketan Umare <kumare3@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>1 parent 9045a5c commit 3f864b9
36 files changed
Lines changed: 399 additions & 93 deletions
File tree
- plugins
- anthropic
- bigquery
- codegen
- dask
- databricks
- echo
- gemini
- hitl
- huggingface
- hydra
- jsonl
- mlflow
- omegaconf
- openai
- pandera
- papermill
- polars
- pytorch
- ray
- redis
- sglang
- snowflake
- spark
- vllm
- wandb
- rs_controller
- src/flyte
- tests/flyte
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments