Skip to content

docsy(v2): document run recovery (--recover / --recover-from) - #1439

Closed
ppiegaze wants to merge 1 commit into
mainfrom
docsy/v2-recover-runs
Closed

docsy(v2): document run recovery (--recover / --recover-from)#1439
ppiegaze wants to merge 1 commit into
mainfrom
docsy/v2-recover-runs

Conversation

@ppiegaze

Copy link
Copy Markdown
Collaborator

Documents run recovery (--recover / --recover-from), the shipped Phase 1 slice of the Recovery Run project. Closes the gap DOC-1211 tracks.

Why now

Readiness verified off the code artifact, not status fields:

Layer Evidence
SDK/CLI --recover / --recover-from landed in flyte-sdk #1237 (4955aa4d, 2026-08-03), first released in v2.5.17
Action identity stable cache-key action names, flyte-sdk #1373 (22f43e6e), released in v2.5.19; this is what fixed "--recover-from with a code change does not work"
IDL flyteidl2 2.0.31+ carries RunSpec.relation, RunSpec.recover, RELATION_TYPE_RECOVER, so the SDK's NotImplementedError gate is open
Backend actions-service recovery logic cloud#17063 + Run Service validation cloud#17064, merged 2026-07-17, no feature flag by design
Phase ACTION_PHASE_RECOVERED across all systems, cloud#16862, merged 2026-07-16

Latest release at time of writing: flyte v2.6.0.

What changed

New page user-guide/tasks/task-deployment/recover-runs.md (-flyte +union, weight 15):

  • recovery vs. rerunning, with the workload framing (fine-tuning, RL, computer vision) requested in #product-v2
  • flyte rerun --recover and flyte run --recover-from
  • what is reused vs. re-executed, from the real identity model: hash(parent + inputs + task identity + call seq) where task identity is task name + interface + per-task code version, deliberately excluding the image, code bundle version, resources, and env vars
  • the trust-model caveat inherited from caching (a helper function or library change does not invalidate reuse)
  • --force-rerun-action / recover_force_rerun_actions as the escape hatch
  • the programmatic form, and the fact that bare flyte.rerun() never recovers
  • the Recovered action phase and recovered_from provenance
  • requirements and limitations: source run must exist, be terminal, be in the same project and domain; remote-only; --allow-missing-outputs; the SDK 2.6.0 floor on the source run

Sibling updates:

  • rerun-runs.md: union-gated pointer to the new page, a note under the rerun/--rerun-from comparison table, a Related entry, and the pre-existing "such as recovering a failed run" clause now links somewhere
  • run-context.md: a union-gated Recovery parameter table (recover, recover_force_rerun_actions, allow_missing_source_outputs). This closes the recover line item of DOC-1267
  • run-command-options.md: --rerun-from (both variants, a pre-existing gap that the rerun page already cross-referenced) plus union-gated --recover-from and --force-rerun-action
  • debug-runs.md: weight 15 → 16 to make room

Variant scope

Union only. Recovery in the OSS backend is still a design-doc PR (flyteorg/flyte#7825, open, unmerged), so nothing recover-related renders in the flyte variant. Verified in the built output for both variants.

Deliberately not documented

  • flyte.replay() / with_replaycontext() (PRD Phase 2). Named in ENG26-830 but absent from the SDK's exports on origin/main. Not shipped, so not documented.
  • The console Recover control. ENG26-1055 (a "Recover from source run" checkbox on the rerun drawer) merged only on 2026-08-13 (cloud#17643, #17658). Rollout to a prod tenant is unverified and no screenshots have been captured to the house format, so the UI path is a follow-up rather than unverified prose.

Verification

  • make variant VARIANT=union and VARIANT=flyte: clean builds, 1403 / 1289 pages
  • make check-links: all internal links valid, both variants
  • Variant gating confirmed in dist/: zero recover occurrences in the flyte build of rerun-runs, run-command-options, run-context

— docsy · automated docs agent · DOC-1211

Add a "Recover a failed run" page covering the shipped CLI and SDK surface:
flyte rerun --recover, flyte run --recover-from, --force-rerun-action,
--allow-missing-outputs, and with_runcontext(recover=...,
recover_force_rerun_actions=..., allow_missing_source_outputs=...).

Also documents the action-identity model that decides what is reused vs.
re-executed (cache-key naming from flyte-sdk#1373), the requirements the Run
Service validates (source run must exist, be terminal, be in the same
project/domain), and the Recovered action phase.

Gated to the union variant: recovery in the OSS backend is still a design
proposal (flyteorg/flyte#7825), not implemented.

Sibling updates: cross-links + a rerun-vs-recover comparison on rerun-runs,
a Recovery parameter table on run-context (part of DOC-1267), and
--rerun-from / --recover-from / --force-rerun-action on run-command-options.

Co-Authored-By: docsy <docsy@union.ai>
Signed-off-by: Peeter Piegaze <1153481+ppiegaze@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

GHA build & deploy preview

Built by .github/workflows/build-pr.yml and deployed to the docs CF Pages project by .github/workflows/deploy-pr-preview.yml.

Branch alias https://pr-1439-docsy-v2-recover-run.docs-dog.pages.dev
This commit https://7bc95474.docs-dog.pages.dev
Commit SHA f496c431daaa95e8ea541d46c1250cfd97ab0551

Updated automatically on every push.

@ppiegaze

ppiegaze commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: superseded, and merging this would regress the docs.

The SDK surface moved six days after this was drafted. flyte-sdk#1452 (2026-08-20, first absent in v2.6.3) removed flyte run --recover-from, removed --rerun-from, and removed recover / recover_force_rerun_actions from with_runcontext(), making recovery remote-only via rerun(). This PR documents all three.

Meanwhile #1464 merged on 2026-08-21 and landed recover-runs.md and fork-runs.md on main against the current surface: flyte rerun --recover / --action-name / --force-rerun-action / --allow-missing-outputs, rerun input overrides, and flyte fork (a separate command in flyteplugins-union, not a flag). That coverage is complete, and run-context.md on main is clean of the removed parameters.

No residual gap for this branch to fill. DOC-1211 moves to Done against #1464.


— docsy · automated docs agent · DOC-1211 · run report

@ppiegaze ppiegaze closed this Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant