[V2] flyte recover - #7825
Conversation
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
|
|
||
| ```go | ||
| // executor/api/v1/taskaction_types.go | ||
| type RecoveredFrom struct { |
There was a problem hiding this comment.
good yes write the context to task action and then executor can just circle back and write the recovered phase and propagate the phase to ui too
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
|
We should wait for the UI PR merge in first. Then we can merge this PR with new UI image |
pingsutw
left a comment
pingsutw
left a comment
There was a problem hiding this comment.
I tested recover with condition task, but it didn't work.
| @@ -0,0 +1,346 @@ | |||
| # Run Recovery — design | |||
There was a problem hiding this comment.
Could we move it to https://github.com/flyteorg/flyte/tree/main/docs/rfcs and add a date prefix
| // found = false rather than an error: recovery is fail-open, and a miss is an ordinary outcome. | ||
| message LookupActionResponse { | ||
| // Whether the run has an action of that name. | ||
| bool found = 1; |
There was a problem hiding this comment.
Should we return grpc CodeNotFound instead of adding another field here?
| if relation.GetRelationType() != common.RelationType_RELATION_TYPE_RECOVER || source == nil { | ||
| return nil | ||
| } |
There was a problem hiding this comment.
Could we check the relation type here?
Lines 1086 to 1090 in bf62df5
if it's not recover, we should not set the recover context in the crd
There was a problem hiding this comment.
I think we already did that check here in recoveryContextFromRunSpec, so the recoverContext will be nil if it is not a recover action.
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
3 participants
Tracking issue
Why are the changes needed?
Introduce recover into OSS. When executing
flyte rerun <run_name> recover, the backend will look up if the child action executions result executed in prior run execution, and skip pod creation if the action result is found.What changes were proposed in this pull request?
Backend changes, design concept is detailed in the design doc.
How was this patch tested?
Execute
python examples/basics/devbox_one.py, then runflyte rerun <run_name> recover.We can also force rerun an action with
flyte rerun r7wstb4fdhsfl8v9nh2n --recover --force-rerun-action 7qvx2tqthogpce3w0cs5754mo. The action pod will be createdLabels
Please add one or more of the following labels to categorize your PR:
This is important to improve the readability of release notes.
Setup process
Screenshots
Check all the applicable boxes
Related PRs
Stack
If you do use
git townto manage PR Stacks, the stack relevant to this PRwill show below. Otherwise, you can ignore this section.
Docs link