Repository navigation
Commit e5feb08
authored
fix(execution): end runs quietly when their workspace is deleted mid-run (#9144)
* fix(execution): end runs quietly when their workspace is deleted mid-run
Deleting an account hard-deletes its solely-owned workspaces, and the cascade
takes each in-flight run's workflow and execution log with it. A run caught in
that window kept executing, then failed every terminal write: completion threw
"Workflow log not found", the cost-only fallback failed the same way, run
counts threw and logged twice, pause persistence logged errors, and the
admission reservation leaked until its TTL. A run whose log insert failed on
the deleted workspace kept executing blocks with no log row, so nothing could
record or stop it. Along the way a file upload failure surfaced raw Drizzle SQL
and bound parameters (user id, workspace id, storage key) in a block error.
- ExecutionScopeDeletedError (no cause) marks the outcome; classifyFailure
attributes it to the owner, so it logs at info.
- completeWorkflowExecution, when the log row is missing, checks the run's
workspace by primary key (inside the completion transaction when the row
vanished mid-completion) and throws the typed error before any redaction,
storage, or billing work. A missing row under a live workspace still throws
"Workflow log not found" as before.
- LoggingSession recognizes it in every completion path: info instead of
error, no cost-only fallback, admission slot released, later completions
and force-fails become no-ops, and getDeletedScope() reports it.
- A log start that fails on a foreign key checks whether the workspace or
workflow still exists; if not, safeStart returns false without the
fallback, releases the slot, and executeWorkflowCore stops before building
the executor. Any other start failure behaves exactly as before.
- Run counts skip a deleted scope, and a zero-row update is an info no-op.
- Pause persistence returns quietly for a deleted scope.
- User-visible error text never carries a query failure's SQL: the execution
upload wrapper keeps its cause and redacts its message, the block executor
redacts any query failure in the cause chain, and the workspace file,
file-output, video, image, TTS, ElevenLabs, LaTeX, and pause sites use a
shared getUserFacingErrorMessage.
* fix(execution): close review gaps in deleted-scope run ending
- Rename the integration suite to scope-deleted.integration.ts so its name
does not repeat its folder (check:file-names).
- Type the new execution-core tests through ExecutionSnapshot and
LoggingSession instead of `as any` (check:explicit-any).
- Define FOREIGN_KEY_VIOLATION once in database-query-error and import it in
the logger and the logging session.
- Cover the guards the first commit left untested: a minimal fallback start
that hits the deleted scope's foreign key, a cost-only fallback that is the
first to find the workspace deleted, and pause persistence skipping a run
whose deletion its paused completion found.
- Give the csv-workflow-execution live test's LoggingSession stub the new
getDeletedScope method.
Correction to the previous commit message: pause persistence is quiet only
when the paused completion itself found the workspace or workflow deleted. A
deletion that lands after the paused log was written still fails
persistPauseResult, which logs at error and force-fails the run, exactly as
before. The TSDoc now says so.
* test(logs): type the logged-message callbacks in the deleted-scope session tests
* fix(execution): keep wrapped causes and quiet usage logging for deleted workspaces
- The file-output wrapper in FileToolProcessor keeps the original failure as
its cause, so diagnostics and the tool boundary still see a database
failure underneath the redacted message.
- recordExecutionUsage logs a missing workflow at info, not error, when the
run's whole workspace is gone. The completion then reports the deleted run.
The lookup runs only on that missing-workflow branch, both branches record
nothing as before, and a failed lookup or a live workspace keeps the error.
* refactor(execution): simplify deleted-scope handling after audit
Fold the deleted-scope note and slot release into one session helper, reuse
the admission-slot release on the normal completion path, share an
ExecutionScope type, drop the unread executionId field and a redundant guard.1 parent 7561e0f commit e5feb08
30 files changed
Lines changed: 924 additions & 55 deletions
File tree
- apps/sim
- executor
- execution
- utils
- lib
- core/errors
- internal
- elevenlabs
- image
- latex
- tts
- video
- logs/execution
- mothership/agent-cli
- uploads/contexts
- execution
- workspace
- workflows
- executor
- tools
- packages/testing/src/mocks
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
504 | 504 | | |
505 | 505 | | |
506 | 506 | | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
507 | 542 | | |
508 | 543 | | |
509 | 544 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| |||
660 | 660 | | |
661 | 661 | | |
662 | 662 | | |
663 | | - | |
| 663 | + | |
| 664 | + | |
664 | 665 | | |
665 | 666 | | |
666 | 667 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
15 | 16 | | |
| 17 | + | |
16 | 18 | | |
17 | 19 | | |
18 | 20 | | |
| |||
194 | 196 | | |
195 | 197 | | |
196 | 198 | | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
197 | 219 | | |
198 | 220 | | |
199 | 221 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
3 | 2 | | |
4 | 3 | | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| |||
122 | 122 | | |
123 | 123 | | |
124 | 124 | | |
125 | | - | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
126 | 129 | | |
127 | 130 | | |
128 | 131 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
2 | 7 | | |
3 | 8 | | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
4 | 12 | | |
5 | 13 | | |
6 | 14 | | |
| |||
9 | 17 | | |
10 | 18 | | |
11 | 19 | | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
12 | 35 | | |
13 | 36 | | |
14 | 37 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
24 | 31 | | |
25 | 32 | | |
26 | 33 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
| |||
63 | 64 | | |
64 | 65 | | |
65 | 66 | | |
66 | | - | |
67 | | - | |
68 | | - | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
69 | 70 | | |
70 | 71 | | |
71 | 72 | | |
| |||
77 | 78 | | |
78 | 79 | | |
79 | 80 | | |
| 81 | + | |
| 82 | + | |
80 | 83 | | |
81 | 84 | | |
82 | 85 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
3 | 2 | | |
4 | 3 | | |
5 | 4 | | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| |||
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
73 | | - | |
| 73 | + | |
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
123 | 124 | | |
124 | 125 | | |
125 | 126 | | |
126 | | - | |
| 127 | + | |
127 | 128 | | |
128 | 129 | | |
129 | 130 | | |
| |||
0 commit comments