Repository navigation
fix(server): keep terminal task states final across replicas - #1391
Draft
mykytanetipa wants to merge 1 commit into
Draft
mykytanetipa wants to merge 1 commit into
mykytanetipa wants to merge 1 commit into
Conversation
When another replica cancels a task (or otherwise finishes it) while the executing replica's agent is still running, the executing replica's next save conflicts and it reloads the terminal state. Events its agent had already enqueued were still applied afterwards: the reload refreshed the version, so the next save passed the compare-and-swap and overwrote the terminal state (e.g. CANCELED became COMPLETED). - VersionedDatabaseTaskStore refuses any write on top of a terminal task, not only CANCELED writes. The VersionedTaskStore contract says so. - The consumer stops applying a stale executor's events once a reload shows another writer finished the task. The in-memory test store now copies tasks in and out like a database and follows the same rule; it can also model a store without the rule, to check the consumer side on its own.
🧪 Code Coverage (vs
|
| Base | PR | Delta | |
|---|---|---|---|
| src/a2a/server/agent_execution/active_task.py | 95.81% | 95.92% | 🟢 +0.11% |
| src/a2a/server/cluster/database_task_store.py | 99.30% | 99.32% | 🟢 +0.02% |
| Total | 93.16% | 93.18% | 🟢 +0.01% |
Generated by coverage-comment.yml
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In cluster mode, a task that another replica finished could be moved out of its terminal state by the replica still running its agent. Found while working on #1324.
Example:
sequenceDiagram participant C as Client participant B as Replica B participant S as Shared store participant A as Replica A (runs agent) Note over A: agent has queued artifact + COMPLETED C->>B: CancelTask(T) B->>S: save CANCELED (v3) A->>S: save artifact (expects v2) S-->>A: conflict A->>S: reload T, gets CANCELED (v3) A->>S: save artifact + COMPLETED (expects v3) S-->>A: accepted Note over S: T is COMPLETED, the client was told CANCELEDFix: a terminal state is now final. VersionedDatabaseTaskStore refuses any write on top of a terminal task, not only CANCELED writes, and the VersionedTaskStore contract says so. On the consumer side, once a reload shows that another writer finished the task, A drops the remaining events from its executor instead of applying them, which also covers a custom VersionedTaskStore that does not have the check. In the example the last save is refused and T stays CANCELED.
Tested: multi-replica tests for a remote cancel followed by a stale burst of events, with and without the store check; full suite and Postgres/MySQL tests pass.