Skip to content

Commit 45bff7a

Browse files
mchwangclaude
andcommitted
Never reject a merge job's run, log quoted, and test the restart that settles an unended job (F6b)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 0eb8a21 commit 45bff7a

3 files changed

Lines changed: 35 additions & 7 deletions

File tree

‎docs/implementation/merge-handoff.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ A rebase moves the pull request's base. The guarded merge would then scan only t
6363
| Cancel while waiting | the wait stops at once; the task closes; the job ends `cancelled` |
6464
| Cancel during the guarded merge | before GitHub is asked, the merge stops and the job ends `cancelled`; once a merge attempt is submitting, a cancel is refused, as before |
6565
| Shutdown | in shutdown's first, synchronous step, before requests drain, the job is told to stop: a wait ends at once, and nothing after it asks GitHub to merge. A preparation in progress stops only when preparation does, after the drain, as for a `prepare-merge` click; the job then stops without merging. The job is awaited before the store closes. While preparing or waiting, it stays `running` on disk. A guarded merge in progress stops as a click's does: if GitHub was not asked yet, the job stays `running`; if it was, the job ends as the merge attempt records it: `blocked` if GitHub refused (`failed` if the head changed, or the review changed while GitHub was asked), otherwise `submitted`, and the attempt reconciles as before |
66-
| Ending the job fails | if the write that ends a job fails (a store error, not shutdown), the server logs "Could not end merge job …; it shows as running until the server restarts". The job stays `running` on disk: the page keeps following it, and a new click is refused while it runs. The next start settles it as below |
66+
| Ending the job fails | if the write that ends a job fails, or reading the task after the job failed does, the server logs "Could not end merge job …; it shows as running until the server restarts" with the store's message. The job stays `running` on disk: the page keeps following it, and a new click is refused while it runs. The next start settles it as below |
6767
| Restart | a `running` job that had not asked GitHub to merge becomes `failed` ("The server restarted before the merge finished"). One that had (its merge action ID is saved before the guarded merge starts) ends as its merge attempt records it: `merged`, `queued`, `submitted`, or `blocked` after a refusal, which also moves the task to approved, merge blocked (`failed`, task unchanged, when the head changed or the review changed while GitHub was asked). A `prepare-merge` action and a push marker are settled as before |
6868

6969
## Storage

‎runner/merge-job.ts‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -96,10 +96,13 @@ export class MergeJobCoordinator {
9696
const abort = new AbortController(); this.#abort = abort;
9797
const run = Promise.resolve().then(() => this.#run(job, abort.signal)).catch(error => {
9898
// Anything unexpected ends the job, unless shutdown stopped it (startup then settles it) or the task was cancelled
99-
// (a cancel during one of the job's own review loads, which await a history worker since #170).
100-
if (abort.signal.aborted && this.#closing) return;
101-
if ((abort.signal.reason as { code?: string } | undefined)?.code === 'CANCELLED' || this.#cancelled()) return this.#end(job, abort.signal);
102-
this.#finish(job.id, 'failed', error instanceof Error ? error.message : String(error));
99+
// (a cancel during one of the job's own review loads, which await a history worker since #170). Never rejects: if
100+
// even reading the task fails here, the job stays running for startup, as when ending it fails.
101+
try {
102+
if (abort.signal.aborted && this.#closing) return;
103+
if ((abort.signal.reason as { code?: string } | undefined)?.code === 'CANCELLED' || this.#cancelled()) return this.#end(job, abort.signal);
104+
this.#finish(job.id, 'failed', error instanceof Error ? error.message : String(error));
105+
} catch (unended) { this.#unended(job.id, unended); }
103106
}).finally(() => { if (this.#active === run) { this.#active = null; this.#abort = null; } });
104107
this.#active = run;
105108
return mergeJobView(job);
@@ -144,7 +147,7 @@ export class MergeJobCoordinator {
144147

145148
/** Startup settles a job that could not be ended now; until then it shows as running and a new click is refused. */
146149
#unended(id: string, error: unknown): void {
147-
if (!this.#closing) console.error(`Could not end merge job ${id}; it shows as running until the server restarts: ${(error as Error)?.message ?? String(error)}`);
150+
console.error(`Could not end merge job ${id}; it shows as running until the server restarts: ${JSON.stringify((error as Error)?.message ?? String(error))}`);
148151
}
149152

150153
async #run(job: MergeJob, signal: AbortSignal): Promise<void> {

‎test/merge-job.test.ts‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -344,6 +344,7 @@ it.each([false, true])('ends submitted, not merged or queued, when GitHub accept
344344
it.each(['finish', 'block'] as const)('reports a job it could not end (%s), which stays running and refuses a new click until startup settles it', async kind => {
345345
const f = setup(kind === 'block' ? { checks: () => [{ context: 'build', appId: null, state: 'failure' }] } : {});
346346
const fail = vi.fn(() => { throw new Error('disk full'); });
347+
const { blockMergeJob, finishMergeJob } = f.store;
347348
// The finish case ends `submitted` (its attempt's outcome was not saved either), so the task stays in review and a new
348349
// click reaches the running-job check.
349350
if (kind === 'block') f.store.blockMergeJob = fail;
@@ -353,12 +354,36 @@ it.each(['finish', 'block'] as const)('reports a job it could not end (%s), whic
353354
const started = await f.coordinator.start('review-token', randomUUID());
354355
await f.coordinator.current;
355356
expect(fail).toHaveBeenCalledTimes(1);
356-
expect(logged).toHaveBeenCalledWith(expect.stringMatching(new RegExp(`Could not end merge job ${started.id}; it shows as running until the server restarts: disk full`)));
357+
expect(logged).toHaveBeenCalledWith(`Could not end merge job ${started.id}; it shows as running until the server restarts: "disk full"`);
357358
expect(f.store.getMergeJob(identity, started.id)).toMatchObject({ state: 'running' });
358359
await expect(f.coordinator.start('review-token', randomUUID())).rejects.toThrow(/already running/);
360+
// The next start settles it (runner/production.ts), once the store writes again.
361+
Object.assign(f.store, { blockMergeJob, finishMergeJob });
362+
f.store.settleInterruptedMergeJobs(identity);
363+
expect(f.store.getMergeJob(identity, started.id)).toMatchObject({ state: kind === 'block' ? 'failed' : 'submitted' });
359364
} finally { logged.mockRestore(); }
360365
});
361366

367+
it('keeps a job running and reports it when even reading the task fails after the job failed', async () => {
368+
const f = setup();
369+
const admission = holdNextLoad(f.review), run = holdNextLoad(f.review);
370+
const logged = vi.spyOn(console, 'error').mockImplementation(() => {});
371+
const getTask = f.store.getTask.bind(f.store);
372+
try {
373+
const starting = f.coordinator.start('review-token', randomUUID());
374+
await admission.loading; admission.release();
375+
const started = await starting;
376+
await run.loading;
377+
f.store.getTask = () => { throw new Error('database is locked'); };
378+
run.fail(new Error('History worker crashed.'));
379+
// The job's promise settles without rejecting, so the server's `.finally` and `close()` see no rejection.
380+
await expect(f.coordinator.current).resolves.toBeUndefined();
381+
f.store.getTask = getTask;
382+
expect(logged).toHaveBeenCalledWith(`Could not end merge job ${started.id}; it shows as running until the server restarts: "database is locked"`);
383+
expect(f.store.getMergeJob(identity, started.id)).toMatchObject({ state: 'running' });
384+
} finally { f.store.getTask = getTask; logged.mockRestore(); }
385+
});
386+
362387
it('leaves the task approved, merge blocked when a retried job is queued', async () => {
363388
const f = setup({ queue: true, checks: n => (n === 1 ? [{ context: 'build', appId: null, state: 'failure' }] : []) });
364389
await f.coordinator.start('review-token', randomUUID());

0 commit comments

Comments
 (0)