[Bug]: `in_progress_at` and `finalizing_at` are null on every completed batch

[sudiptob2]

Contact Details

sudiptobaral.me@gmail.com

What happened?

in_progress_at and finalizing_at are each set when the batch enters that state, then cleared by the next transition. A completed batch reports both as null. The batch itself succeeds; only the timestamps are lost.

// while running
{"status":"in_progress","created_at":1785688675,"in_progress_at":1785688677,"finalizing_at":null,"completed_at":null}

// once completed
{"status":"completed","created_at":1785688675,"in_progress_at":null,"finalizing_at":null,"completed_at":1785689462}

OpenAI returns these populated on a completed batch Ref: https://developers.openai.com/api/reference/resources/batches/methods/retrieve

I did some investigation and here are the probable cause for this bug,

Each transition rebuilds the whole status blob from a snapshot taken before the job started, so it overwrites whatever the previous transition wrote.

1. The worker fetches the job row once when it picks the job up, and passes that same *db.BatchItem to every transition of the run. It is never re-read.

llm-d-batch-gateway/internal/processor/worker/worker.go

Lines 288 to 289 in 82f0af6

	// get job item from db
	jobItem, err := p.poller.fetchJobItemByID(pollCtx, task.ID)

2. UpdatePersistentStatus unmarshals its base status from that stale dbJob.Status.

llm-d-batch-gateway/internal/processor/worker/status_updater.go

Lines 97 to 104 in 82f0af6

	var original openai.BatchStatusInfo
	if err := json.Unmarshal(dbJob.Status, &original); err != nil {
	return err
	}
	// Build base status first (status, timestamps, counts), then apply modifiers
	// to set additional fields (e.g. file IDs) that aren't part of the standard transition.
	updated, err := batch_utils.BuildUpdatedStatusInfo(&original, newStatus, counts, slo)

3. BuildUpdatedStatusInfo copies the snapshot and sets exactly one timestamp for the new state. The other timestamps stay at whatever the snapshot had, which at pickup time is null.

llm-d-batch-gateway/internal/shared/batch_utils/status_utils.go

Lines 49 to 65 in 82f0af6

	now := time.Now().Unix()
	updated := *originalStatus
	updated.Status = newStatus
	switch newStatus {
	case openai.BatchStatusInProgress:
	updated.InProgressAt = &now
	case openai.BatchStatusCompleted:
	updated.CompletedAt = &now
	case openai.BatchStatusFailed:
	updated.FailedAt = &now
	case openai.BatchStatusCancelled:
	updated.CancelledAt = &now
	case openai.BatchStatusExpired:
	updated.ExpiredAt = &now
	case openai.BatchStatusFinalizing:
	updated.FinalizingAt = &now

4. The write replaces the entire status column rather than merging into it, so those nulls land in the database.

llm-d-batch-gateway/internal/database/postgresql/db_core.go

Lines 371 to 375 in 82f0af6

	if len(contents.Status) > 0 {
	setClauses = append(setClauses, fmt.Sprintf(colStatus+" = $%d", argIdx))
	args = append(args, contents.Status)
	argIdx++
	}

So in_progress writes in_progress_at over a blank snapshot, finalizing rebuilds from that same blank snapshot and wipes it, and completed wipes finalizing_at in turn.

Version

main @ 82f0af6

Steps to Reproduce

1. Bring up the dev environment with make dev-deploy
2. Upload the input file to POST /v1/files with purpose=batch, then create a batch from the returned file ID via POST /v1/batches.
3. Poll GET /v1/batches/{id} every couple of seconds until the batch finishes.

While the batch is running, in_progress_at holds a real timestamp. As soon as it reaches completed, it is null again. finalizing_at is also set and cleared between polls

Environment

- Kubernetes v1.34.0 (kind v0.30.0), deployed with `make dev-deploy`
- macOS 26.5.1, arm64

Relevant log output

{
  "status": "completed",
  "completed_at": 1785689462,
  "finalizing_at": null,
  "in_progress_at": null,
  "output_file_id": "file_e91839da-95fe-4032-9dde-95c5a3ed0461",
  "request_counts": { "total": 200, "failed": 0, "completed": 200 }
}