fix(training): serialize queue recovery across processes

[DrAbdulmalek]

Summary

• Adds an OS-backed lock file alongside the in-process thread lock, serializing append/count/flush queue operations across independent worker processes sharing OMNI_HF_QUEUE_DIR.
• Adds a POSIX multiprocess regression test proving a second process blocks until the shared queue lock is released.
• Records a metadata-only remote_present reconciliation event when a retry finds every eligible payload in the verified remote baseline.
• Applies archive and queue-cleanup failure event recording to that reconciliation path as well as the normal post-push path.
• Uses unique, exclusive archive filenames so consecutive flushes cannot overwrite an earlier archive by reusing a same-second timestamp.

Scope and safety

• Stacked on PR fix(training): guard HF dataset snapshots with parent-commit CAS #195; intended chain remains feat(training): add content-bound approval and append-only export ledger #191 → feat(training): record append-only export delivery outcomes #194 → fix(training): guard HF dataset snapshots with parent-commit CAS #195 → fix(training): serialize queue recovery across processes #196.
• The lock coordinates processes sharing the same local queue directory. CAS remains necessary for writers operating against the same remote dataset from different machines.
• Tests must run offline; no live Hugging Face upload.
• Keep Draft until targeted tests pass on this exact head and the entire dependency chain is reviewed. Do not merge before its base PR is resolved.

[DrAbdulmalek]

Follow-up audit — queue concurrency and ambiguous delivery

The branch now includes:

• A shared OS lock file plus an in-process re-entrant thread lock; append/count/flush use the common guard, and the lock spans the remote CAS through archive and queue cleanup.
• A POSIX multiprocess regression test for blocking on the same queue lock.
• A remote_present metadata-only event on the retry path where the verified remote baseline already contains every eligible row, with explicit archive/queue-cleanup failure events on that path.
• Unique exclusive archive filenames to prevent same-second archive overwrite.

Important boundary: this serializes processes sharing the same queue directory on a filesystem that honors advisory locks. It does not coordinate processes on separate machines; the remote CAS remains the guard for those writers.

I am waiting for the focused offline tests and security/drift/Docker checks on this exact HEAD. No live Hugging Face upload or merge has been performed.

[DrAbdulmalek]

CI diagnosis and correction:

• Run 38229238394 on the prior head failed: 41 passed, 1 failed. The failure was in the new ambiguous-commit regression fixture: it wrote the two literal characters \\n instead of a JSONL newline, so the queue parser correctly treated the staged row as corrupt and the simulated push path was never reached.
• Fixed the fixture on commit 119d9cca210b6a3cd8e3ed9db9c8a859f7a8e9e8. New targeted CAS/export tests are running on that exact head; I will only treat them as verified after reading the completed job log.
• No production branch or remote dataset was changed.

[DrAbdulmalek]

Validation update — head 119d9cc

• HF Dataset CAS Tests: success. I read the completed job log: 42 passed in 1.10s, with HF_HUB_OFFLINE=1 and HF_DATASETS_OFFLINE=1.
• Secret Gate: success.
• HF Space Drift Gate: success.
• Docker: API image build passed; Gradio image build is still in progress at the latest poll.

The 42-test result includes the ambiguous remote-commit/timeout recovery case, the process-lock regression, and migration of an existing delivery ledger while preserving its prior ambiguous event. No live Hugging Face dataset write was used.

Disposition remains TESTS_GREEN_PENDING_DOCKER_AND_INDEPENDENT_REVIEW. Keep Draft until the remaining job completes and the stacked chain receives independent review.