fix: prevent duplicate fill booking in live order reconciliation (stale live_fill_sync marker)

[z20251130]

Fixes #231

What happened (live incident)

A Binance USD-M market order (0.1 @ 685.48) was booked twice into the platform ledger after a trading-worker restart: two strategy_order_fills rows with the same exchange_order_id, identical price and qty; pending_orders.filled went 0.1 → 0.2 and the UI position doubled. Binance's API reported executedQty = 0.10 / positionAmt = -0.10 throughout — the exchange never had the doubled position; only the platform ledger did.

Root cause

tracked_fill_baseline() in app/services/pending_orders/sent_order_recovery.py short-circuits on the row-local exchange_response_json.live_fill_sync.tracked_filled marker. A sync that runs between order submission and fill writes tracked_filled = 0; the executor's fill update then bumps the row's filled column but does not rewrite the marker. After a worker restart, reconciliation re-claims the still-sent row, queries the exchange (cumulative_filled = 0.1), subtracts the stale baseline (0), and re-books the full quantity.

The issue contains a deterministic minimal reproduction plus the live timeline.

Fix

tracked_fill_baseline() now cross-checks every known source and takes the largest fill count (avg price travels with it), preserving the existing multi-leg semantics:

1. executor phases.executor.market_summary when it names this exchange order — the row then tracks legs separately (e.g. a maker/limit leg booked next to the market leg), so the row aggregate must NOT compete (keeps test_live_sent_sync_tracks_market_leg_without_overwriting_limit_fill green);
2. otherwise the row aggregate (filled/avg_price) — for a single-leg order it IS this leg's fill (keeps test_live_sent_sync_finalizes_after_restart_without_duplicate_fill green);
3. the live_fill_sync marker — candidate only, never short-circuits the fresher sources;
4. new persisted_order_fill_baseline() (fill_records.py): sums strategy_order_fills by exchange_order_id — the ledger is authoritative whenever it is ahead of the row, so reconciliation can never re-book quantity the ledger already holds. On DB failure it degrades to the previous behavior.

pending_order_worker.py passes the ledger baseline into the sync before computing the delta; the delta math and downstream booking are unchanged.

Tests

• new test_live_sent_sync_ignores_stale_zero_fill_sync_marker — regression reproducing the live incident (stale zero marker + executor fill + ledger fill ⇒ no re-booking, snapshot stays 0.1);
• new baseline-ranking unit tests (stale zero marker, partial stale marker vs. ledger);
• full tests/ suite in the backend image: 1617 passed with this patch vs. 1614 passed on main (same 8 pre-existing environmental failures in both runs, unrelated to this change) — i.e. +3 = the new tests, zero regressions.

[brokermr810]

Thanks for the detailed incident report, reproduction, and tests. We confirmed the root cause: a stale live_fill_sync.tracked_filled marker could override fresher fill data and cause the fill to be booked again after a worker restart. We’ve fixed this on main in commit cc6f523 by treating the sync marker as one candidate and selecting the freshest leg-aware baseline from the row and executor state. This keeps the fix focused and avoids adding a database query to every reconciliation cycle. Regression tests covering the stale-zero-marker scenario have also been added, and the full backend suite passes. Since the issue is now resolved on main, we won’t merge this PR. Thank you again for identifying the issue and providing such a thorough analysis.