[lesnik512]

Problem

After #262, a primary that goes silent costs two query_timeout_seconds. First the fetch loop times out and reconnects. Then the worker gets the first post-failover row and writes it on its own idle connection to the dead primary, so the worker waits out a second timeout before anything is acknowledged.

Fix

• The subscriber now keeps a generation counter, which the fetch loop bumps on every error.
• Each worker records the generation when it opens its writer connection. Before it runs the handler for a row, it checks whether the generation has moved on.
• When the worker's connection is stale, it:
  1. puts the row back on the in-memory queue;
  2. flushes any buffered deletes;
  3. invalidates its connection;
  4. raises _StaleWriterError.
• The reconnect loop handles that error with an INFO log. It doesn't back off and doesn't count it as a failure, so the worker reopens straight away and picks the row up again.
• The check happens before the handler runs, so the row isn't delivered twice.
• It only applies when there is a real writer connection. The test-broker path is unaffected.

Tradeoffs:

• Any fetch-loop error recycles the writers, not only timeouts. A transient SQL error on the fetch side therefore costs every worker one reconnect. That seemed cheaper than trying to tell a dead socket from other errors.
• The requeued row goes to the back of the queue. Rows within one fetch batch were never ordered across workers anyway.

Verification

• New integration test test_worker_recycles_writer_connection_after_fetch_loop_reconnects. It makes one fetch hang so it trips query_timeout_seconds, then checks that the next terminal delete runs on a different backend pid. It failed before the change (assert 102 != 102).
• Full suite: 824 passed at 100% coverage. test_publish_with_activate_in_delays_delivery failed once in an earlier full run; it's timing-sensitive and passed on its own 3 times and in the next full run.
• just test-dr, with the drill's query_timeout_seconds=10:

Scenario	Before (0.20.0)	After
kill	4.9s	1.8s
blackhole	33.1s	26.0s

The blackhole number now breaks down as:

• the idle wait until the next fetch, up to max_fetch_interval, which is 10s here;
• one query timeout;
• one backoff.

The worker no longer adds a second timeout. The docs (Database failover section and the production checklist) now describe recovery as about query_timeout_seconds plus max_fetch_interval.

[github-actions]

Benchmark gate

✅ gate passed

scenario	msg/s	delete/msg	WALrec/msg	WALB/msg	fpi	upd	del	dead_tup
consumer/w1/b10	1424	1.000	6.72	922	242	5000	5000	10000
consumer/w1/b100	1546	1.000	7.11	1154	243	5000	5000	10000
consumer/w2/b10	1674	1.000	6.71	922	242	5000	5000	10000
consumer/w2/b100	2102	1.000	7.11	1153	243	5000	5000	10000
consumer/w4/b10	1718	1.000	6.74	947	242	5000	5000	10000
consumer/w4/b100	2016	1.000	7.11	1153	243	5000	5000	10000
consumer/w1/b100/tfbs100	6998	0.010	6.11	1130	243	5000	5000	10000
consumer/w1/b100/interval1-10	1558	1.000	7.17	1196	265	5000	5000	10000
producer/w1/b100	3264	0.000	3.04	597	1	0	0	0
producer/w1/b100/ppt1	1096	0.000	4.04	635	0	0	0	0

Gated (fails the build): delete_calls + tuple counters (upd/del/ins) + the producer's insert_calls, exact; select_calls and other_calls within +2; wal_records within a 10% band; fetch_calls within 2x. msg/s, WAL bytes and total calls are informational (timing/FPI noise).