Skip to content

Fix the DR drill's replica startup race; add a never-promoted standby - #266

Merged
lesnik512 merged 2 commits into
mainfrom
test/dr-three-node
Oct 28, 2026
Merged

lesnik512 merged 2 commits into
mainfrom
test/dr-three-node

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Two changes to the failover drills (just test-dr). Neither touches library code.

Replica startup race. The DR cluster sometimes came up with the replica exited. The primary's healthcheck ran pg_isready over the Unix socket. That passes while the postgres entrypoint's temporary init server is up, but that server doesn't listen on TCP, so the replica's pg_basebackup got "Connection refused". To reproduce it, I added an init script that sleeps 5s. The primary went healthy and the replica failed straight away. With pg_isready -h 127.0.0.1, the same setup waits for the real server and the replica starts streaming. Both replicas use the TCP check too.

Three-node drill. A new pg-standby streams from the primary and is never promoted. The drill's async_creator tries the hosts in the order primary, standby, replica. After failover, every connect has to reject the standby on target_session_attrs="primary" before it reaches the promoted replica. The drill now asserts that at least one connect was rejected by the standby and that the standby is still in recovery.

Results from just test-dr:

Scenario Publish recovered Subscriber recovered Redeliveries Standby rejected
kill 0.9s 2.2s 0 4x
blackhole 11.9s 16.4s 0 4x

@github-actions

Copy link
Copy Markdown

Benchmark gate

✅ gate passed

scenario msg/s delete/msg WALrec/msg WALB/msg fpi upd del dead_tup
consumer/w1/b10 1132 1.000 6.72 921 242 5000 5000 10000
consumer/w1/b100 1235 1.000 7.11 1154 243 5000 5000 10000
consumer/w2/b10 1451 1.000 6.71 922 242 5000 5000 10000
consumer/w2/b100 1679 1.000 7.11 1154 243 5000 5000 10000
consumer/w4/b10 1461 1.000 6.80 985 261 5000 5000 10000
consumer/w4/b100 1787 1.000 7.11 1155 244 5000 5000 10000
consumer/w1/b100/tfbs100 6323 0.010 6.11 1130 243 5000 5000 10000
consumer/w1/b100/interval1-10 1279 1.000 7.11 1154 243 5000 5000 10000
producer/w1/b100 4278 0.000 3.04 597 1 0 0 0
producer/w1/b100/ppt1 1020 0.000 4.04 635 0 0 0 0

Gated (fails the build): delete_calls + tuple counters (upd/del/ins) + the producer's insert_calls, exact; select_calls and other_calls within +2; wal_records within a 10% band; fetch_calls within 2x. msg/s, WAL bytes and total calls are informational (timing/FPI noise).

@lesnik512
lesnik512 merged commit 774f9b6 into main Oct 28, 2026
15 checks passed
@lesnik512
lesnik512 deleted the test/dr-three-node branch October 28, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant