Repository navigation
fix(sync): repair outdated rows in the health check; seed last_sync from the dump snapshot - #142
Merged
Merged
Conversation
…rom the dump snapshot A full comparison against upstream found 134 rows whose upstream `updated` is newer than the mirror's copy. The weekly health check compares id lists only, so it can't see them. It now fetches `?fields=id,updated`, and re-fetches rows where upstream changed after our copy through the same paced id__in path (missing first, shared per-table cap). The report gains outdated/refreshed, also in the mirror-health GitHub summary. Unparseable timestamps never trigger a refetch. migrate-to-d1.sh seeded each table's last_sync with the time the import finished. Rows that changed upstream between the dump snapshot and then were in neither the dump nor any later ?since= window: the 2026-04-08 rebuild (snapshot 18:42Z, import ~23:00Z) lost 12 rows and left 22 stale that way. last_sync is now the dump's meta.generated minus a 1h overlap, and the import refuses to seed a watermark from a dump without one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A full comparison of every table against upstream (2026-10-09) found 688 rows missing from the mirror, 353 rows upstream no longer has, and 134 outdated rows (upstream
updatednewer than our copy). The upstream causes are tracked in peeringdb/peeringdb#2057. This PR fixes the two causes on our side.1. The health check now repairs outdated rows
The weekly check (
sync/health.js) compared id lists only, so it could fix missing and stale rows but not outdated ones.?fields=id,updatedinstead of?fields=id: same number of upstream requests, larger responses.updatedis newer are re-fetched through the existing pacedid__inpath. Missing rows go first, under the shared per-table cap (MISSING_FETCH_CAP).outdated(count) andrefreshed(ids). A refresh counts as a repair, so the mirror-health workflow opens or updates its issue.mirror_health_report.pyrenders both and tolerates older reports without them.2.
migrate-to-d1.shseedslast_syncfrom the dump snapshotIt set each table's
last_synctostrftime('%s','now'), the time the import finished. Rows that changed upstream between the dump snapshot and that moment were in neither the dump nor any later?since=window. The 2026-04-08 rebuild (snapshot 18:42:20Z, files fetched 22:56Z) lost 12 rows that way and left 22 outdated. Facilities 17066 and 17067 are two of them.last_syncis now each dump'smeta.generatedminusSYNC_OVERLAP_S(1h). Re-fetching overlapping rows is harmless, since writes replace by id.meta.generatedstops the import instead of seeding an unsafe watermark.Not in this PR
The Sept 23 cluster (569 netixlans with
updatedat 04:05 that no?since=request returns) is upstream's to fix (#2057). Widening oursincewindow would multiply sync writes for every run. The improved health check repairs it weekly.Testing
health_sqlite.test.js(12/12) has new cases: outdated rows are refreshed and reported apart from inserted ones; up-to-date rows aren't rewritten; rows where ours is equal or newer, or upstream's timestamp is unparseable, cause no re-fetch. The cron wiring test now assertsfields=id,updatedrequests.1775670140= 2026-04-08T17:42:20Z; poc from its Apr 15 snapshot), and a dump withoutmeta.generatedexits 1.mirror_health_report.pywith new and old report shapes.After release
The next run (Sunday 03:07 UTC, or
POST /sync/health.<secret>) should fix the current drift: about 662 missing, 303 stale and 113 outdatednetixlanrows plus smaller amounts elsewhere, all within the guards. The mirror-health workflow at 03:40 will report it.🤖 Generated with Claude Code