Repository navigation
fix(db): treat an unreadable image as proof of corruption - #36
Merged
Merged
Conversation
6e41ad7 narrowed quarantine to a positive corruption verdict after two incidents destroyed databases that were never damaged. The narrowing is right for contention, but it went one branch too far: when quick_check could not run at all the database was kept, whatever the reason. That inverted the policy. Quarantine could then only fire for a file that opens fine and grades bad - the milder damage - while a file whose header SQLite cannot even recognise was kept for ever. The docstring's reason for quarantining at all, that "looping on a permanently corrupt file would just restart Predbat for ever", describes exactly the file that was being kept, and DatabaseManager.recover_from_error calls _cleanup_db_retry precisely so it "recovers or starts a fresh database rather than looping" - which it could no longer do. The result was Predbat running with db_primary set and every history write failing for the life of the process, with no notification. Both incidents behind 6e41ad7 were contention: a lock, and a file a later integrity_check proved healthy. Contention is already excluded one branch earlier by the OperationalError check, and every benign SQLite failure - BUSY, LOCKED, CANTOPEN, IOERR - is an OperationalError. What still reaches quick_check as a plain DatabaseError carrying a result code is SQLITE_CORRUPT or SQLITE_NOTADB, and both are positive reports of damage. Quarantine now also fires when quick_check itself fails with one of those two codes. The match is on the result code rather than the exception class, because ProgrammingError ("Cannot operate on a closed database") is a DatabaseError too and a bug of our own must never cost the user their history; the code is masked to its low byte so extended codes such as SQLITE_CORRUPT_INDEX still match. The test asserted the old behaviour and had been failing on main since 6e41ad7, which never updated it. It now pins every branch: transient error, lock contention, a clean verdict with a stuck cleanup, and a DatabaseError with no corruption code all keep the database; a graded-bad image and an unreadable one are both quarantined, sidecars included. The positive-verdict case the old test claimed to cover was in fact exercising the unreadable path, so it is now tested for real against a damaged database image. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pre-commit has been failing on main independently of any change here, which also stops the workflow before it reaches the unit tests. All of it is in files this branch does not otherwise touch: - cspell: datasheet, gasbill and unsubtracted were unknown words in annual_heat.py, annual.py and web_annual.py - ruff: two unused imports - black: three files needed reformatting Every change is the hooks' own auto-fix, applied with the pinned versions (ruff 0.11.4, black 23.11.0) so it matches what CI generates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the pre-existing
db_managertest failure onmain(the "corruption" subtest), by fixing the code rather than the test.What was wrong
6e41ad7 (#35) narrowed quarantine to a positive corruption verdict after two incidents destroyed databases that were never damaged. That narrowing is right for contention, but it went one branch too far: when
quick_checkcould not run at all, the database was kept whatever the reason.That inverted the policy. Quarantine could only fire for a file that opens fine and grades bad — the milder damage — while a file whose header SQLite cannot even recognise (
file is not a database) was kept for ever.Two places in the codebase already documented the behaviour that had been lost:
_cleanup_db_retry's own docstring: quarantine exists because "looping on a permanently corrupt file would just restart Predbat for ever" — which is precisely the file that was being kept.DatabaseManager.recover_from_errorcalls_cleanup_db_retryso that it "recovers or starts a fresh database rather than looping". It could no longer do either, leaving Predbat running withdb_primaryset and every history write failing for the life of the process, unnotified.Both incidents behind 6e41ad7 were contention — a lock, and a file a later
integrity_checkproved healthy. Contention is already excluded one branch earlier by theOperationalErrorcheck, and every benign SQLite failure (BUSY,LOCKED,CANTOPEN,IOERR) is anOperationalError. What still reachesquick_checkas a plainDatabaseErrorcarrying a result code isSQLITE_CORRUPTorSQLITE_NOTADB— both positive reports of damage.The change
Quarantine now also fires when
quick_checkitself fails with one of those two codes.The match is on the result code, not the exception class:
ProgrammingError("Cannot operate on a closed database") is aDatabaseErrortoo, and a bug of our own must never cost the user their history. The code is masked to its low byte so extended codes such asSQLITE_CORRUPT_INDEX(779) still match.Everything 6e41ad7 protects is untouched — contention, a clean verdict, and an error with no corruption code all still keep the database.
Tests
The test asserted the old behaviour and had been failing on
mainsince 6e41ad7, which never updated it. It now pins every branch:OperationalError(lock contention)quick_checkgradesok, cleanup still failingDatabaseErrorwith no corruption code (ProgrammingError)quick_checkgrades the image damaged-journal/-wal/-shmwith the databasequick_checkcannot run:file is not a databaseThe positive-verdict case the old test claimed to cover was in fact exercising the unreadable path — it is now tested for real, against a genuinely damaged database image (garbage written over pages 3-6, header left intact) rather than mocks.
Verified:
db_manager9/9 pass, and the new test fails against the pre-changedb_engine.pyon the unreadable case, so it genuinely pins the semantics.blackclean andruff --select=F401(as CI runs it) clean.🤖 Generated with Claude Code