Skip to content

Update to SQLCipher 4.20.0 (SQLite 3.54.1) on go-sqlite3 v1.14.52 - #4

Merged
laodouya merged 368 commits into
developfrom
feature/sqlcipher-4.20
Nov 4, 2026
Merged

laodouya merged 368 commits into
developfrom
feature/sqlcipher-4.20

Conversation

@laodouya

@laodouya laodouya commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

develop bundles SQLite 3.28.0 (April 2019) patched with SQLCipher 3.4.2. It cannot be updated by replacing the amalgamation: SQLite 3.32.0 removed the SQLITE_HAS_CODEC hooks that patch relies on. This rebuilds the fork on top of go-sqlite3 v1.14.52 with the SQLite that SQLCipher itself maintains.

Reviewing

GitHub shows 368 commits and about 167,000 changed lines because this also brings in the 347 go-sqlite3 commits between the old base of develop (v1.10.0, May 2019) and v1.14.52; the regenerated amalgamation alone accounts for about 152,000 of the lines. The changes on top of go-sqlite3 v1.14.52 show only what this fork does: the generated amalgamation, the vendored LibTomCrypt sources, and about 235 added lines in sqlite3.go, the tests, go.mod and the README.

The branch merges develop with the ours strategy, so this PR replaces the whole tree of develop while keeping its history, and later go-sqlite3 updates only bring in the new upstream commits.

Changes

  • SQLCipher 4.20.0, SQLite 3.54.1. sqlite3-binding.c, sqlite3-binding.h and sqlite3ext.h are generated from the SQLCipher v4.20.0 sources (./configure --with-tempstore=yes and make sqlite3.c) and wrapped the way go-sqlite3 wraps the official amalgamation.
  • LibTomCrypt 1.18.2 as the crypto provider, as before, so no system crypto library is needed. Only the sources SQLCipher uses are included: AES, CBC, HMAC, PBKDF2, SHA1/SHA256/SHA512, Fortuna and rng_get_bytes. develop had an untagged LibTomCrypt snapshot without SHA512, which SQLCipher 4 uses by default.
  • _crypto_key, carried over from develop, is applied with PRAGMA key on every new connection before any other statement. Opening now fails if the SQLite library is not SQLCipher, as with the libsqlite3 tag and a plain system SQLite, where the key used to be ignored and the database written in plain text. Keys containing NUL are rejected; they cut the statement short, and the syntax error showed part of the key. Both checks run before the database file is opened, so a rejected key leaves no file behind.
  • SQLCipher's default logging to stderr is turned off.
  • A go-sqlite3 regression is fixed: since 4954048 (v1.14.x), Exec without any arguments takes a fast path that skips the parameter check, so Exec("INSERT INTO t VALUES (?)") inserted NULL instead of failing with not enough args to execute query. CovenantSQL's TestConn relies on that error. The fast path now leaves statements with parameters unexecuted and returns the same error as before.
  • ICU: sqlite3_opt_icu.go adds the libraries that a static link of ICU needs (libicudata, the C++ runtime), as develop did. Dynamic linking keeps working.
  • Module path github.com/CovenantSQL/go-sqlite3-encrypt: v1.14.52 declares github.com/mattn/go-sqlite3, which CovenantSQL could not require under this repository's path.
  • v1.0.0 to v1.9.0 are retracted, in place of the v2.0.x retraction of go-sqlite3, as this repository has no v2 tags. The v1 tags come from go-sqlite3 and have no SQLCipher, and go get of the latest version resolved to v1.9.0, which ignores _crypto_key. After merging, a v1.10.0 tag makes go get resolve to this version.
  • Removed: the upgrade tool, which would overwrite the SQLCipher amalgamation with the official one and then commit, push and open a pull request; the CIFuzz workflow, which builds the OSS-Fuzz project of go-sqlite3 and fails on every pull request here; the .gitignore entries of the upgrade tool; and the sqlcipher.h and sqlite3-binding.c.diff of develop, since the SQLCipher sources now produce the amalgamation directly.
  • The README describes the fork, how to install it, the limits of the encryption and how to update SQLCipher, and the license files of SQLCipher and LibTomCrypt are included.

Not compatible with develop

New databases use the SQLCipher 4 defaults, and databases encrypted by develop (SQLCipher 3) do not open. A ConnectHook cannot set cipher_compatibility either, because Open runs its own pragmas right after the key. CovenantSQL has no deployment to keep compatible, so neither an option for this nor a migration note is added.

The README also documents two limits of SQLCipher itself: with cache=shared, a connection with a wrong key or no key can read a database another connection of the process opened with the right key (CovenantSQL opens its shared-cache connections with the same key), and backups between an encrypted and a plain database fail.

Testing

  • On linux/amd64 with Go 1.27.3: go vet and go test pass with the default tags, with the tags CovenantSQL builds its miner with (sqlite_omit_load_extension sqlite_vtable sqlite_fts5 sqlite_icu), and with libsqlite3 against the Debian SQLite.
  • TestCryptoKey checks that a database written with _crypto_key is not a plain SQLite file, that a wrong key or no key fails with ErrNotADB, that it reads back with the key, and that a key containing a quote works. TestCryptoKeyWithNUL and, with the libsqlite3 tag, TestCryptoKeyWithoutSQLCipher cover the new errors and check that no file is created, and TestNotEnoughArgsErrorMessage the Exec regression.
  • The go-sqlite3 workflow runs the tests with -race on Ubuntu and macOS with Go 1.24, 1.25 and 1.26. On Windows it only runs go build, which does not link the package, so Windows is not verified.
  • CovenantSQL's whole test suite passes with this driver on linux/amd64, and its static release build on golang:1.27-alpine produces working binaries.

eatonphil and others added 30 commits May 29, 2022 21:06
Support returning any from callbacks
(For mattn#965.)

This retraction will take effect when this commit is included in the
latest v1 release (presumably v1.14.11).
This commit adds the `sqlite_os_trace` build tag which sets the
`SQLITE_FORCE_OS_TRACE` and `SQLITE_DEBUG_OS_TRACE` compilation
flags. This produces verbose debugging output of every operating
system call made by SQLite.
Fixes a test which did not correctly exercise the multi-statement
Queryer functionality
Always rollback on a commit error
Add support for SQLITE_ENABLE_MATH_FUNCTIONS compile-time option via the sqlite_math_functions build tag.

Co-authored-by: Dominik Kraus <dominik.kraus@nktek.de>
# Enhancement

Update `README.md` with new instructions for Mac OS cross compiling.

# Why?

The current suggested solution `xgo` is no longer maintained (GitHub archived repository).

# Credits

Credits go to Pieter Claerhout (@pieterclaerhout) and his blog post: https://www.yellowduck.be/posts/cross-compile-a-go-package-which-uses-sqlite3
Added documentation for sqlite_vtable build tag to the README.
As it was, the vtable example will give an error when adding any kind of SQL constraint in the SQL statement. Updating the BestIndex method will ensure that adding SQL constraints will not result in errors

Signed-off-by: David Vassallo <davevassallo@gmail.com>

Signed-off-by: David Vassallo <davevassallo@gmail.com>
Add support for Serialize and Deserialize, which wrap sqlite3_serialize and sqlite3_deserialize.
…n#1069)

* Add build tags to support both x86 and ARM compilation on macOS

* Documentation fix: command line for build under macOS

* Global replace 'macOS X' -> 'macOS'. Fix typo in -tags cmd line param

* `README.md`: fix all `--tags` -> `-tags`

---------

Co-authored-by: Denis Dmitriev <dmitriev@itspartner.net>
use same workaround as authelia/authelia#6404
before mattn#1177 fixes the build
otherwise
mattn and others added 24 commits September 5, 2026 03:01
Flush statement cache when the schema changes
Replace schema probe with eager first step for cached statements
Generated from the SQLCipher v4.20.0 sources (SQLite 3.54.1) with
./configure --with-tempstore=yes and make sqlite3.c, then wrapped the
same way upgrade/upgrade.go wraps the official amalgamation.
SQLCipher uses LibTomCrypt for AES-256-CBC, HMAC and PBKDF2 with
SHA1, SHA256 and SHA512, and the Fortuna PRNG. Vendoring them keeps the
package buildable without a system crypto library.
The key is applied with PRAGMA key right after opening, before any
other statement reads the database.
Static builds need libicudata and the C++ runtime as well, in this
order.
hmac_init hashes long keys with hash_memory, and SQLCipher seeds its
Fortuna PRNG from rng_get_bytes. Also skip the encryption test with the
libsqlite3 tag, which links the system SQLite without SQLCipher.
This branch starts from go-sqlite3 v1.14.52 and replaces the SQLCipher
3.4.2 setup of develop, so none of the develop changes are kept.
It builds the OSS-Fuzz project of mattn/go-sqlite3, which clones the
upstream repository and cannot find the commits of this fork, so it
fails on every pull request here.
The fast path for Exec without arguments, added in 4954048, executes
the statement without checking its parameters, so Exec("INSERT ...
VALUES (?)") inserted NULL instead of failing with "not enough args to
execute query" as before. Leave statements with parameters unexecuted
in that path and return the same error.
SQLCipher writes a line to stderr on errors such as a wrong key, which a
library should not do on its own. Applications can still enable it with
PRAGMA cipher_log.
With the libsqlite3 tag and a plain system SQLite, PRAGMA key does
nothing and the database was written in plain text. A NUL in the key
cut the PRAGMA statement short, and the syntax error showed part of the
key. Also check that a wrong or missing key fails with ErrNotADB.
It replaces the SQLCipher amalgamation with the official SQLite one, and
upgrade.sh also commits, pushes and opens a pull request.
These tags come from go-sqlite3 and have no SQLCipher, so go get of the
latest version gave a driver that ignores _crypto_key.
Drop the advice to run PRAGMA key, which only reaches one connection of
the pool.
The checks need no connection, and running them after sqlite3_open_v2
left an empty database file behind when they failed.
No databases encrypted by earlier versions of this fork need migrating.
It comes from go-sqlite3; this repository has no v2 tags.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.