All notable changes to SirixDB are documented in this file.
- Projection indexes (experimental, analytical) are maintained incrementally by the
transactions that touch the record set: inserts, updates, deletes and moves resolve each dirty
record through exact locators or a bounded fence probe and rewrite only the touched persistent
units (row groups, order/fence chunks, Bloom chunks, set summaries, dictionary radix paths).
There is no dirty-record cliff, no whole-index rebuild, and no update-time invalidation — an
unattributable or corrupt touched unit fails the owning transaction (rollback-only) instead of
silently degrading the index. Column lookup is by the declared path relative to the record set,
so nested columns sharing a trailing name with another path are accepted. See
docs/PROJECTION_INDEXES.mdanddocs/PROJECTION_INDEX_INCREMENTAL_MAINTENANCE.md. - Load-time projection builds —
BasicJsonDBStore#create(collection, resource, reader|parser, ProjectionSpec)catalogues a projection on the still-empty resource and lets the load fill it, so a load plus its projection is ONE pass instead of a shred followed by a fulljn:create-projection-indexwalk. Until the load's final commit the projection's metadata slot holds the stale tombstone, so an interrupted load leaves queries on the generic pipeline rather than on a half-filled index. Seedocs/PROJECTION_INDEXES.md. - Bulk JSON loaders for fresh resources —
BulkJsonTreeAssembler(sequential) andParallelBulkJsonImporter(feeder scan + worker page builders + ordered adoption, for corpora whose top level is an array; NDJSON ridesNdjsonAsArrayInputStream). Both build trees structurally identical to cursor-based insertion, enforced by a full-field differential oracle, and refuse up front what they do not reproduce faithfully (hashTypeother thanNONE, stored DeweyIDs, node history, a non-empty target). Path statistics are built during the load by both loaders, through the very accumulator the cursor path defers through, so the arms cannot drift. (A resource written WITH path statistics now carries the versionedPathStatsrecord, a one-directional format break recorded indocs/DISK_FORMAT.md.) The parallel importer maintains PATH, CAS and NAME definitions and armed projection builds in the load's single pass — the workers extract each family's tuples from the primitives they already hold and the coordinator drains them into the families' ordinary bulk loaders; a catalogued projection with no armed load-time build is refused rather than silently left unmaintained, and valid-time interval maintenance is the one family that still refuses. API, scope, verification, tuning and measured numbers:docs/BULK_IMPORT.md. - Asynchronous durable commits (
AfterCommitState.KEEP_OPEN_ASYNC_COMMIT) — the middle ground between synchronous auto-commits and the async pre-flush: every threshold crossing creates a real, durable, queryable revision, but the durability barriers (index-catalogue fsync, buffered-tail flush, data force, uber-beacon writes) run on a background thread while the transaction keeps inserting into the next epoch. Depth-1 pipeline with backpressure; readers see a revision exactly when it hardens (durable-before-visible); a hardening failure poisons the transaction. FILE_CHANNEL backend, count-based auto-commit only. Seedocs/ASYNC_COMMIT_DESIGN.md. BasicJsonDBStore.Builder#useAsyncFlushForImports— bulk imports (e.g.jn:store) now use the asynchronous background pre-flush by default on the FILE_CHANNEL and MEMORY_MAPPED backends (the latter is the store's default on 64-bit Linux/macOS; both append through the file-channel writer): one semantically meaningful revision per import instead of parser-progress checkpoint revisions, with leaf serialization and I/O overlapped with parsing (parallel double-buffered background flush) and memory still bounded bynumberOfNodesBeforeAutoCommit. Crash-durability semantics change accordingly: nothing of an in-flight import is durable until its single final commit (the sync mode's checkpoint revisions each survived a crash). Pages whose serialization spills overlong records into OverflowPages are exempted from the background flush and committed by the final recursive commit (their durable image needs the overflow disk keys). Passfalse(or-Dsirix.import.asyncFlush=false) to restore synchronous intermediate auto-commits;-Dsirix.asyncFlush.parallelismsizes the shared background-serialization pool.
- Wide projection string columns serve through windowed leaf loads instead of whole-column
eager materialization. A projection whose worst-case resident size exceeds
-Dsirix.projection.eagerMaterializeBytes(default: the smaller of half the projection cache budget and a quarter of the heap) is served from bounded 128-leaf windows, and a column fill that would exceed the budget declines — the query re-enters the windowed whole-leaf route where one exists, otherwise the record path answers. A decline is a routing decision, not a corruption signal, so the index stays valid. Whole-column materialization is unchanged below the budget. Seedocs/PROJECTION_INDEXES.md. - A projection abandoned mid-load reports unconditionally. The one silent load-degradation —
a resource-wide value dictionary breaching its byte budget, which abandons the projection while
the load still succeeds — now prints
[proj] PROJECTION ABANDONEDon stderr with the quantity that breached the budget (the shipped log configuration discards the warning), and the stale tombstone records a machine-readableStaleReasonplus its remedy. The reason rides previously reserved flag bits, so thePIXMwire format is unchanged for existing tombstones (docs/DISK_FORMAT.md). - JSON revision-diff sidecars carry an integrity envelope (
sirix-diff-format,operation-count,operations-sha256). Readers (jn:diff, the REST diff handler, multi-revision resource copy) validate identity, count and digest before use and fall back to recomputing the authoritative diff otherwise; sidecars written without those fields are not accepted by this build. The REST diff response shape is unchanged — the envelope fields are internal and stripped before serving. Seedocs/DISK_FORMAT.md. - Breaking rename: the async intermediate "commit" is now called what it is — an async
flush.
AfterCommitState.KEEP_OPEN_ASYNC→KEEP_OPEN_ASYNC_FLUSH,StorageEngineWriter.asyncIntermediateCommit()→asyncFlush(),awaitPendingAsyncCommit()→awaitPendingAsyncFlush(). The mechanism creates no revision and no commit record; the old names asserted otherwise.
- Every write transaction leaked its three writer file descriptors (#1109) —
NodeStorageEngineWriter.close()never closed the underlying storage writer, and the storage-engine reader deliberately skips its page reader for write transactions, so nobody closed theFileChannelWriter: the buffered-data, SYNC-revisions and DSYNC-beacon channels leaked per commit until the GC's channel cleaner happened to reclaim them. Long-running, auto-committing workloads leaked 3 descriptors per commit and hit sporadicToo many open filesfailures. The writer is now closed when the page write transaction closes. - Optimizer walkers closed the shared database on every compile (#1109) — the CAS/path
index walker and the valid-time index walker closed the store's cached collection (which
closes the whole database and evicts it from the store) and potentially the cached shared
resource session after every compiled query. Besides the churn, a transient I/O failure
during the forced reopen was silently swallowed and disabled the VALIDTIME index rewrite for
that compile — the source of the flaky
windows-latestValidTimeIndexOptimizerRewriteTestfailures. The walkers now borrow the store-owned objects, close only the transactions they open, and log resolution failures. - Unbounded reader file-descriptor growth (#1109) —
FileChannelStorage.createReader()opened two freshFileChannels per reader while read-only transactions stay cached in their session, so descriptor usage grew with every query evaluated against a long-lived session. Readers now share a striped, lazily-opened, reference-counted channel pool (up tomin(availableProcessors, 8)pairs per storage, closed when the last borrowing reader closes): serial workloads keep the previous footprint, concurrent readers are capped at the stripe count, idle sessions hold zero descriptors, and striping keeps positional reads uncontended on Windows. As a side effect the valid-time optimizer gate test dropped from ~8 minutes to ~30 seconds.
- macOS and Windows support — the Umbra-style off-heap frame-slot allocator now runs on
all three platforms via a
VirtualMemorySPI (POSIXmmapon Linux/macOS,VirtualAllocreserve/commit with guaranteed-zero decommit-recommit reuse on Windows); the legacy Windows pool allocator stays reachable via-Dsirix.allocator=windowspoolfor rollback. Cross-platform CI lanes (macOS, Windows) back the claim; known limitation: crash-recovery re-initialization withMEMORY_MAPPEDstorage is unsupported on Windows (seedocs/KNOWN_LIMITATIONS.md).
- Path-summary corruption on nested-array removal (#1099) — removing an array element whose
subtree held the only references to nested
__array__path entries left stale in-memory references; a laterremoveFieldin the same transaction crashed withFailed to move to nodeKey: N. Cursor fast paths now validate in-memory references against the authoritative node mapping, and a removed subtree root that is a plain array releases its own__array__path entry (previously leaked). - RevisionEpochTracker poisoning on double-close (#1102) — a concurrent or reentrant transaction close deregistered its epoch ticket twice, permanently corrupting the tracker's free stack so no further transactions could be opened in the process. Tickets are now generation-tagged and ABA-safe, deregistration is idempotent, and both close paths run exactly once via a CAS latch.
- Large values crashed the commit (#1076) — string values beyond the largest slotted-page
size class (~512 KB) failed with
IndexOutOfBoundsExceptioninstead of diverting to an overflow page; multi-megabyte values now round-trip (regression-tested at 200 KB, 600 KB and 2 MB). - LZ4 decompression buffer leak (#1074) — the allocator-owned output buffer leaked when the
native decompression call itself threw; repeated corrupt reads could drain the frame-slot
budget into an
OutOfMemoryError. - macOS startup failure of the off-heap allocator (
__errno_locationbinding, Linux-only mmap flags) fixed; allocator symbols bind lazily and per-OS.
- The standalone launchers generated by
installDist/distZip(sirix-shell,sirix-cli,sirix-mcp) now bake in the required JVM flags; previouslysirix-queryshipped none andsirix-mcplacked--add-modules=jdk.incubator.vector, so any write operation (e.g.jn:store) crashed withNoClassDefFoundError: jdk/incubator/vector/Vector. sirix-shellstarts the interactive REPL when stdin is a terminal instead of requiring the undocumented-iqflag (piped input still executes as a single query), and Control-D/Control-C exit the shell cleanly instead of printingError: nullwith a non-zero exit code.- README quick-start corrections: the JSONiq field-access example uses
$$.name(the previous.namefailed to parse), the update example no longer produces an object with a duplicate key, and thesirix-shelltranscript matches the actual prompt and empty-line query termination.
- Bumped brackit to
1.0-alpha7, fixing the sequence functions (fn:subsequence/reverse/remove/insert-before) over JSON arrays and objects.
- Valid-time interval index — a persistent HOT-backed Relational-Interval-Tree that accelerates
jn:valid-at/jn:open-bitemporalwith anO(h)point stab, plus a CAS-index narrowing path and a linear-scan fallback. Bumped brackit to1.0-alpha6.
- Version bump and packaging fixes.
- V0 on-disk format with write-through (preallocated, buffered-beacon) commits.
- Typed, fail-closed vectorized analytics — a columnar group-by/aggregate path that lands within a small factor of DuckDB at 100M records.
- Durability and operational hardening across core, query, and the REST API; the REST read/query hot path no longer serializes concurrent requests (unordered
executeBlocking).
A rapid correctness, durability, and performance hardening series on the way to beta. Highlights:
- Serializer correctness — invalid JSON on single-named-scalar projections; unescaped object keys; number round-trip (exponent-without-dot, overflow, subnormal). (alpha12–alpha14, alpha21)
- Query semantics — int/double comparison (
XPTY0004) over mixed-numeric fields via brackit; a predicate-over-unwrapped-array optimizer that destructively returned empty;jn:open/xml:openbefore the first revision now returns an empty sequence. (alpha14, alpha16, alpha19) - Latency — node-history/query latency (event-loop blocking + an uncached history path); array-unbox
O(n²). (alpha20) - Durability / IO — streaming-shredder back-pressure deadlock; flaky
UberPageCorruptionTest(page data-length bounded to the file size); Docker fat-jar glob for versioned release builds. (alpha15, alpha21, alpha11)
See the GitHub releases for full per-version notes.
The first 1.0 alpha series — the API is stabilizing toward a production 1.0 release.
- MCP Server — Model Context Protocol server module for AI agent integration, with all 13 tool handlers wired to the SirixDB API
- Vector Embeddings — HNSW index for semantic search, with tombstone deletion, query-time efSearch tuning, and serialization versioning
- Cost-Based Query Optimizer — Multi-milestone optimizer with PathSummary statistics, selectivity estimation, cardinality propagation, predicate pushdown, DPhyp join ordering, and cost-driven pipeline routing
- Columnar Vectorized Execution — Zero-copy columnar extraction with late materialization, SIMD filters, ColumnBatch pipeline, and Mesh data structure for join fusion
sdb:explain()function — Inspect query plans from JSONiq- Comprehensive JSON test suite — 199 tests across 14 files
- Fuzz tests — Structural correctness fuzz tests for JSON mutations, DeltaVarIntCodec, DeweyIDEncoder, and PageLayout
- HOT (Height Optimized Trie) — PEXT-routed HOTLeafPage with prefix compression, zero-alloc MSDB, compact-first splits, atomic split+insert
- Removed heavy dependencies — Eliminated Guava, Dagger, Checker Framework, lz4-java, snappy-java, and brownies-collections
- CI pipeline — Parallelized test jobs and native image tests with matrix strategy
- Code quality — Replaced all star imports with explicit imports project-wide
- Production-readiness hardening across cost-based optimizer, HOT implementation, and MCP server
- UberPage dual-beacon fallback for corruption recovery
- Resource cleanup: try-with-resources for read-only transactions, defensive cleanup in snapshots
- Lock leak prevention, bounds checks, and input validation audit
- FSST compressed-domain comparison fixes and thread-safe extractors
- Bitemporal query support with
jn:valid-at(),jn:open-bitemporal(), and valid time configuration - Sliding snapshot page versioning strategy
- Native binary builds via GraalVM (sirix-cli, sirix-shell, REST API server)
- Interactive JSONiq/XQuery shell (sirix-shell)
- Kotlin CLI with full database operations
- REST API with Keycloak OAuth2/OpenID Connect authentication
- Merkle hash tree verification for tamper detection
- Path, CAS, and Name indexes
For the full commit history, see GitHub Commits.