Skip to content

Commit 47e0ea5

Browse files
riteshshukla04Ritesh Shuklaclaudechrispader
authored
perf: improve query and write performance (#440)
Co-authored-by: Ritesh Shukla <riteshshukla@Riteshs-Mac-Studio.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Christoph Pader <chris@margelo.com>
1 parent c6d6aab commit 47e0ea5

37 files changed

Lines changed: 848 additions & 357 deletions

‎docs/content/docs/configuration/android.mdx‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,9 +39,30 @@ To omit R\*Tree from the bundled build, set this boolean in your app's `package.
3939

4040
`enableRTree` defaults to `true` independently of `performanceMode`. A non-boolean value fails Gradle configuration. Disabling it omits the default `SQLITE_ENABLE_RTREE` definition. Do not add `-DSQLITE_ENABLE_RTREE=0` to `nitroSqliteFlags` to disable it: SQLite checks whether the macro is defined, so that value still enables the module. Custom compile definitions can enable the module even when the package setting is `false`.
4141

42+
## Commit I/O
43+
44+
SQLite makes a committed transaction durable by writing it to storage and asking the operating system to flush the written data. In the default rollback journal mode, a commit first copies the original pages into a journal file and flushes it, then writes the new pages to the database file and flushes again, and finally removes the journal. For small writes, these flushes take most of a commit's time.
45+
46+
Android builds compile SQLite with two I/O options that Android's own platform SQLite also uses. Neither changes `PRAGMA synchronous`, and a commit is as durable as before:
47+
48+
- `HAVE_FDATASYNC=1` flushes with `fdatasync()` instead of `fsync()`. On Linux, `fdatasync()` still flushes the metadata needed to read the data back, such as the file size, and skips metadata such as modification times.
49+
- `SQLITE_ENABLE_BATCH_ATOMIC_WRITE=1` commits a transaction through the F2FS filesystem's atomic write when the database file supports it. The filesystem then stores either all of the transaction's pages or none of them, so SQLite does not need a journal file. When the filesystem lacks the feature or the atomic write fails, SQLite commits with the rollback journal as usual.
50+
51+
To build without batch atomic writes, set this boolean in your app's `package.json` and rebuild the native app:
52+
53+
```json
54+
{
55+
"nitroSQLite": {
56+
"enableBatchAtomicWrite": false
57+
}
58+
}
59+
```
60+
61+
`enableBatchAtomicWrite` defaults to `true`, and a non-boolean value fails Gradle configuration. As with R\*Tree, SQLite only checks whether the macro is defined, so `-DSQLITE_ENABLE_BATCH_ATOMIC_WRITE=0` in `nitroSqliteFlags` does not disable it. To flush with `fsync()` instead, add `-DHAVE_FDATASYNC=0` to `nitroSqliteFlags`.
62+
4263
## Threading
4364

44-
The native library opens each database with `SQLITE_OPEN_FULLMUTEX` and serializes calls on each handle. The JavaScript connection helper coordinates work per managed connection and rejects conflicting synchronous work. `nitroSqliteFlags` can change compile-time SQLite behavior, including `SQLITE_THREADSAFE`. Nitro SQLite rejects [independent connections](/docs/guides/multiple-connections) when SQLite was built with `SQLITE_THREADSAFE=0`; other database handles can still run concurrently, so an app disabling mutexes must serialize SQLite calls across the process.
65+
The native library serializes all work on each database handle with its own lock, so it opens handles with `SQLITE_OPEN_NOMUTEX` instead of also taking SQLite's per-call connection mutex. Each connection runs its async queries, batches, and file imports one at a time on a native thread of its own. The thread starts with the connection's first such call and stops once the connection is closed and its pending work has finished. The JavaScript connection helper coordinates work per managed connection and rejects conflicting synchronous work. `nitroSqliteFlags` can change compile-time SQLite behavior, including `SQLITE_THREADSAFE`. Nitro SQLite rejects [independent connections](/docs/guides/multiple-connections) when SQLite was built with `SQLITE_THREADSAFE=0`; other database handles can still run concurrently, so an app disabling mutexes must serialize SQLite calls across the process.
4566

4667
The app's `nitroSQLite.threadSafe` and `nitroSQLite.performanceMode` settings also configure Android's SQLite defaults. Custom `nitroSqliteFlags` override individual defaults. The native build additionally applies its CMake compiler options, including `-O2`.
4768

‎docs/content/docs/guides/parameters-and-results.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,4 +78,4 @@ The [`NitroSQLiteExceptionType` constants](/api/react-native-nitro-sqlite/variab
7878

7979
`DatabaseCannotBeOpened` covers several causes, including invalid paths, an already-open default connection, and filesystem failures. It does not by itself establish that the device is low on storage.
8080

81-
For exact types, see [`QueryResult`](/api/react-native-nitro-sqlite/type-aliases/QueryResult), [`NitroSQLiteQueryResult`](/api/react-native-nitro-sqlite/hybrid-objects/NitroSQLiteQueryResult), and [`NitroSQLiteQueryResultRows`](/api/react-native-nitro-sqlite/type-aliases/NitroSQLiteQueryResultRows). The row generic narrows `rows._array` and `rows.item()`, but not the raw `results` array. Query results also inherit Nitro hybrid object members; see [native access](/docs/guides/sync-and-async#global-helpers-and-native-access).
81+
For exact types, see [`QueryResult`](/api/react-native-nitro-sqlite/type-aliases/QueryResult), [`NitroSQLiteQueryResult`](/api/react-native-nitro-sqlite/interfaces/NitroSQLiteQueryResult), and [`NitroSQLiteQueryResultRows`](/api/react-native-nitro-sqlite/type-aliases/NitroSQLiteQueryResultRows). The row generic narrows `rows._array` and `rows.item()`, but not the raw `results` array. Query results are plain JavaScript objects; see [native access](/docs/guides/sync-and-async#global-helpers-and-native-access) for results returned without the connection helper.

‎docs/content/docs/guides/performance.mdx‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,21 @@ console.log(rows._array)
2121

2222
An async call still consumes device CPU and database I/O. Paginate large result sets instead of bringing every row into JavaScript at once.
2323

24+
## Reuse SQL text
25+
26+
Before SQLite runs a statement, it compiles the SQL text into a program for its virtual machine. Compiling one statement is quick, but the cost adds up when the same statement runs many times, such as an `INSERT` in a loop.
27+
28+
Each connection keeps the compiled programs of its 32 most recently used queries and data changes: statements that start with `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `REPLACE`, `WITH`, or `VALUES`. When `execute()` or `executeAsync()` receives the same SQL text again, it reuses the program with the new parameters. Pass values as parameters rather than formatting them into the SQL so the text stays identical between calls. Other statements, such as `PRAGMA`, `ATTACH`, and `BEGIN`, compile on every call because SQLite can apply their effect while compiling them. Use [prepared statements](/docs/guides/prepared-statements) when you want to manage a compiled statement's lifetime yourself.
29+
30+
```ts
31+
for (const metric of metrics) {
32+
db.execute('INSERT INTO metrics (key, value) VALUES (?, ?)', [
33+
metric.key,
34+
metric.value,
35+
])
36+
}
37+
```
38+
2439
## Group related writes
2540

2641
One `executeBatchAsync()` call executes a fixed set of statements inside a native transaction. Use nested parameter arrays for repeated SQL. Use `db.transaction()` when application logic must inspect a result between writes. Both keep a transaction open while they work; keep the scope short and await each operation inside a transaction callback.

‎docs/content/docs/guides/sync-and-async.mdx‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ console.log(smallResult.results, largerResult.results)
2020

2121
Use synchronous calls when immediate results are useful and the work is small. Prefer async calls for queries that may scan many rows, batches, or file imports. [Performance guidance](/docs/guides/performance) covers other ways to keep work bounded.
2222

23+
An async query still delivers its rows to JavaScript, and only the JavaScript thread can turn them into objects. For a large result, `executeAsync()` hands rows over in batches while SQLite reads the rest, so building the objects overlaps with the query instead of starting after it. The promise still resolves once, with every row in order.
24+
2325
An `open()` connection submits ordinary async statements in call order. Native operations on that connection run one at a time in submission order, but await a write before starting a read that depends on it so you can handle write errors. Batches and transactions wait for preceding statements and reserve the connection until complete. A synchronous operation, `close()`, or `delete()` throws a busy error if work is pending or active. Await async work before calling a synchronous method on the same connection:
2426

2527
```ts
@@ -38,7 +40,7 @@ Prepared statement `executeAsync()` reserves the connection queue until it finis
3840

3941
The [`NitroSQLite` export](/api/react-native-nitro-sqlite/variables/NitroSQLite) also has `execute`, `executeAsync`, `prepare`, `executeBatch`, `executeBatchAsync`, and `transaction` helpers that take a database name. `NitroSQLite.open(options)` is the same helper as the named [`open()` export](/api/react-native-nitro-sqlite/functions/open). Global queries join the default connection's JavaScript queue when the name was opened through `open()`. They do not address independent connections; use the object returned by `open()` for those. Without a default managed connection, global `execute` and `executeAsync` call the native methods directly and still need an open native database handle. Global `prepare`, batches, and transactions need a default connection opened through `open()`.
4042

41-
`NitroSQLite.native` exposes the underlying [`NitroSQLiteNative` hybrid object](/api/react-native-nitro-sqlite/hybrid-objects/NitroSQLiteNative). Its methods take a database name and return raw results without the connection helper's `rows` container. They also throw native errors without converting them to `NitroSQLiteError`.
43+
`NitroSQLite.native` exposes the underlying [`NitroSQLiteNative` hybrid object](/api/react-native-nitro-sqlite/hybrid-objects/NitroSQLiteNative). Its methods take a database name and return raw results without the connection helper's `rows` container. They also throw native errors without converting them to `NitroSQLiteError`. Its `executeAsync()` accepts an optional `onRows` callback that receives leading rows in batches, in order; the resolved `results` then holds only the rows after the last batch.
4244

4345
```ts
4446
import { NitroSQLite } from 'react-native-nitro-sqlite'
@@ -52,6 +54,6 @@ try {
5254
}
5355
```
5456

55-
Native calls bypass the JavaScript queue. If you use them alongside an `open()` connection, coordinate access yourself. A native statement can run inside an active connection transaction without joining its callback's sequence. The raw result is a Nitro hybrid object with `name`, `toString()`, `equals(other)`, and `dispose()` members. Disposing it makes that result unusable. `dispose()` is not a database `close()` call, and ordinary garbage collection handles these objects. Do not dispose the shared `NitroSQLite.native` instance during normal database cleanup.
57+
Native calls bypass the JavaScript queue. If you use them alongside an `open()` connection, coordinate access yourself. A native statement can run inside an active connection transaction without joining its callback's sequence. The raw result is a plain JavaScript object with `rowsAffected`, `insertId`, `results`, and `metadata`; ordinary garbage collection reclaims it. `NitroSQLite.native` itself is a Nitro hybrid object with a `dispose()` member. Do not dispose that shared instance during normal database cleanup; `dispose()` is not a database `close()` call.
5658

5759
The `NitroSQLite` export spreads the hybrid instance, but inherited native methods do not appear on that top-level object. Call them through `.native`. If you compile SQLite with `SQLITE_THREADSAFE=0`, also serialize access across separate database handles and native threads. See the [iOS](/docs/configuration/ios#thread-safety-and-performance-mode) and [Android](/docs/configuration/android#threading) configuration pages.

‎docs/next.config.mjs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,9 +47,9 @@ const config = {
4747
},
4848
{
4949
source:
50-
'/api/react-native-nitro-sqlite/interfaces/NitroSQLiteQueryResult',
51-
destination:
5250
'/api/react-native-nitro-sqlite/hybrid-objects/NitroSQLiteQueryResult',
51+
destination:
52+
'/api/react-native-nitro-sqlite/interfaces/NitroSQLiteQueryResult',
5353
permanent: true,
5454
},
5555
]

‎docs/scripts/check-api.mjs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -239,7 +239,7 @@ assert.ok(
239239
corePackageHome.indexOf('## Classes'),
240240
'Hybrid Objects must appear before other API groups',
241241
)
242-
for (const name of ['NitroSQLiteNative', 'NitroSQLiteQueryResult']) {
242+
for (const name of ['NitroSQLiteNative']) {
243243
assert.ok(
244244
pagePaths.has(`react-native-nitro-sqlite/hybrid-objects/${name}.mdx`),
245245
`Missing Hybrid Object page: ${name}`,

‎example/tests/unit/specs/operations/execute.spec.ts‎

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ import type { ColumnType } from 'react-native-nitro-sqlite'
77
import type { NitroSQLiteQueryResult } from '@nitro-sqlite/specs/NitroSQLiteQueryResult.nitro'
88

99
const QUERY_RESULT_SIZES = [60, 1_000, 10_000]
10+
// Asynchronous reads hand rows to JavaScript in batches of 256; cover the batch edges.
11+
const ASYNC_BATCH_EDGE_SIZES = [255, 256, 257, 512, 513]
1012
const metadataQuery = `
1113
SELECT
1214
boolean_value,
@@ -177,14 +179,29 @@ export default function registerExecuteUnitTests() {
177179
const db = createQueryResultTestDb('query_result_rows_async')
178180

179181
try {
180-
for (const size of QUERY_RESULT_SIZES) {
182+
for (const size of [...QUERY_RESULT_SIZES, ...ASYNC_BATCH_EDGE_SIZES]) {
181183
const result = await db.executeAsync(
182184
'SELECT * FROM QueryResultRows ORDER BY id LIMIT ?',
183185
[size],
184186
)
185187

186188
expectQueryResultRows(result, size)
189+
expect(result.results).toBe(result.rows._array)
190+
expect(
191+
result.rows._array.every((row, index) => row.id === index + 1),
192+
).toBe(true)
187193
}
194+
195+
await db.transaction(async (tx) => {
196+
const result = await tx.executeAsync(
197+
'SELECT * FROM QueryResultRows ORDER BY id LIMIT ?',
198+
[1_000],
199+
)
200+
expectQueryResultRows(result, 1_000)
201+
expect(
202+
result.rows._array.every((row, index) => row.id === index + 1),
203+
).toBe(true)
204+
})
188205
} finally {
189206
db.close()
190207
db.delete()
@@ -275,7 +292,7 @@ export default function registerExecuteUnitTests() {
275292
})
276293

277294
describe('Select', () => {
278-
it('keeps positional columns and repeated result reads independent', () => {
295+
it('keeps positional columns in one plain result array', () => {
279296
const result = testDb.execute(
280297
'SELECT 1 AS duplicate, 2 AS duplicate, 3.5 AS "café", NULL AS nullable, zeroblob(2) AS payload',
281298
)
@@ -290,14 +307,9 @@ export default function registerExecuteUnitTests() {
290307
).toEqual([0, 0])
291308
expect(result.metadata?.duplicate?.index).toBe(0)
292309

293-
const firstRead = result.results
294-
const secondRead = result.results
295-
expect(secondRead).not.toBe(firstRead)
296-
expect(secondRead[0]).not.toBe(firstRead[0])
297-
firstRead[0]!.duplicate = 9
298-
expect(secondRead[0]?.duplicate).toBe(2)
299-
expect(result.results[0]?.duplicate).toBe(2)
300-
expect(result.rows.item(0)?.duplicate).toBe(2)
310+
// The native result is a plain object: `results` is one array, shared with `rows`.
311+
expect(result.results).toBe(result.results)
312+
expect(result.rows._array).toBe(result.results)
301313
})
302314

303315
it('preserves column metadata for empty results', () => {

‎packages/react-native-nitro-sqlite/android/sqlite-flags.gradle‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,24 @@ project.ext.resolveNitroSqliteDefaultFlags = { File appPackageFile, String custo
1919
def threadSafe = readBooleanFlag("threadSafe")
2020
def performanceMode = readBooleanFlag("performanceMode")
2121
def enableRTree = readBooleanFlag("enableRTree")
22+
def enableBatchAtomicWrite = readBooleanFlag("enableBatchAtomicWrite")
2223
def customFlagNames = (customFlags =~ /-D([A-Za-z_][A-Za-z0-9_]*)/).collect { it[1] }.toSet()
2324

2425
def defaultFlags = [threadSafe ? '-DSQLITE_THREADSAFE=1' : '-DSQLITE_THREADSAFE=0']
2526
// Omit the macro when disabled; SQLITE_ENABLE_RTREE=0 still enables SQLite's module.
2627
if (enableRTree) {
2728
defaultFlags += '-DSQLITE_ENABLE_RTREE=1'
2829
}
30+
// Without this SQLite substitutes fsync() for fdatasync(). Linux's fdatasync() also flushes the
31+
// metadata needed to read the data back, such as file size, so commits stay durable. Android's
32+
// platform SQLite makes the same choice.
33+
defaultFlags += '-DHAVE_FDATASYNC=1'
34+
// On F2FS, commit transactions with the filesystem's atomic batch write instead of a rollback
35+
// journal file. SQLite falls back to the journal when the filesystem lacks support.
36+
// Like RTree, SQLite only checks whether the macro is defined.
37+
if (enableBatchAtomicWrite) {
38+
defaultFlags += '-DSQLITE_ENABLE_BATCH_ATOMIC_WRITE=1'
39+
}
2940
if (performanceMode) {
3041
defaultFlags += [
3142
"-DSQLITE_DQS=0",

0 commit comments

Comments
 (0)