Skip to content

Commit fd5d689

Browse files
committed
feat: migrate non-unique index to composite layout for improved performance
- Updated TextIndex to use ArrayList instead of CopyOnWriteArrayList for index storage. - Modified TransactionalMap to handle deep copies of ArrayList and CopyOnWriteArrayList. - Enhanced IndexMapTest and SingleFieldIndexTest to validate changes in index behavior. - Added tests for non-unique index handling in various scenarios, including ordering, range queries, and null values. - Introduced IndexEntryKey class for composite key representation in non-unique index layout. - Implemented IndexEntryKeySerializer for order-preserving serialization in RocksDB. - Created NonUniqueIndexScaleTest and NonUniqueIndexCompositeTest to ensure performance and correctness under high load and edge cases. - Bumped version to 4.4.0-SNAPSHOT in pom.xml and potassium-nitrite/pom.xml.
1 parent a0689d7 commit fd5d689

27 files changed

Lines changed: 1089 additions & 84 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,16 @@
1-
## Release 4.3.4 - Unreleased
1+
## Release 4.4.0 - Unreleased
2+
3+
### Upgrade Notes
4+
5+
- Non-unique single-field indexes now use a composite-key on-disk layout (one `(value, id)` row per entry) instead of a single growing id list per value #1260
6+
- The public API is unchanged. Existing databases are upgraded automatically: a legacy array-format non-unique index is rebuilt into the new layout the first time it is opened, and the old index map is dropped.
7+
- The storage format change is forward-only — once a database has been opened by this version, it can no longer be read by an earlier version of Nitrite. Back up before upgrading if you may need to roll back.
28

39
### Performance Improvements
410

11+
- Fixed performance degradation when inserting thousands of documents that share the same non-unique index key #1260
12+
- The old layout re-wrote (and, on persistent stores, re-serialized) an ever-growing id list on every insert, making bulk inserts O(n²). The composite-key layout makes each insert and removal an O(log n) point operation across all backends (in-memory, MVStore and RocksDB).
13+
- RocksDB orders keys by their serialized bytes, so the composite key uses an order-preserving encoding (correct ranges over negative numbers, variable-length strings, dates and booleans).
514
- Made `in` filter index scans look up each value directly instead of scanning every index entry, so `in` queries on large indexed collections are now as fast as `eq` instead of degrading to a full index walk #1258
615

716
## Release 4.3.3 - Jun 26, 2026

‎nitrite-bom/pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
<parent>
55
<groupId>org.dizitart</groupId>
66
<artifactId>nitrite-java</artifactId>
7-
<version>4.3.4-SNAPSHOT</version>
7+
<version>4.4.0-SNAPSHOT</version>
88
</parent>
99

1010
<artifactId>nitrite-bom</artifactId>

‎nitrite-jackson-mapper/pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
<parent>
55
<groupId>org.dizitart</groupId>
66
<artifactId>nitrite-java</artifactId>
7-
<version>4.3.4-SNAPSHOT</version>
7+
<version>4.4.0-SNAPSHOT</version>
88
</parent>
99

1010
<artifactId>nitrite-jackson-mapper</artifactId>

‎nitrite-mvstore-adapter/pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
<parent>
55
<groupId>org.dizitart</groupId>
66
<artifactId>nitrite-java</artifactId>
7-
<version>4.3.4-SNAPSHOT</version>
7+
<version>4.4.0-SNAPSHOT</version>
88
</parent>
99

1010
<artifactId>nitrite-mvstore-adapter</artifactId>

‎nitrite-mvstore-adapter/src/main/resources/META-INF/native-image/org.dizitart/nitrite-mvstore-adapter/serialization-config.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,9 @@
5959
{
6060
"name": "org.dizitart.no2.common.DBValue"
6161
},
62+
{
63+
"name": "org.dizitart.no2.index.IndexEntryKey"
64+
},
6265
{
6366
"name": "org.dizitart.no2.common.Fields"
6467
},
Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
/*
2+
* Copyright (c) 2017-2021 Nitrite author or authors.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*
16+
*/
17+
18+
package org.dizitart.no2.integration.collection;
19+
20+
import org.dizitart.no2.Nitrite;
21+
import org.dizitart.no2.collection.Document;
22+
import org.dizitart.no2.collection.NitriteCollection;
23+
import org.dizitart.no2.index.IndexOptions;
24+
import org.dizitart.no2.index.IndexType;
25+
import org.junit.After;
26+
import org.junit.Before;
27+
import org.junit.Test;
28+
29+
import static org.dizitart.no2.integration.TestUtil.createDb;
30+
import static org.dizitart.no2.integration.TestUtil.deleteDb;
31+
import static org.dizitart.no2.integration.TestUtil.getRandomTempDbFile;
32+
import static org.dizitart.no2.filters.FluentFilter.where;
33+
import static org.junit.Assert.assertEquals;
34+
35+
/**
36+
* Issue #1260: inserting thousands of documents that share the same non-unique index key must
37+
* scale - the old array layout re-serialized an ever-growing id list on every insert (O(n²) on
38+
* a persistent store). The composite-key layout makes each insert an O(log n) point write. This
39+
* test exercises that path at scale on a file-backed MVStore and verifies it stays correct.
40+
*
41+
* @author Anindya Chatterjee
42+
*/
43+
public class NonUniqueIndexScaleTest {
44+
private String filePath;
45+
private Nitrite db;
46+
47+
@Before
48+
public void setUp() {
49+
filePath = getRandomTempDbFile();
50+
db = createDb(filePath);
51+
}
52+
53+
@After
54+
public void tearDown() throws Exception {
55+
if (db != null && !db.isClosed()) {
56+
db.close();
57+
}
58+
deleteDb(filePath);
59+
}
60+
61+
@Test
62+
public void testManyDocumentsWithSameKey() {
63+
NitriteCollection collection = db.getCollection("scale");
64+
collection.createIndex(IndexOptions.indexOptions(IndexType.NON_UNIQUE), "category");
65+
66+
int perCategory = 10000;
67+
for (int i = 0; i < perCategory; i++) {
68+
collection.insert(Document.createDocument("category", "hot").put("seq", i));
69+
collection.insert(Document.createDocument("category", "cold").put("seq", i));
70+
}
71+
db.commit();
72+
73+
// every id under a high-cardinality key must be retrievable through the index
74+
assertEquals(perCategory, collection.find(where("category").eq("hot")).size());
75+
assertEquals(perCategory, collection.find(where("category").eq("cold")).size());
76+
assertEquals(0, collection.find(where("category").eq("missing")).size());
77+
78+
// reopen the database: the on-disk composite index must survive a round trip
79+
db.close();
80+
db = createDb(filePath);
81+
collection = db.getCollection("scale");
82+
assertEquals(perCategory, collection.find(where("category").eq("hot")).size());
83+
84+
// removing one whole key must clear it without touching the other
85+
collection.remove(where("category").eq("hot"));
86+
db.commit();
87+
assertEquals(0, collection.find(where("category").eq("hot")).size());
88+
assertEquals(perCategory, collection.find(where("category").eq("cold")).size());
89+
}
90+
}

‎nitrite-native-tests/pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
<parent>
88
<groupId>org.dizitart</groupId>
99
<artifactId>nitrite-java</artifactId>
10-
<version>4.3.4-SNAPSHOT</version>
10+
<version>4.4.0-SNAPSHOT</version>
1111
</parent>
1212

1313
<artifactId>nitrite-native-tests</artifactId>

‎nitrite-rocksdb-adapter/pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
<parent>
55
<groupId>org.dizitart</groupId>
66
<artifactId>nitrite-java</artifactId>
7-
<version>4.3.4-SNAPSHOT</version>
7+
<version>4.4.0-SNAPSHOT</version>
88
</parent>
99

1010
<artifactId>nitrite-rocksdb-adapter</artifactId>

0 commit comments

Comments
 (0)