Fjall very slow in benchmarks

[adriangb]

I did a little benchmark to compare various KV stores and the only one I had true issues with is Fjall, which seems to deadlock and/or be very slow.
Granted I've never used Fjall before and I mostly had Claude write these benchmarks, so I'm probably doing something completely wrong, but I wanted to share in case it was interesting to anyone here:

https://github.com/adriangb/blob-bench

[marvin-j97]

Ah so that is a bit of an API thing. Because the keyspace is dropped in new, it kills all the background workers which prevents any flushes which makes it essentially deadlock.

Other than that, you are using key value separation in RocksDB (BlobDB), but not in Fjall.
The rest of the (read) performance may get better with V3 because of various optimizations.

That being said, I cannot really recommend storing huge blobs in databases. You either pay with double writes (in LSM-trees, or B-trees with WAL (like SQLite WAL mode)), or difficult space reclamation. Also you lose the ability to partially read values from disk if needed.

[adriangb]

Thanks for the info!

Because the keyspace is dropped in new, it kills all the background workers which prevents any flushes which makes it essentially deadlock

Should I be calling some .close() method?

Other than that, you are using key value separation in RocksDB (BlobDB), but not in Fjall.

Is this something I can enable in Fjall?

FWIW the reason I'm trying to store large blobs in a database instead of on disk directly (what I currently do) is that I've essentially hit a similar to kubernetes/kubernetes#43916 but with dentree cache, so I'm hoping that by using a db I can reduce filesystem file churn since dbs will manage segments / similar. Feel like an "I just want to serve 5TB moment" but "I just want a high throughput system that persistently stores large blobs" 😭

[marvin-j97]

Should I be calling some .close() method?

Keep the keyspace in the struct:

 pub struct FjallKv {
+    keyspace: Keyspace,
     partition: Arc<fjall::Partition>,
 }

Is this something I can enable in Fjall?

PartitionCreateOptions::with_kv_separation

[adriangb]

Thanks that worked! I pushed it so that if someone else runs across this at least it's a somewhat reasonable comparison.

[adriangb]

If you have any recommendations for a KV store that would perform well storing large blobs I'd very much appreciate your input.

[marvin-j97]

LMDB may perform well because it only writes once. I would expect it to have generally better performance than ReDB. But it comes with sharp trade-offs such as fully OS-controlled page cache, DB files never being able to shrink (only space can be reclaimed), no compression etc.