Increasing size of the database

[pearlpepper]

I noticed that the database size increases when adding documentation, and it also increases when removing documentation. I’m not sure how normal this behavior is, but I think the database size should decrease.

[pearlpepper]

Yesterday, I conducted many tests related to the database size during the addition and removal of documentation, and I want to share the findings:

• After removing documentation, the database size does not decrease.
• After removing documentation, the size may increase slightly.
• After removing documentation, you can perform a manual VACUUM, and the size will decrease slightly, but not to the same extent as before adding the documentation.

Hermes Agent also studied this issue and reported that:

• Only document data can be cleared using VACUUM, while vector data is preserved.
• He also noted that vector data is stored in JSON format, not in binary, and therefore takes up several times more space than binary data.

Perhaps I didn’t fully understand how it should work, but numerous tests confirmed this.
What particularly disappointed me was that indexing a few (5–10 pcs) .md pages can take up 10 MB of disk space.

[arabold]

Hermes and your review is right: The table is never shrinking unless a VACUUM is being performed. This is how SQLite works. For the sake of performance, deleted pages are not reclaimed automatically and the database grows indefinitely. The problem is, that a VACUUM locks the whole database, so no other operation can run anymore during the process, which can take anywhere between seconds to minutes.

In #482 I added a compact CLI option to manually perform that VACUUM. That should help, but is also just a workaround, of course. The probably best course of action would be to detect "downtime" and perform automatic periodic vacuum.

Regarding the size itself: Storing vector information needs a lot (!) of space in the database. Depending on the embedding model used, this will be several megabytes, as you see for yourself. If that's not acceptable, then disabling embeddings would be a valid option. You would still have regular full-text search, which works reliably well, too. It is just not able to find semantic relationships as efficiently as a vector database would.