Performance Improvements

[kderme]

As mentioned in #1119 there is an increase in the memory usage in db-sync. Sync time can also be improved.

Ideas for this

• Removal of withdrawlal validation validateRewardWithdrawals. Since rewards seem to work properly we no longer need this.
• Removal or simplification of validateEpochRewards. We can validate the number of rewards instead of summing them.
• Investigate laziness in the Statement Cache of Persistent. Persistant caches all queries, which can result in a huge amount of memory of db-sync runs for long. Also the structures it uses are not strict.
• Make all fields and states in db-sync strict.
• Check strictness of LRU cache.
• Investigate how we maintain the ledger state. Reading/Storing it to disk, apply it in a strict way. Keep a list of ledger states in a way that there is sharing of heap space.
• Rewards and stake distribution come in chunks of variable length. Make sure we handle them and insert them in chunks of fixed length to avoid sudden memory increase.
• Query the db for datum and scripts before inserting. This reduces the overhead of serialising the same scripts/datums multiple times.
• ghc profiling
• Remove Foreign Keys Remove Foreign Keys #1082
• Removing unused keys/indexes Remove unused Indexes, Unique and Primary Keys #1087

[erikd]

Rather than removing validateRewardWithdrawals I would prefer to make the validation optional. Maybe by seeing if an environment variable is set.

[profd2004]

Closing this as part of a triage pass over the older open issues.

This umbrella has been overtaken by the Persistent→Hasql rewrite in 13.7.x, which replaced the old insertion pipeline and brought significant performance improvements; newer perf work continues in current issues/PRs (e.g. #2075).

Happy to reopen anything specific that remains.