Improving Test Execution Time

[sgillespie]

Running the full suite of tests currently takes ~15 minutes, the majority of which is running the database integration tests. For this reason, we need to be judicious in choosing what to test. Speeding this up would allow us to include more variations of tests, such as on-disk and in-memory backends, or different insert options (full, UTxO, governance, etc).

The most effective things that would reduce execution time are:

• Run tests concurrently: Currently, tests are run sequentially, because they share a single PostgreSQL database. However, by creating multiple copites of the database schema, we could run multiple tests at the same time.

• Tune assertion backoff delays: The assertion backoff delays have a maximum wait of ~50 seconds, and a maximum interval time of 2.5 seconds. This could be tuned to reduce idle time. We could also have non-waiting variants of all the query assertions.

Furthermore, the fingerprinting mechanism is used to speed up tests, but provides only a minimal boost. I would therefore propose removing this.

[sgillespie]

To test out the effects of parallelizing the test suite, I updated the top-level test
groups to run in parallel with each of them getting their own dedicated database. The
tests within each per-era ran sequentially.

This makes the top-level groups:

testGroup  -- testGroup runs in parallel
  "cardano-chain-gen"
  [ Conway.unitTests iom knownMigrationsPlain db1,
    Babbage.unitTests iom knownMigrationsPlain db2,
    Alonzo.unitTests iom knownMigrationsPlain db3,
    testProperty "QSM" $ Property.prop_empty_blocks db4 iom knownMigrationsPlain
  ]

And the era test groups look like this:

dependentTestGroup    -- dependentTestGroup runs sequentially
    "blocks with txs"
    AllFinish         -- run all tests regardless of whether earlier tests succeed
    [ test "simple tx" Tx.addSimpleTx
    , test "simple tx in Shelley era" Tx.addSimpleTxShelley
    , test "simple tx with ledger disabled" Tx.addSimpleTxNoLedger
    , test "tx with treasury donation" Tx.addTxTreasuryDonation
    , test "consume utxo same block" Tx.consumeSameBlock
    , test "tx with metadata" Tx.addTxMetadata
    , test "tx with metadata disabled" Tx.addTxMetadataDisabled
    , test "tx with metadata whitelist" Tx.addTxMetadataWhitelist
    ]

Then to measure, I ran:

# Test the original, sequential test suite
env time -v cabal test cardano-chain-gen:tests

# Run the updated test suite, each era running in parallel
env time -v cabal test cardano-chain-gen:tests --test-option --num-threads=4

Summarizing the results:

Run	Wall	User CPU	Sys CPU	%CPU	Max RSS
Sequential	14:04	178s	49s	26%	748 MB
Parallel (4 threads)	6:45	172s	50s	54%	768 MB

This resulted in a speedup of about 2x, with Conway taking up the full 6:45 wall clock
time. Some more observations:

• Memory use hardly increased at 4 threads
• The test-suite is I/O bound, so there could be room for more threads.
• The majority of tests are in Conway, so this can't scale without further reorganization