RGS crashing every few minutes. `Result::unwrap()` on an `Err` value: Elapsed(())

[SeverinAlexB]

I am getting lots of Result::unwrap() errors since a week. RGS crashes every few minutes. This is happening since 1 week.
I am running the current main branch.

Seems like the insert times out. The server has plenty of memory/disk space/cpu available.

My channel_updates table has 67,409,921 rows. There seems to be a unique contraint on this table which I highly suspect slows things down to the extend that the timeout is triggered.

A simple select count(*) from channel_updates takes 17s already so the insert will likely take even longer.

rgs=# SELECT COUNT(*) FROM channel_updates;
  count   
----------
 67409921
(1 row)

Time: 17512.966 ms (00:17.513)

Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: thread 'tokio-runtime-worker' panicked at /home/rgs/rgs2/src/persistence.rs:297:18:
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: called `Result::unwrap()` on an `Err` value: Elapsed(())
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]: stack backtrace:
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:    0:     0x60cac57a134c - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h304520fd6a30aa07
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:    1:     0x60cac55e41cb - core::fmt::write::hf5713710ce10ff22
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:    2:     0x60cac57765a2 - std::io::Write::write_fmt::hda708db57927dacf
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:    3:     0x60cac57a5df8 - std::panicking::default_hook::{{closure}}::he1ad87607d0c11c5
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:    4:     0x60cac57a6c96 - std::panicking::rust_panic_with_hook::had2118629c312a4a
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:    5:     0x60cac57a6795 - std::panicking::begin_panic_handler::{{closure}}::h7fa5985d111bafa2
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:    6:     0x60cac57a66f9 - std::sys::backtrace::__rust_end_short_backtrace::h704d151dbefa09c5
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:    7:     0x60cac57a66e4 - rust_begin_unwind
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:    8:     0x60cac55ac4a2 - core::panicking::panic_fmt::h3eea515d05f7a35e
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:    9:     0x60cac55ac885 - core::result::unwrap_failed::h7c8d8bbbcf45dc13
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:   10:     0x60cac5692804 - tokio::runtime::task::core::Core<T,S>::poll::h07188bddb2eac8bc
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:   11:     0x60cac55cc134 - tokio::runtime::task::raw::poll::h2037b2fc13a43256
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:   12:     0x60cac57b5035 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h098373b6dcc81fb6
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:   13:     0x60cac57b363b - tokio::runtime::scheduler::multi_thread::worker::run::h06b00e2af4810ce2
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:   14:     0x60cac57ac338 - tokio::runtime::task::raw::poll::hc895a065d1c83131
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:   15:     0x60cac57adf35 - std::sys::backtrace::__rust_begin_short_backtrace::h6910da00ffe66e61
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:   16:     0x60cac57badcd - core::ops::function::FnOnce::call_once{{vtable.shim}}::h73e9d65efeb1127e
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:   17:     0x60cac57a80ab - std::sys::pal::unix::thread::Thread::new::thread_start::hcdbd1049068002f4
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:   18:     0x7d119629caa4 - <unknown>
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:   19:     0x7d1196329c6c - <unknown>
Nov 12 13:28:37 rapid-gossip-sync.europe-west6-b.c.bitkit-prod.internal rapid-gossip-sync-server[4081452]:   20:                0x0 - <unknown>

Config:

# Environment variables can be set directly in the file
Environment="RAPID_GOSSIP_SYNC_SERVER_DB_HOST=localhost"
Environment="RAPID_GOSSIP_SYNC_SERVER_DB_USER=postgres"
Environment="RAPID_GOSSIP_SYNC_SERVER_DB_PASSWORD=****"
Environment="RAPID_GOSSIP_SYNC_SERVER_DB_NAME=rgs"
Environment="RAPID_GOSSIP_SYNC_SERVER_NETWORK=mainnet"
Environment="RAPID_GOSSIP_SYNC_SERVER_SNAPSHOT_INTERVAL=10800"
Environment="RAPID_GOSSIP_SYNC_SERVER_CACHES_PATH=/var/rgs_cache"
Environment="BITCOIN_REST_DOMAIN=10.172.0.3"
Environment="BITCOIN_REST_PORT=8332"
Environment="BITCOIN_REST_PATH=/rest/"
Environment="LN_PEERS=***"
Environment="RUST_BACKTRACE=full"

[SeverinAlexB]

Workaround provided by @TheBlueMatt.

-#[tokio::main]
+#[tokio::main(worker_threads = 20)]

System Info:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.1 LTS
Release:	24.04
Codename:	noble

$ psql --version
psql (PostgreSQL) 17.0 (Ubuntu 17.0-1.pgdg24.04+1)

[TheBlueMatt]

Should be fixed by #105