Fix deadlock with many peers, few threads, and slow postgres writes

When we have many peers but relatively few threads, and postgres
writes fall behind enough to block the async stream of gossip, we
can end up deadlocking on the `PeerManager` `peers` thread.
Specifically, because `PeerManager::process_events` holds a
`peers` (read) lock while fetching messages (during which time we
`block_in_place` trying to push gossip to the async stream), any
other tasks which try to touch the `peers` lock in write mode (e.g.
connecting to a peer or processing a disconnection) we'll hang
until some postgres writes finish. If we get enough threads
blocked, we end up blocking the tokio reactor and the postgres
writes will stall, fully deadlocking.

There's not a great solution to this, instead we just have to
ensure we have enough threads which we enforce.

Author: TheBlueMatt

src/config.rs

@@ -362,7 +362,7 @@ pub(crate) async fn upgrade_db<L: Deref + Clone + Send + Sync + 'static>(
 	let _ = client.execute("ALTER TABLE channel_announcements SET ( autovacuum_vacuum_insert_scale_factor = 0.005 );", &[]).await;
 }
 
-pub(crate) fn ln_peers() -> Vec<(PublicKey, SocketAddr)> {
+pub fn ln_peers() -> Vec<(PublicKey, SocketAddr)> {
 	const WALLET_OF_SATOSHI: &str = "035e4ff418fc8b5554c5d9eea66396c227bd429a3251c8cbc711002ba215bfc226@170.75.163.209:9735";
 	let list = env::var("LN_PEERS").unwrap_or(WALLET_OF_SATOSHI.to_string());
 	let mut peers = Vec::new();

src/lib.rs

@@ -37,10 +37,10 @@ mod lookup;
 mod persistence;
 mod serialization;
 mod snapshot;
-mod config;
 mod hex_utils;
 mod verifier;
 
+pub mod config;
 pub mod types;
 
 #[cfg(test)]

src/main.rs

@@ -1,9 +1,28 @@
 use std::sync::Arc;
+
 use rapid_gossip_sync_server::RapidSyncProcessor;
+use rapid_gossip_sync_server::config;
 use rapid_gossip_sync_server::types::RGSSLogger;
 
-#[tokio::main]
-async fn main() {
-	let logger = Arc::new(RGSSLogger::new());
-	RapidSyncProcessor::new(logger).start_sync().await;
+use tokio::runtime::Builder;
+
+fn main() {
+	Builder::new_multi_thread()
+		.enable_all()
+		// Ensure we have a nontrivial number of threads per peer. This works around a deadlock
+		// where we have one thread doing `PeerManager::process_events` which does a
+		// `block_in_place` with the `PeerManager` peers lock held (to push gossip into the gossip
+		// persistence queue, if its full). Once this happens, if any peer operations happen for
+		// new connections to peers or peer disconnections, they may wait on the `PeerManager`
+		// peers mutex. If there aren't enough threads for at least one to be blocked on the
+		// `PeerManger` peers mutex per (configured) peer, we may just end up hanging.
+		// There isn't really a great way to fix it, so instead we just make sure we have plenty of
+		// threads.
+		.worker_threads(config::ln_peers().len() * 2 + 4)
+		.build()
+		.unwrap()
+		.block_on(async {
+			let logger = Arc::new(RGSSLogger::new());
+			RapidSyncProcessor::new(logger).start_sync().await;
+		});
 }