Fix data race in SSLSocketStream causing TLS session corruption in WebSocket - #2550
Fix data race in SSLSocketStream causing TLS session corruption in WebSocket#2550Hyukya wants to merge 2 commits into
Conversation
Protect TLS session operations in SSLSocketStream with a mutex. This prevents races when WebSocket send and receive paths concurrently access the same TLS session, including pending checks, peer-close detection, reads, and writes.
|
@Hyukya thanks for digging into this, nice repro and good tests, this is a real bug. I ended up fixing it a bit differently in #2551: instead of locking inside SSLSocketStream (which every HTTPS request goes through), wss:// WebSocket connections now get their own stream class, so plain HTTP/HTTPS isn't touched at all. It also keeps the socket non-blocking so the lock is never held while waiting on the network, so a stalled peer can't block a concurrent send()/ping for the whole read timeout. Going to close this in favor of #2551, but your report and tests did the hard part here. Thanks again! |
|
I got so focused on fixing the WebSocket synchronization that I accidentally put a lock on all HTTPS traffic. I completely agree that regular HTTP/HTTPS shouldn't be affected at all, so switching to a dedicated stream class is definitely the better approach. Great work on this! |
Problem
SSLSocketStreamcan be shared across threads, allowing concurrent access to the same TLS session.The WebSocket ping thread, the application's
send(), andclose()'s wait-for-response read all enter the same session.The write path also reads (
wait_writable() -> is_peer_closed() -> SSL_peek), soSSL_readandSSL_peekcollide on the same record layer buffer, causing a buffer overflow.This only affects
wss://.ws://doesn't use a TLS session, so it's unaffected.Reproduction
Added
test_websocket_thread_safety.cc. One thread loopsread()while another callssend()/close().CloseWhileAnotherThreadReads(macOS, ASan): heap-buffer-overflow viaWebSocketClient::close() -> read_websocket_frame() -> SSLSocketStream::read() -> SSL_read()(during OpenSSL GCM cipher param handling).SendWhileAnotherThreadReads(macOS/Linux):sentfalls far short of the expected 2000 (macOS: 7, Linux: 19).On Linux,
frames_read == 0also failed.No crash — messages are silently dropped.
Being a race, the exact failure point varies per run; a single pass doesn't
prove safety.
Fix
Add one mutex per
SSLSocketStream, held around every call that enters thesession:
tls::pending,tls::read,tls::write,tls::is_peer_closed.select_read/select_writestay outside the lock since they're socketoperations, not session operations. Locking is not applied at the WebSocket
layer, so an idle reader doesn't block a sender for the whole read timeout.
Out of scope
I/O.
std::mutexdoesn't guarantee fairness, so send starvation underheavy read load is still possible. A single I/O owner with an outbound
queue would be a better long-term structure.
identical internal corruption on other backends.