tls: coalesce buffered records into one read

[ronag]

SSL_read() returns at most one TLS record per call, and TLSWrap::ClearOut() emits a separate read for each one. Records can be up to 16 KiB, but plenty of servers send smaller ones: nginx defaults ssl_buffer_size to 4 KiB, and servers that do dynamic record sizing start out at around 1 KiB. Every record then pays for its own buffer allocation and call into JS.

With this change ClearOut() keeps calling SSL_read() into the same 16 KiB buffer while the next record in enc_in_ is application data. It never waits for more input, it only merges records that have already arrived, and reads never get bigger than one full record already makes them.

Some notes:

• Batching stops in front of any other record type, and in front of application data records that are short enough to be empty, since SSL_read() skips empty records and goes straight on to the next one. In TLS 1.2 that means handshake and alert records are still processed after the preceding data has been emitted, so a 'data' handler can e.g. still call disableRenegotiation() before a renegotiation that arrived in the same read. test-tls-coalesce-renegotiation covers that, and fails if the record type check is removed.
• When SSL_read() fails in the middle of a batch, SSL_get_error() and the error are still read right away, as before. The batched data is emitted next, then the error or EOF.
• TLS 1.3 encrypts post-handshake messages, so a NewSessionTicket or KeyUpdate can't be told apart from data by its record header. If one arrives together with data, 'session' or 'keylog' can now be emitted before the 'data' that came before it. There is no documented ordering between those events and 'data'. Keeping it would mean emitting data from inside OpenSSL callbacks, with the reentrancy problems that brings, so I didn't.
• 'data' chunks no longer line up with records, so test-tls-max-send-fragment now checks the record sizes on the wire through a small proxy.
• I also tried a 64 KiB buffer, which lets full 16 KiB records merge too. It was only 2-5 points faster than 16 KiB in the runs below, and a paused stream then buffers up to one extra 64 KiB chunk past its high water mark (pummel/test-https-no-reader fails), so I kept 16 KiB.

benchmark/tls/throughput-s2c.js gets a recordlen option, which sets the server's setMaxSendFragment().

Results on an AMD EPYC 9355P (Linux, Docker, 16 CPUs on one NUMA node, benchmark pinned to one CPU), 10 runs each, MiB/s:

recordlen	recvbuflen	before	after
1024	0	667 ± 10	945 ± 7	+41.6% ***
4096	0	1456 ± 41	1833 ± 17	+25.9% ***
16384	0	2271 ± 63	2235 ± 66	-1.6%
1024	65536	913 ± 14	1083 ± 15	+18.6% ***
4096	65536	2011 ± 16	2127 ± 13	+5.8% ***
16384	65536	2826 ± 10	2816 ± 11	-0.4%

$ node benchmark/compare.js --old ./node-base --new ./node-new --runs 10 \
    --filter throughput-s2c --set CPUSET=21 --set dur=3 --set type=buf \
    --set sendchunklen=16777216 --set recvbufgenfn=false \
    --set recvbuflen=0 --set recvbuflen=65536 --analyze tls

[nodejs-github-bot]

Review requested:

• @nodejs/crypto

[Copilot]

🟡 Changes recommended

Additional SSL reads can let renegotiation callbacks overtake preceding application data.

1 open finding

• TLS renegotiation callback can overtake buffered application data · New

What changed in this PR

Coalesces buffered TLS records into 16 KiB reads to reduce per-record overhead.

Changes:

• Batches decrypted records while preserving EOF and TLS error handling.
• Adds coalescing, small-buffer, EOF, and corruption tests.
• Updates fragment-size validation to inspect wire records.

File	Description
src/​crypto/​crypto_tls.cc	Implements record coalescing and deferred error emission.
src/​crypto/​crypto_tls.h	Declares extracted TLS error helpers.
test/​parallel/​test-tls-coalesce-records.js	Adds coalescing regressions.
test/​parallel/​test-tls-max-send-fragment.js	Validates fragment sizes on the wire.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

TLS 1.3 protocol records can trigger callbacks before preceding application data is emitted.

1 open finding

• TLS 1.3 control records can overtake buffered application data · New

1 resolved since last review

• TLS renegotiation callback can overtake buffered application data

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

The implementation raises the read cap to 64 KiB despite the stated requirement to preserve 16 KiB behavior.

1 open finding

• Restore the 16 KiB chunk limit or justify the 64 KiB increase · New

2 resolved since last review

• TLS 1.3 control records can overtake buffered application data
• TLS renegotiation callback can overtake buffered application data

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

A data callback can synchronously destroy the SSL object while an OpenSSL callback is still using it.

2 open findings

• Defer SSL destruction during callbacks to prevent use-after-free · New
• Restore the 16 KiB chunk limit or justify the 64 KiB increase

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

Native TLS callback and lifetime changes require human review and completion of the stated runtime validation.

1 open finding

• Restore the 16 KiB chunk limit or justify the 64 KiB increase

1 resolved since last review

• Defer SSL destruction during callbacks to prevent use-after-free

🧠 Review effort: Balanced

[codecov]

Codecov Report

❌ Patch coverage is 92.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.49%. Comparing base (c883ccf) to head (7ae2c5a).
⚠️ Report is 74 commits behind head on main.

Files with missing lines	Patch %	Lines
src/crypto/crypto_tls.cc	92.00%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #66698      +/-   ##
==========================================
- Coverage   92.82%   90.49%   -2.34%     
==========================================
  Files         422      797     +375     
  Lines      194189   277791   +83602     
  Branches    30039    53367   +23328     
==========================================
+ Hits       180257   251382   +71125     
- Misses      13607    16797    +3190     
- Partials      325     9612    +9287     

Files with missing lines	Coverage Δ
src/crypto/crypto_tls.cc	78.50% <92.00%> (ø)

... and 528 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

🔵 Needs a closer look

An unresolved empty-record ordering issue and changes to TLS callback behavior require human review and CI validation.

1 open issue

• Empty TLS record bypasses renegotiation guard before payload handler · New

1 resolved since last review

• Restore the 16 KiB chunk limit or justify the 64 KiB increase

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The corruption regression does not exercise error handling while plaintext remains buffered.

1 open issue

• Corruption test fails to cover pending plaintext flush · New

1 resolved since last review

• Empty TLS record bypasses renegotiation guard before payload handler

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Core TLS buffering and callback-ordering changes require final human review and CI validation.

No open issues 🎉

1 resolved since last review

• Corruption test fails to cover pending plaintext flush

_{Review effort: Balanced}

[mcollina]

There is a relevant CI failure