Add QUIC_PARAM_CONN_PATH_STATISTICS for per-path network statistics - #82
Conversation
QUIC_PARAM_CONN_NETWORK_STATISTICS only ever reports Paths[0], which on a multipath connection leaves every other path unobservable. The new parameter returns one QUIC_PATH_STATISTICS per path: the path's Rtt, MinRtt, MaxRtt and Mtu, plus the QUIC_NETWORK_STATISTICS read from that path's own congestion control. The path count is not known ahead of time and changes over the connection's life, so it is retrieved the usual two-step way: a BufferLength of 0 reports the size needed, and on success BufferLength is the number of bytes written. Paths with no path ID assigned yet -- added before the handshake is confirmed -- are not reported, having nothing to identify them by and no congestion control to read. Works with or without multipath negotiated. PathId is carried in each entry because array position is not stable: removing a path moves the ones behind it up, so a caller cannot otherwise tell which entry belongs to which path across calls. It matches the PathId used by QUIC_PARAM_CONN_PATH_STATUS. MinRtt and MaxRtt are reported as zero until the path has an RTT sample. QuicPathInitialize uses MinRtt = UINT32_MAX as its "no sample yet" sentinel and leaves MaxRtt at zero; passing that through would read as a minimum RTT of roughly 4295 seconds. This differs from QUIC_STATISTICS_V2, which passes the sentinel through as-is. Both congestion control implementations of the network statistics hook read the RTT off Connection->Paths[0] rather than the path the instance belongs to, so every path would have reported path 0's RTT -- and, in cubic, path 0's bandwidth, which is derived from it. Both now take the path from the path ID that owns the instance, the way the rest of both files already reach it. Single path connections are unaffected. Basic/WithFamilyArgs.PathStatistics brings up a second path and checks the array grows by one, that the entries carry distinct path IDs and their own figures, and that a short buffer is refused with the required size. Asserting each entry's Rtt against its own NetworkStatistics.SmoothedRTT is what pins the congestion control correction: reverting the cubic change alone fails it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
| // Congestion control is per path ID, so the RTT to report is the one of the | ||
| // path this instance belongs to, not whichever path happens to be first. | ||
| // | ||
| const QUIC_PATH* Path = QuicCongestionControlGetPathID(Cc)->Path; |
There was a problem hiding this comment.
QuicCongestionControlGetPathID(Cc)->Path is not a reliable pointer to the path this congestion control is currently driving when multipath is not negotiated, so this swaps a always-valid pointer for one that can be stale.
QuicPathSetActive() (src/core/path.c, non-multipath branch) swaps the contents of Connection->Paths[0] and *Path and never fixes up PathID->Path. QuicPathRemove() only re-points PathID->Path inside if (Connection->State.MultipathNegotiated).
Concrete scenario (server, multipath off, client migrates / NAT rebind):
QuicConnGetPathForPacket()creates the new path atPaths[1]for the existing path ID and setsPathID->Path = &Paths[1].- The migration is accepted and
QuicPathSetActive(Connection, &Paths[1])swaps the two slots — the active path is nowPaths[0], butPathID->Pathstill points atPaths[1], which now holds the old path. QUIC_PARAM_CONN_NETWORK_STATISTICSand theQUIC_CONNECTION_EVENT_NETWORK_STATISTICSevent now report the old path'sSmoothedRtt. Before this change they readConnection->Paths[0], which is always the active path.- Once the old path is removed,
PathID->Pathrefers to a slot pastPathsCountholding stale data, so the reported RTT is whatever was left there.
Multipath connections are fine (the back-references are maintained there), which is why the new test passes. Consider having the caller supply the path (e.g. pass the QUIC_PATH* into the hook, or fill SmoothedRTT in QuicConnGetPathStatistics/QuicConnGetNetworkStatistics from the path being iterated) rather than depending on the PathID->Path back-pointer.
| // Congestion control is per path ID, so the RTT to report is the one of the | ||
| // path this instance belongs to, not whichever path happens to be first. | ||
| // | ||
| const QUIC_PATH* Path = QuicCongestionControlGetPathID(Cc)->Path; |
There was a problem hiding this comment.
Same issue as in cubic.c: QuicCongestionControlGetPathID(Cc)->Path is not guaranteed to be the path this CC is driving when multipath is not negotiated.
QuicPathSetActive() (src/core/path.c) swaps Connection->Paths[0] with *Path in the non-multipath branch without updating PathID->Path, and QuicPathRemove() only repairs the back-references under if (Connection->State.MultipathNegotiated).
Scenario: non-multipath server connection, client migrates. QuicConnGetPathForPacket() sets PathID->Path = &Paths[1]; QuicPathSetActive() then swaps Paths[0]/Paths[1], leaving PathID->Path on the now-inactive old path. BbrCongestionControlGetNetworkStatistics (used by both QUIC_PARAM_CONN_NETWORK_STATISTICS and the QUIC_CONNECTION_EVENT_NETWORK_STATISTICS indication) then reports that stale path's SmoothedRtt; the previous &Connection->Paths[0] was always the active path. After the old path is removed the pointer refers to a slot beyond PathsCount and the RTT reported is leftover data.
| // the handshake is confirmed has neither yet. | ||
| // | ||
| uint8_t PathCount = 0; | ||
| for (uint8_t i = 0; i < Connection->PathsCount; ++i) { |
There was a problem hiding this comment.
This counts/emits one entry per QUIC_PATH, but a QUIC_PATHID is not one-to-one with a QUIC_PATH: two paths can hold the same PathID pointer at the same time.
QuicConnGetPathForPacket() (src/core/path.c) handles a rebind/migration by creating a new QUIC_PATH at Paths[1] for an existing path ID (QuicPathIDAddRef(PathID, QUIC_PATHID_REF_PATH); Path->PathID = PathID; PathID->Path = Path;) while the old path, still carrying the same PathID pointer, remains in the array until it is validated away.
In that window QUIC_PARAM_CONN_PATH_STATISTICS returns two entries with the same PathId (0 on a non-multipath connection, where every path shares path ID 0) and byte-identical NetworkStatistics, since both read the one PathID->CongestionControl. That directly contradicts the documented contract added in docs/Settings.md ("PathId identifies which path an entry describes") and leaves the caller unable to tell the two apart.
Related: PathStats->Rtt is read from Paths[i].SmoothedRtt while PathStats->NetworkStatistics.SmoothedRTT is read (via the CC hook) from PathID->Path->SmoothedRtt. Whenever those are not the same object — the duplicate case above, or the non-multipath QuicPathSetActive() swap — the two fields of a single entry disagree, breaking the TEST_EQUAL(PathStats[i].Rtt, PathStats[i].NetworkStatistics.SmoothedRTT) invariant the new test relies on.
| | `QUIC_PARAM_CONN_REMOVE_CANDIDATE_ADDRESS` <br> 34| QUIC_CANDIDATE_ADDRESS | Set-only | Remove a candidate address. Client only. | | ||
|
|
||
| | `QUIC_PARAM_CONN_UNCONNECTED_UDP_SOCKET` <br> 37 | uint8_t (BOOLEAN) | Both | Set on client only. Must be set before start, and requires `QUIC_PARAM_CONN_SHARE_UDP_BINDING`. See [QUIC_PARAM_CONN_UNCONNECTED_UDP_SOCKET](#quic_param_conn_unconnected_udp_socket). | | ||
| | `QUIC_PARAM_CONN_PATH_STATISTICS` <br> 38 | QUIC_PATH_STATISTICS[] | Get-only | Network statistics for every path at once, one array entry per path. See [QUIC_PARAM_CONN_PATH_STATISTICS](#quic_param_conn_path_statistics). | |
There was a problem hiding this comment.
Rendering nit, but it makes the new row invisible in the docs: line 225 is blank, which terminates the parameter table above. A block of |-delimited lines with no header/delimiter row is not a GFM table, so this row (and the QUIC_PARAM_CONN_UNCONNECTED_UDP_SOCKET row above it, which has the same pre-existing problem) renders as literal | ... | text rather than as table rows.
Moving both rows up so they are contiguous with the table that ends at QUIC_PARAM_CONN_REMOVE_CANDIDATE_ADDRESS (deleting the blank line at 225) fixes it.
Review of #82 found that deriving the path from the congestion control's owning path ID regresses QUIC_PARAM_CONN_NETWORK_STATISTICS on connections without multipath. QuicPathSetActive swaps the *contents* of Paths[0] and the promoted path: QUIC_PATH PrevActivePath = Connection->Paths[0]; Connection->Paths[0] = *Path; *Path = PrevActivePath; The PathID pointer travels with the contents, but the path ID's own back reference is left behind, and QuicPathRemove only repairs back references under if (MultipathNegotiated). So after a migration on a non-multipath connection, PathID->Path points at the slot the old path was moved into, and the statistics would have come from the inactive path -- and, once that path was removed, from a slot past PathsCount holding stale data. The path is now a parameter of the hook, so no caller depends on that back reference. QuicConnGetNetworkStatistics and the BBR connection event pass &Connection->Paths[0], which is the expression the hook used internally before, so both keep their existing behaviour exactly. QuicConnGetPathStatistics passes the path it is reporting on, which also removes the possibility of an entry's Rtt disagreeing with its own NetworkStatistics.SmoothedRTT. Review also found that a rebind leaves two paths sharing one path ID for a while -- QuicConnGetPathForPacket assigns the path ID to the new path without clearing it from the old one -- so PathId is not unique in the array while that lasts. Both entries are real paths and the shared congestion control makes their NetworkStatistics agree; documented rather than filtered, since dropping one would hide a path that exists. The parameter table in docs/Settings.md had a blank line in the middle, which terminated it early and rendered the last rows as literal text. Removed; this also fixes the QUIC_PARAM_CONN_UNCONNECTED_UDP_SOCKET row added earlier. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
All four findings were real. Verified each against the source before acting; fixed in ac05aa7. The cubic/bbr regression (findings 1 and 2). Confirmed. QUIC_PATH PrevActivePath = Connection->Paths[0];
Connection->Paths[0] = *Path;
*Path = PrevActivePath;The Rather than reinstate the hardcoded index, the path is now a parameter of the hook, so no caller depends on that back reference at all. The unit tests in Duplicate Documented rather than filtered. Both entries are real paths, and since they share one path ID they share its congestion control, so their The docs table (finding 4). Confirmed — a blank line after the Verification after the fix. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Brings in #82 (QUIC_PARAM_CONN_PATH_STATISTICS). One conflict: src/test/bin/quic_gtest.cpp Both sides added a TEST_P after PathKeepAlive -- qmux-01 the three QMux tests, seera-main the PathStatistics one. Kept both, PathStatistics first so it stays next to the other path tests. QMux connections are unaffected by the new parameter. QuicConnQMuxAlloc never initializes Paths[], so PathsCount is zero and the new QuicConnGetPathStatistics iterates nothing: it reports zero entries rather than reaching for a path ID or its congestion control. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Description
QUIC_PARAM_CONN_NETWORK_STATISTICSonly ever reports the first path:On a multipath connection that leaves the other paths unobservable.
QUIC_PARAM_CONN_PATH_STATISTICSreturns one entry per path instead.The path count is not known ahead of time and changes over the connection's life, so it is retrieved the usual two-step way: a
BufferLengthof0reports the size needed, and on successBufferLengthis the number of bytes written, giving the entry count. Paths with no path ID assigned yet — added before the handshake is confirmed — are not reported, having nothing to identify them by and no congestion control to read. Works with or without multipath negotiated; a single-path connection returns one entry.Two things beyond the field list
PathIdis included. Array position is not stable — removing a path moves the ones behind it up — so without an identifier the caller cannot tell which entry belongs to which path across calls. It matches thePathIdused byQUIC_PARAM_CONN_PATH_STATUS.It is not unique, however.
QuicConnGetPathForPacketassigns a path ID to a rebound path without clearing it from the path being replaced, so during a rebind two entries carry the samePathId. Both are real paths, and sharing a path ID means sharing its congestion control, so theirNetworkStatisticsagreeing is correct rather than a defect; the per-pathRtt,MinRtt,MaxRttandMtuare what tell them apart. Documented rather than filtered — suppressing one would hide a path that exists from an API whose purpose is to report all of them.MinRtt/MaxRttare normalised.QuicPathInitializesetsMinRtt = UINT32_MAXas its "no sample yet" sentinel and leavesMaxRttat zero. Passing that through would report a minimum RTT of roughly 4295 seconds, so a path with nothing measured reports zero for both.Rttneeds no such treatment: it starts from the configuredInitialRttMs.This differs from
QUIC_STATISTICS_V2, which passes the sentinel through as-is.The network statistics hook now takes a path
Both implementations read the RTT off
Connection->Paths[0]:Congestion control is per path ID, so every path would have reported path 0's RTT — and, in cubic, path 0's bandwidth, which is derived from it.
Deriving the path from the congestion control's owning path ID does not work, because that back reference is not always current.
QuicPathSetActive's non-multipath branch swaps the contents of the two slots:The
PathIDpointer travels with the contents while the path ID's back reference stays behind, andQuicPathRemoveonly repairs back references underif (MultipathNegotiated). After a migration without multipath the back reference points at the slot the old path was moved into — and, once that path is removed, at a slot pastPathsCountholding stale data.So the path is a parameter of the hook instead, and no caller depends on the back reference:
QuicConnGetNetworkStatisticsand the BBR connection event pass&Connection->Paths[0], the expression the hook used internally before, so both keep their behaviour exactly.QuicConnGetPathStatisticspasses the path it is reporting on, which also rules out an entry'sRttdisagreeing with its ownNetworkStatistics.SmoothedRTT.BbrTest.cppandCubicTest.cppare updated for the signature.Testing
New
Basic/WithFamilyArgs.PathStatistics, registered inMsQuicTests.h,quic_gtest.cppandwinkernel/control.cpp. It queries a one-path connection, brings up a second path, and checks the array grows by exactly one entry; that both entries carry distinct path IDs, a non-zero MTU and RTT, and aMinRttno greater thanMaxRtt; that a buffer one entry short is refused with the required size; and that each entry'sRttequals its ownNetworkStatistics.SmoothedRTT.That last assertion is what pins the per-path plumbing. Instrumented, the two paths report genuinely different figures:
Pointing
QuicConnGetPathStatisticsback atPaths[0]turnspath[1]'snsRttinto path 0's and fails the test.PathStatistics× 3 repeats*Multipath*:*Path*:*Migration*:*UnconnectedSocket*:*KeepAlive*:*Statistics**Basic*:*Datagram*:*Receive*:*Recv*msquiccoretest *Bbr*:*Cubic*cargo test --features preview-apiBuild is clean.
connection.c,cubic.candbbr.cwere run through the CI's clang-tidy 21 inghcr.io/microsoft/msquic/linux-build-xcomp:ubuntu-26.04-cross, since the local one is too old to reproduce what-CodeChecksees.Not covered by a test: the non-multipath migration case above. It is the reason the hook signature changed, but reaching it needs a connection to migrate and then have the old path removed, which the existing path tests do not set up. The two connection-level callers pass the same expression the hook used before, so the change is behaviour-preserving there by construction rather than by test.
Documentation
docs/Settings.mdgains the table row and a section covering the two-step sizing, whatPathIdis for and why it is not unique, the zero-until-sampled RTTs, and the fact that paths without a path ID yet are not reported.The parameter table had a blank line in the middle, which terminated it early and left the last rows rendering as literal
| ... |text. Removed, which also repairs theQUIC_PARAM_CONN_UNCONNECTED_UDP_SOCKETrow that had been broken since it was added.Rust bindings are regenerated;
win_bindings.rscannot be regenerated on Linux, so the identical hunk was applied by hand and diffed against the Linux one. The C# bindings are unchanged, as with previous fork settings.