QUIC performance limited to 150 MB/s on my use case on localhost

[Pierre Chatelier (PierreChatelier)]

I am new to msquic and try to test its performance for a very simple streaming scenario "unidirectional server -> client" on localhost.

I have built my own msquic (2.6.0), with Visual Studio under Windows, with openssl as TLS lib.

I have a (custom, C++) server that queues 5MB payloads as fast as possible, with a single stream for each payload, sequentially but overlapping (closing streams on SEND_COMPLETE, don't wait for previous stream to start a new one, but with a custom, fixed, limit on the number of active streams)
I have a (custom, C++) client that efficiently receives the payloads.

I can reach 150 MB/s measured in the client, which would be OK on a 1GB/s network, but terribly slow on "localhost".
I don't understand what is limiting me.

I have tuned many things already.

[server]

  constexpr const uint32_t kb = 1U*1024;
  constexpr const uint32_t mb = 1U*1024*kb;

  QUIC_SETTINGS Settings = { 0 };
  Settings.StreamRecvWindowUnidiDefault = 64*mb;
  Settings.IsSet.StreamRecvWindowUnidiDefault = TRUE;
  /*Settings.StreamRecvWindowBidiLocalDefault = 64*mb;
  Settings.IsSet.StreamRecvWindowBidiLocalDefault = TRUE;
  Settings.StreamRecvWindowBidiRemoteDefault= 64*mb;
  Settings.IsSet.StreamRecvWindowBidiRemoteDefault = TRUE;*/
  Settings.PeerUnidiStreamCount = exnative_data_t::maxStreamsCount;//1
  Settings.IsSet.PeerUnidiStreamCount = TRUE;
  Settings.ConnFlowControlWindow = Settings.PeerUnidiStreamCount*Settings.StreamRecvWindowUnidiDefault;
  Settings.IsSet.ConnFlowControlWindow = TRUE;

  Settings.IdleTimeoutMs = 1000;
  Settings.IsSet.IdleTimeoutMs = TRUE;

  Settings.ServerResumptionLevel = QUIC_SERVER_RESUME_AND_ZERORTT;
  Settings.IsSet.ServerResumptionLevel = TRUE;

  Settings.PacingEnabled = FALSE;
  Settings.IsSet.PacingEnabled = TRUE;

  Settings.SendBufferingEnabled = FALSE;
  Settings.IsSet.SendBufferingEnabled = TRUE;

  Settings.MaxAckDelayMs = 1;
  Settings.IsSet.MaxAckDelayMs = TRUE;

  Settings.MaximumMtu = 65527; // Max UDP payload size within a 65535 IP packet
  Settings.IsSet.MaximumMtu = TRUE;

  Settings.EcnEnabled = FALSE;
  Settings.IsSet.EcnEnabled = TRUE;

[Client]

  constexpr const size_t buffersCapacity = 1;

  constexpr const size_t kb = 1U*1024;
  constexpr const size_t mb = 1U*1024*kb;

  QUIC_SETTINGS Settings = { 0 };
  Settings.StreamRecvWindowUnidiDefault = 64*mb;
  Settings.IsSet.StreamRecvWindowUnidiDefault = TRUE;
  Settings.PeerUnidiStreamCount = buffersCapacity;
  Settings.IsSet.PeerUnidiStreamCount = TRUE;
  Settings.ConnFlowControlWindow = static_cast<unsigned __int32>(
    std::min((size_t)std::numeric_limits<unsigned __int32>::max(),
             (size_t)Settings.PeerUnidiStreamCount*(size_t)Settings.StreamRecvWindowUnidiDefault));
  Settings.IsSet.ConnFlowControlWindow = TRUE;

  Settings.StreamRecvWindowDefault = Settings.StreamRecvWindowUnidiDefault;
  Settings.IsSet.StreamRecvWindowDefault = TRUE;
  
  Settings.PacingEnabled = FALSE;
  Settings.IsSet.PacingEnabled = TRUE;

  Settings.IdleTimeoutMs = 10'000;
  Settings.IsSet.IdleTimeoutMs = TRUE;

  Settings.MaxAckDelayMs = 1;
  Settings.IsSet.MaxAckDelayMs = TRUE;

  Settings.MaximumMtu = 65527; // Max UDP payload size within a 65535 IP packet
  Settings.IsSet.MaximumMtu = TRUE;

  Settings.InitialWindowPackets = 10000;
  Settings.IsSet.InitialWindowPackets = TRUE;

I also (correctly) use flags such as QUIC_STREAM_START_FLAG_IMMEDIATE, QUIC_SEND_FLAG_ALLOW_0_RTT, QUIC_SEND_FLAG_FIN

I think that I am an order of magnitude below what I should expect.
Looks like that 150 MB/s limit must be related to some behaviour of msQUIC that could be tuned about congestion or some kernel events rate policy.

• my server QUIC_BUFFERs are reused (no allocation) since I have a fixed payload size
• my msquic callbacks are heavily optimized to be lightweight and do not even allocate memory ; they use fast locks with very reduced contention (I even tried a client without any useful work, just receiving buffers and counting bytes)
• Disabling encryption has no measurable effect
• Trying to use more than 1 PeerUnidiStreamCount degrades performance
• Trying to use a persistent stream for all payloads (no QUIC_SEND_FLAG_FIN, stream reuse) degrades performance

I would like to use one of the msquic tools to get a local perf test, but I can't find any relevant documentation for secnetperf, for instance. The link "For performance data, see the Performance dashboard." is broken

What should be my next step ?

[Pierre Chatelier (PierreChatelier)]

I have finally found https://microsoft.github.io/msquic/msquicdocs/src/perf/readme.html?q=secnetperf

And I don't known what to think.

I start the server :
>secnetperf -bind:127.0.0.1 -port:4567 -exec:lowlat -stats:1
or
>secnetperf -bind:127.0.0.1 -port:4567 -exec:maxtput -stats:1

Then the client:

>secnetperf -target:127.0.0.1 -down:1gb -ptput:1
Started!

Result: Download 3505446 kbps. //for lowlat
Result: Download 4381674 kbps.//for maxtput
App Main returning status 0

So the bandwidth (in lowlat) seems to be 3505446 kbps.... close to 430 MB/s, which is something I can observe with raw TCP on my localhost with the same server/client architecture and payloads.

So definitely, there's something wrong in my QUIC settings.

[Pierre Chatelier (PierreChatelier)]

By inspecting the source code of secnetperf, I saw that I could use CXPLAT_ALLOC_NONPAGED to allocate the QUIC_BUFFERs.
I tried, but without any positive impact on performance

[Pierre Chatelier (PierreChatelier)]

Just by changing the server from QUIC_EXECUTION_PROFILE_LOW_LATENCY to QUIC_EXECUTION_PROFILE_TYPE_MAX_THROUGHPUT , I get 250 MB/s instead of 150 MB/s.

That's better, but still far below the expectations.

[Pierre Chatelier (PierreChatelier)]

Looks similar to #3926 but there were no final words

[Pierre Chatelier (PierreChatelier)]

I have implemented an alternative test server to deliver dummy data.

With my real server (from a custom software), one 5MB QUIC_BUFFER needs 40ms between StreamSend() and QUIC_STREAM_EVENT_SEND_COMPLETE (and I get a limited bandwith)
With my test server (dummy data, no other task than sending QUIC data), one 5MB QUIC_BUFFER needs 12ms between StreamSend() and QUIC_STREAM_EVENT_SEND_COMPLETE (and I get the expected bandwidth)

Both servers share the same optimized C++ implementation and msquic integration.

They are configured the same relatively to msQUIC settings.
The real server is very sensistive to QUIC_EXECUTION_PROFILE_LOW_LATENCY (slow) vs QUIC_EXECUTION_PROFILE_TYPE_MAX_THROUGHPUT (faster)
The test server performs the same whatever the QUIC_EXECUTION_PROFILE

So now I think it might have something to do with CPU occupation, because my old server is heavily threaded and does a lot of work in parallel to produce the data to be sent.

However the overall CPU usage is still rather low

Could it be something to investigate more ? Is there useful knownledge about msquic and processor affinity ?

[Pierre Chatelier (PierreChatelier)]

I have played a little wit hQUIC_GLOBAL_EXECUTION_CONFIG on the server side, but it does not change the performance of the real and test servers.

[Pierre Chatelier (PierreChatelier)]

I try to generate a trace https://github.com/microsoft/msquic/blob/main/src/plugins/trace/README.md

I have downloaded WPA, the plugin
I have installed the plugin

I start the trace with

netsh trace start provider="{ff15e657-4f26-570e-88ab-0796b258d11c}" level=5 keywords=0xffffffffffffffff bufferSize=1024 maxsize=1024 tracefile=msquic.etl overwrite=yes

After loading msquic.etl in WPA+msquic plugin, I just get an empty result

[Pierre Chatelier (PierreChatelier)]

Ok, this is not in https://github.com/microsoft/msquic/blob/main/src/plugins/trace/README.md but msQUIC must be built with QUIC_ENABLE_LOGGING and QUIC_LOGGING_TYPE=etw

Now I get events in WPA

[Guillaume Hetier (guhetier)]

Hi, I would consider trying the following, as generic advice:

• keep most settings as default unless you can see it a different value is clearly an improvement. Some, like the ack delay, can backfire by causing more traffic
• now that you enabled ETW, look in WPA at what are the limiting factor for your connection (congestion control / CPU bound / app processing done)
• make sure the app keep the processing in MsQuic callbacks to a minimum. Those callbacks run in the workers, which can impact the delay before an ack and the congestion control limits.

If you think CPU utilization is a problem, collecting CPU traces using WPR/WPA can help to see if msquic workers get to run as you expect. Providing a CPU map in the execution options can help.

[Pierre Chatelier (PierreChatelier)]

Now that I am able to generate traces, I also have to learn how to analyze them (I am not familiar with WPA at all).

Will you help me analyzing if I attach traces in this discussion, or is it not appropriate ?

[Guillaume Hetier (guhetier)]

On localhost specifically, there also can be some side effects due to how we try to keep MsQuic connection processing aligned with RSS cores - since Windows will often loopback the traffic directly on the core that sends it. You can check if the server and client side are sharing a worker or are spread between two different workers.

[Guillaume Hetier (guhetier)]

Will you help me analyzing if I attach traces in this discussion, or is it not appropriate ?

I won't be able to help more for now, the team is currently very busy. The documentation should give some insights about what to look for at first. The MsQuic plugin should help identifying limiting factors in your connection.

[Pierre Chatelier (PierreChatelier)]

I think I solved my problem !

Because of subtle differences in the toolchain, my "real server" was not deploying the same msquic.dll as my "test server".
The "test server" was picking the msquic.dll release build (as expected)
The "real server" was picking the msquic.dll from "CMAKE_INSTALL_PREFIX" folder, fed by the INSTALL step of the Visual Studio project.

But since both Debug and Release builds of msquic.dll share the same name (instead of some "msquicd.dll" for the debug), the INSTALL step of the Debug build was overwriting the DLL of the INSTALL step of the Release.

Indeed, when using Visual Studio to build several configurations, Debug is compiled after Release

So ultimately, my whole problem was that the debug build of msquic is not using a "d" suffix.