memtier_benchmark aborts when an arbitrary command is a multi-channel pub/sub command
(SUBSCRIBE / PSUBSCRIBE / UNSUBSCRIBE with two or more arguments):
memtier_benchmark: shard_connection.cpp:700: request* shard_connection::pop_req():
Assertion `m_pending_resp.load(std::memory_order_relaxed) >= 0' failed.
Aborted (core dumped)
This is reachable from ordinary CLI usage — no fuzzer, no --monitor-input, and no debug build
required.
Steps to reproduce
Plain release build (autoreconf -ivf && ./configure && make) against a stock local Redis:
$ memtier_benchmark --command="SUBSCRIBE chan1 chan2" \
--server=127.0.0.1 --port=6379 --test-time=2 --clients=1 --threads=1 --hide-histogram
...
memtier_benchmark: shard_connection.cpp:700: request* shard_connection::pop_req(): Assertion `m_pending_resp.load(std::memory_order_relaxed) >= 0' failed.
=== MEMTIER_BENCHMARK BUG REPORT START: Cut & paste starting from here ===
# memtier_benchmark crashed by signal: 6
# Crashed running signal <SIGABRT>
...
# thread=0 client=0 conn=0 addr=127.0.0.1:6379 local_port=48936 state=connected pending=-1 last_cmd=ARBITRARY
Note pending=-1 in the client-list dump.
Equivalently via --monitor-input, with this single line:
[ p ] 1.0 [0 127.0.0.1:1] "SUBSCRIBE" "chan1" "chan2"
Expected: the run completes (or exits with a clean diagnostic).
Actual: SIGABRT.
Which commands trigger it
--command |
result |
SUBSCRIBE chan1 |
clean |
SUBSCRIBE chan1 chan2 |
abort |
SUBSCRIBE a b c |
abort |
PSUBSCRIBE a* b* |
abort |
UNSUBSCRIBE a b |
abort |
GET k1 (control) |
clean |
The cutoff at two arguments is the tell.
Root cause
Redis replies to SUBSCRIBE/PSUBSCRIBE/UNSUBSCRIBE once per channel or pattern, but
shard_connection accounts for exactly one response per request: push_req() increments
m_pending_resp once, and every reply consumed calls pop_req(), which decrements it. With N
channels, N replies arrive for one queued request, so m_pending_resp goes negative and the assert
in pop_req() fires.
Worth noting for severity: pop_req() does
request *req = m_pipeline->front();
m_pipeline->pop();
m_pending_resp.fetch_sub(1, std::memory_order_relaxed);
assert(m_pending_resp.load(std::memory_order_relaxed) >= 0);
so the surplus reply pops a request that was never queued before the assert is evaluated. The
assert catches the accounting error after the fact; built with -DNDEBUG the same path would
instead consume from a possibly-empty pipeline.
Affected versions
Reproduced deterministically (3/3) on both branches:
master @ 272eeb6 — assert at shard_connection.cpp:700
2.4 @ db83a52 (2.4.4) — assert at shard_connection.cpp:658
Not debug-only: the default build carries no -DNDEBUG
(CXXFLAGS = -O2 -g -Wall -Werror=vla -std=c++11), so asserts are live in release builds and in
the published packages.
How it surfaced
The nightly Monitor-Input Fuzz job on the 2.4 matrix leg:
https://github.com/redis/memtier_benchmark/actions/runs/30605076666 (1 failure in 1280 iterations,
seed 17_all_command_types.txt, exit=-6).
A mutation merged corpus lines 61–62 so the SUBSCRIBE line picked up extra arguments. The
pristine seed passes because its SUBSCRIBE has exactly one channel — which is also why the
master leg of the same nightly passed: it is the same bug on master, just not hit by that night's
random mutations. Minimizing the fuzz artifact reduced it to the one line above.
Environment
- Redis server 8.6.0 (
sha=006e6a6a, jemalloc-5.3.0), local, no TLS
- Ubuntu, GCC 13.3.0, libevent 2.1.12-stable, OpenSSL 3.0.13
- Also reproduced under the CI configuration (
--enable-sanitizers --enable-ubsan, GCC 11.4.0,
OpenSSL 3.0.2, ubuntu-22.04)
Suggested direction
Make the reply accounting expect N responses for the multi-argument pub/sub commands
(SUBSCRIBE, PSUBSCRIBE, UNSUBSCRIBE, PUNSUBSCRIBE) — i.e. the number of pending responses a
request contributes should not be hardcoded to 1 — or reject these commands up front with a clear
error if replaying them is out of scope for arbitrary-command mode.
memtier_benchmarkaborts when an arbitrary command is a multi-channel pub/sub command(
SUBSCRIBE/PSUBSCRIBE/UNSUBSCRIBEwith two or more arguments):This is reachable from ordinary CLI usage — no fuzzer, no
--monitor-input, and no debug buildrequired.
Steps to reproduce
Plain release build (
autoreconf -ivf && ./configure && make) against a stock local Redis:Note
pending=-1in the client-list dump.Equivalently via
--monitor-input, with this single line:Expected: the run completes (or exits with a clean diagnostic).
Actual: SIGABRT.
Which commands trigger it
--commandSUBSCRIBE chan1SUBSCRIBE chan1 chan2SUBSCRIBE a b cPSUBSCRIBE a* b*UNSUBSCRIBE a bGET k1(control)The cutoff at two arguments is the tell.
Root cause
Redis replies to
SUBSCRIBE/PSUBSCRIBE/UNSUBSCRIBEonce per channel or pattern, butshard_connectionaccounts for exactly one response per request:push_req()incrementsm_pending_responce, and every reply consumed callspop_req(), which decrements it. With Nchannels, N replies arrive for one queued request, so
m_pending_respgoes negative and the assertin
pop_req()fires.Worth noting for severity:
pop_req()doesso the surplus reply pops a request that was never queued before the assert is evaluated. The
assert catches the accounting error after the fact; built with
-DNDEBUGthe same path wouldinstead consume from a possibly-empty pipeline.
Affected versions
Reproduced deterministically (3/3) on both branches:
master@ 272eeb6 — assert atshard_connection.cpp:7002.4@ db83a52 (2.4.4) — assert atshard_connection.cpp:658Not debug-only: the default build carries no
-DNDEBUG(
CXXFLAGS = -O2 -g -Wall -Werror=vla -std=c++11), so asserts are live in release builds and inthe published packages.
How it surfaced
The nightly Monitor-Input Fuzz job on the
2.4matrix leg:https://github.com/redis/memtier_benchmark/actions/runs/30605076666 (1 failure in 1280 iterations,
seed
17_all_command_types.txt,exit=-6).A mutation merged corpus lines 61–62 so the
SUBSCRIBEline picked up extra arguments. Thepristine seed passes because its
SUBSCRIBEhas exactly one channel — which is also why themasterleg of the same nightly passed: it is the same bug on master, just not hit by that night'srandom mutations. Minimizing the fuzz artifact reduced it to the one line above.
Environment
sha=006e6a6a, jemalloc-5.3.0), local, no TLS--enable-sanitizers --enable-ubsan, GCC 11.4.0,OpenSSL 3.0.2, ubuntu-22.04)
Suggested direction
Make the reply accounting expect N responses for the multi-argument pub/sub commands
(
SUBSCRIBE,PSUBSCRIBE,UNSUBSCRIBE,PUNSUBSCRIBE) — i.e. the number of pending responses arequest contributes should not be hardcoded to 1 — or reject these commands up front with a clear
error if replaying them is out of scope for arbitrary-command mode.