Skip to content

SIGABRT: m_pending_resp goes negative on multi-channel SUBSCRIBE/PSUBSCRIBE/UNSUBSCRIBE #515

Description

@fcostaoliveira

memtier_benchmark aborts when an arbitrary command is a multi-channel pub/sub command
(SUBSCRIBE / PSUBSCRIBE / UNSUBSCRIBE with two or more arguments):

memtier_benchmark: shard_connection.cpp:700: request* shard_connection::pop_req():
Assertion `m_pending_resp.load(std::memory_order_relaxed) >= 0' failed.
Aborted (core dumped)

This is reachable from ordinary CLI usage — no fuzzer, no --monitor-input, and no debug build
required.

Steps to reproduce

Plain release build (autoreconf -ivf && ./configure && make) against a stock local Redis:

$ memtier_benchmark --command="SUBSCRIBE chan1 chan2" \
    --server=127.0.0.1 --port=6379 --test-time=2 --clients=1 --threads=1 --hide-histogram
...
memtier_benchmark: shard_connection.cpp:700: request* shard_connection::pop_req(): Assertion `m_pending_resp.load(std::memory_order_relaxed) >= 0' failed.

=== MEMTIER_BENCHMARK BUG REPORT START: Cut & paste starting from here ===
# memtier_benchmark crashed by signal: 6
# Crashed running signal <SIGABRT>
...
# thread=0 client=0 conn=0 addr=127.0.0.1:6379 local_port=48936 state=connected pending=-1 last_cmd=ARBITRARY

Note pending=-1 in the client-list dump.

Equivalently via --monitor-input, with this single line:

[ p ] 1.0 [0 127.0.0.1:1] "SUBSCRIBE" "chan1" "chan2"

Expected: the run completes (or exits with a clean diagnostic).
Actual: SIGABRT.

Which commands trigger it

--command result
SUBSCRIBE chan1 clean
SUBSCRIBE chan1 chan2 abort
SUBSCRIBE a b c abort
PSUBSCRIBE a* b* abort
UNSUBSCRIBE a b abort
GET k1 (control) clean

The cutoff at two arguments is the tell.

Root cause

Redis replies to SUBSCRIBE/PSUBSCRIBE/UNSUBSCRIBE once per channel or pattern, but
shard_connection accounts for exactly one response per request: push_req() increments
m_pending_resp once, and every reply consumed calls pop_req(), which decrements it. With N
channels, N replies arrive for one queued request, so m_pending_resp goes negative and the assert
in pop_req() fires.

Worth noting for severity: pop_req() does

request *req = m_pipeline->front();
m_pipeline->pop();
m_pending_resp.fetch_sub(1, std::memory_order_relaxed);
assert(m_pending_resp.load(std::memory_order_relaxed) >= 0);

so the surplus reply pops a request that was never queued before the assert is evaluated. The
assert catches the accounting error after the fact; built with -DNDEBUG the same path would
instead consume from a possibly-empty pipeline.

Affected versions

Reproduced deterministically (3/3) on both branches:

  • master @ 272eeb6 — assert at shard_connection.cpp:700
  • 2.4 @ db83a52 (2.4.4) — assert at shard_connection.cpp:658

Not debug-only: the default build carries no -DNDEBUG
(CXXFLAGS = -O2 -g -Wall -Werror=vla -std=c++11), so asserts are live in release builds and in
the published packages.

How it surfaced

The nightly Monitor-Input Fuzz job on the 2.4 matrix leg:
https://github.com/redis/memtier_benchmark/actions/runs/30605076666 (1 failure in 1280 iterations,
seed 17_all_command_types.txt, exit=-6).

A mutation merged corpus lines 61–62 so the SUBSCRIBE line picked up extra arguments. The
pristine seed passes because its SUBSCRIBE has exactly one channel — which is also why the
master leg of the same nightly passed: it is the same bug on master, just not hit by that night's
random mutations. Minimizing the fuzz artifact reduced it to the one line above.

Environment

  • Redis server 8.6.0 (sha=006e6a6a, jemalloc-5.3.0), local, no TLS
  • Ubuntu, GCC 13.3.0, libevent 2.1.12-stable, OpenSSL 3.0.13
  • Also reproduced under the CI configuration (--enable-sanitizers --enable-ubsan, GCC 11.4.0,
    OpenSSL 3.0.2, ubuntu-22.04)

Suggested direction

Make the reply accounting expect N responses for the multi-argument pub/sub commands
(SUBSCRIBE, PSUBSCRIBE, UNSUBSCRIBE, PUNSUBSCRIBE) — i.e. the number of pending responses a
request contributes should not be hardcoded to 1 — or reject these commands up front with a clear
error if replaying them is out of scope for arbitrary-command mode.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions