Summary

The parser's max_tokens limit — the documented defence against oversized GraphQL
documents — did not count comment tokens. A document padded with comments could
therefore pass a max_tokens check of any size while still allocating one Token
object per comment, all of which stay reachable for the lifetime of the parsed
document. A 38 MB comment-padded payload allocated roughly 92 MB of additional
resident memory with max_tokens=5 in force.

Details

Lexer.lookahead() reads tokens one at a time and skips over comments, returning only
the first non-comment token. Each skipped comment is nevertheless fully tokenized,
allocated as a Token object, and linked into the persistent token list via
token.next / token.prev — the chain that powers source-location reporting, and
which is retained as long as the document is.

Parser.advance_lexer() incremented the token counter only for the tokens the lexer
actually returned. Comments were therefore invisible to the accounting, and
max_tokens bounded only the non-comment tokens.

The practical consequence is a gap between what users reasonably believe max_tokens
protects and what it did protect. The limit was not broken for ordinary documents —
real tokens were still capped correctly — so the shortfall was not observable without
specifically testing comment padding.

As with the companion issue, the payload compresses extremely well (# c\n repeated),
so body-size limits and WAF inspection applied to the compressed request do not see
its true size, while the framework decompresses it in memory before the parser is
reached.

Proof of concept

import gc
import resource
from graphql import parse

# 100,000 comments, a 5-token query, and max_tokens=5 — accepted.
query = "# c\n" * 100_000 + "{ a }"
parse(query, max_tokens=5)          # no error raised

# Memory amplification.
big = "# padding comment used to amplify memory per token\n" * 500_000 + "{ a }"
gc.collect()
before = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
parse(big, max_tokens=5)            # ~38 MB source
after = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print(f"RSS delta: +{(after - before) / 1024:.0f} MB")

Measured on GraphQL-core 3.2.8 / CPython 3.12 / Linux x86_64: the 391 KB payload
parsed in 127 ms, and the 38.1 MB payload produced an RSS delta of +92 MB — about
2.4× the source size, on top of the source string itself.

Impact

Denial of service through memory exhaustion. A modest number of concurrent requests
can push a worker past a typical 256 MB or 512 MB container limit and trigger an
OOM kill. No authentication is required, and no data is disclosed or modified.

Depth limiters and complexity analyzers do not help: they run on the AST after parsing
has already succeeded, and the document contains only five real tokens.

Patches

Fixed in 3.2.12 and 3.3.0rc1.

Parser.advance_lexer now counts every token consumed, including the comment tokens
the lexer skips between the previously focused token and the new one, and including
the advance onto EOF so that trailing comments are covered. max_tokens therefore now
bounds every Token the parser allocates.

Note the behaviour change: comments now count toward max_tokens. A
comment-heavy document that was previously accepted may now be rejected. If you parse
documents containing substantial comments — SDL files with documentation comments, for
instance — and you set max_tokens, you may need to raise the limit. The
token_count attribute of a parsed document likewise now includes comments.

Workarounds

If you cannot upgrade, apply a request body-size limit to the decompressed body
before the document reaches parse().

Such a limit is worth keeping after upgrading as well: max_tokens is one guard among
several and bounds the token count, not the overall cost of a request.

References

• GraphQL specification § 2.1.4 — Comments

Credits

Reported by @BZHunt, who also reviewed the fixes
before release. In memory of ZaX, who was the driving force behind this discovery.