ReDoS via structured_outputs.regex compiled without timeout in xgrammar and outlines backends

Summary

The structured_outputs.regex API parameter passes a user-supplied regex string directly to grammar compiler backends with no compilation timeout. In the xgrammar backend, the string reaches compile_regex() with no guard. In the outlines backend, validate_regex_is_buildable() blocks structural issues (lookarounds, backreferences) but provides zero protection against exponential DFA state-space explosion. Patterns like (a+)+b pass all checks and hang the inference worker.

Root Cause

backend_xgrammar.py:91 — no timeout:

ctx = self.compiler.compile_regex(grammar_spec)

backend_outlines.py:299–330 — structural checks only, no complexity analysis:

def validate_regex_is_buildable(regex: str) -> None:
    sre_parse.parse(regex)   # AST parse only — does not detect exponential patterns
    _check_unsupported(...)  # blocks lookarounds/backrefs, not nested quantifiers

backend_outlines.py:64 — no timeout:

oc.Index(regex_string, vocabulary.inner)

Impact

Denial of service — one request with an adversarial regex pattern hangs an inference worker indefinitely.

Remediation

Wrap compile_regex() and oc.Index() calls in a thread with a deadline (e.g., 5 seconds). Add complexity analysis to validate_regex_is_buildable() to detect nested quantifier patterns before compilation.