repr: bound regex compile memory, not just the pattern's byte length - #37976
Open
def- wants to merge 1 commit into
Open
repr: bound regex compile memory, not just the pattern's byte length#37976def- wants to merge 1 commit into
def- wants to merge 1 commit into
Conversation
`MAX_REGEX_SIZE_BEFORE_COMPILATION` caps `pattern.len()` at 1 MiB to keep a
user-supplied regex from OOMing envd. A byte budget cannot do that job. The
memory goes to `regex-syntax` translating the pattern's AST into its HIR, a
stage that runs before anything `size_limit` measures, and while cost there is
linear in the AST's node count, the cost per node spans two orders of magnitude:
group, alternation, repetition, assertion 0.33 - 0.53 KB
literal, `.`, `[a-c]` 0.42 - 0.60 KB
`\d`, `\s` 0.65 - 1.1 KB
`\w`, `\W`, case-sensitive `\p{L}` 6 - 18 KB
case-insensitive `\p{L}`, `\p{Alphabetic}` 39 - 41 KB
So a pattern one byte under the limit could allocate ~7.8 GB. Confirmed on a
single-node environment with unprivileged SQL, no table and no cluster work,
from ~90 bytes of query text:
SELECT 'x' ~* repeat(chr(92) || 'p{L}', 209715);
envd VmRSS went 320 MB -> 7 620 MB over ~18 s (VmHWM 8 303 MB) while clusterd
stayed flat at ~420 MB, and only then did the statement error with "Compiled
regex exceeds size limit". `statement_timeout` does not help, since the
allocation precedes the error. Any user who can issue a `SELECT` could do this
on demand, and two concurrent queries exceed any realistic envd memory limit.
Project the cost from the AST instead, which is cheap to obtain (~30 ms and
~10 MB at 174 KB, flat in the pattern's shape) and charge each node a
conservative upper bound: 48 KB for a Unicode-property or Perl class item, the
kinds that expand to hundreds of ranges, and 768 bytes for everything else.
Reject above 1 GiB. Classes nested in a bracketed class are charged too, else
`[\p{L}]` repeated evades the budget at the same cost.
This is additive. The byte limit stays, since it is what bounds the AST parse
this check needs, and its number stays valid in the user-facing docs. Patterns
that were cheap remain accepted: a 1 MiB pattern of plain literals, the shape a
machine-generated alternation takes, still compiles. A counted repetition needs
no budget of its own, since `\p{L}{200000}` stays two AST nodes and the NFA
compiler's incremental `size_limit` check already rejects it in 40 ms at 17 MB.
A pattern we cannot parse falls through to `RegexBuilder` so users keep its
error message. That is not an escape hatch: both parse through the same
`regex-syntax` 0.8.11 with the same default configuration, verified against 35
syntax corner cases to never split in the direction that would skip the check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
def-
force-pushed
the
def/fuzz-06-repr-regex
branch
from
July 31, 2026 10:07
74a118a to
c88566b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MAX_REGEX_SIZE_BEFORE_COMPILATIONcapspattern.len()at 1 MiB to keep auser-supplied regex from OOMing envd. A byte budget cannot do that job. The
memory goes to
regex-syntaxtranslating the pattern's AST into its HIR, astage that runs before anything
size_limitmeasures, and while cost there islinear in the AST's node count, the cost per node spans two orders of magnitude:
group, alternation, repetition, assertion 0.33 - 0.53 KB
literal,
.,[a-c]0.42 - 0.60 KB\d,\s0.65 - 1.1 KB\w,\W, case-sensitive\p{L}6 - 18 KBcase-insensitive
\p{L},\p{Alphabetic}39 - 41 KBSo a pattern one byte under the limit could allocate ~7.8 GB. Confirmed on a
single-node environment with unprivileged SQL, no table and no cluster work,
from ~90 bytes of query text:
SELECT 'x' ~* repeat(chr(92) || 'p{L}', 209715);
envd VmRSS went 320 MB -> 7 620 MB over ~18 s (VmHWM 8 303 MB) while clusterd
stayed flat at ~420 MB, and only then did the statement error with "Compiled
regex exceeds size limit".
statement_timeoutdoes not help, since theallocation precedes the error. Any user who can issue a
SELECTcould do thison demand, and two concurrent queries exceed any realistic envd memory limit.
Project the cost from the AST instead, which is cheap to obtain (~30 ms and
~10 MB at 174 KB, flat in the pattern's shape) and charge each node a
conservative upper bound: 48 KB for a Unicode-property or Perl class item, the
kinds that expand to hundreds of ranges, and 768 bytes for everything else.
Reject above 1 GiB. Classes nested in a bracketed class are charged too, else
[\p{L}]repeated evades the budget at the same cost.This is additive. The byte limit stays, since it is what bounds the AST parse
this check needs, and its number stays valid in the user-facing docs. Patterns
that were cheap remain accepted: a 1 MiB pattern of plain literals, the shape a
machine-generated alternation takes, still compiles. A counted repetition needs
no budget of its own, since
\p{L}{200000}stays two AST nodes and the NFAcompiler's incremental
size_limitcheck already rejects it in 40 ms at 17 MB.A pattern we cannot parse falls through to
RegexBuilderso users keep itserror message. That is not an escape hatch: both parse through the same
regex-syntax0.8.11 with the same default configuration, verified against 35syntax corner cases to never split in the direction that would skip the check.