globset: add fast path for case-insensitive literal/basename patterns - #3488
Open
dhmztr wants to merge 1 commit into
Open
globset: add fast path for case-insensitive literal/basename patterns#3488dhmztr wants to merge 1 commit into
dhmztr wants to merge 1 commit into
Conversation
Case-insensitive glob patterns that are pure literals (no wildcards), e.g. "foo.txt" or "**/FOO" with case_insensitive(true), previously fell through to full regex matching for every candidate path, even though the pattern has no wildcards at all. On large pattern sets this is orders of magnitude slower than the literal fast path already used for case-sensitive literals (see benchmark in BurntSushi#1086). - literal()/basename_literal() no longer bail out early on case-insensitive patterns; they now ASCII-lowercase the extracted literal and let it flow into the existing Literal/BasenameLiteral match strategies. - LiteralStrategy/BasenameLiteralStrategy gained a case-insensitive mode: literals are lowercased on insert, and lookups lowercase the candidate path/basename before hashing. - GlobSet::new() now builds separate case-sensitive and case-insensitive instances of each strategy, routing each pattern into the right one. - ext()/prefix()/suffix() are intentionally left untouched (still bail out on case-insensitive patterns) since their strategies (ExtensionStrategy, prefix/suffix MultiStrategyBuilder) don't have a case-insensitive comparison mode; wiring case-insensitive literals into them without that support would silently produce wrong matches for real files with different-case extensions. - ASCII-only lowercasing (not Unicode `to_lowercase()`) to match the existing behavior of this crate's regex fallback, which is built without the regex-syntax `unicode-case` feature and cannot fold non-ASCII case pairs either (confirmed: even the pre-existing `(?i)` regex path fails to compile for non-ASCII case-insensitive literals in this build configuration). Addresses the literal and basename-literal cases from BurntSushi#1086, which cover the reported `ignore::overrides` benchmark scenario. Extension, prefix, and suffix case-insensitive patterns still fall back to regex and are left for a follow-up, so this does not fully close BurntSushi#1086.
There was a problem hiding this comment.
Pull request overview
Improves globset performance for case-insensitive glob patterns that are pure literals (full-path literals and basename-only literals) by avoiding the regex-based fallback and using dedicated literal match strategies instead.
Changes:
- Route case-insensitive literal and basename-literal patterns into dedicated
LiteralStrategy/BasenameLiteralStrategyinstances. - Add case-insensitive support to
LiteralStrategyandBasenameLiteralStrategyby ASCII-lowercasing stored keys and lowercasing candidates on lookup. - Update
Globliteral/basename-literal extraction to support case-insensitive literals (ASCII-only) and adjust tests accordingly.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| crates/globset/src/lib.rs | Splits literal/basename-literal strategies into case-sensitive and case-insensitive variants and wires routing in GlobSet::new(). |
| crates/globset/src/glob.rs | Enables extracting literals/basename-literals for case-insensitive globs (ASCII lowercased) and updates matching/tests. |
Comments suppressed due to low confidence (2)
crates/globset/src/lib.rs:771
matches_intoalso allocates a lowercased Vec for every case-insensitive candidate lookup. Similar tois_match, it can avoid allocating for already-lowercase candidates by attemptinggetwith the original bytes first and only lowercasing on a miss.
let hits = if self.1 {
self.0.get(&candidate.path.to_ascii_lowercase())
} else {
self.0.get(candidate.path.as_bytes())
};
crates/globset/src/lib.rs:823
BasenameLiteralStrategy::matches_intoallocates a lowercased Vec on every case-insensitive lookup. As withis_match, you can avoid allocations for already-lowercase basenames by attemptinggetwith the original bytes first and only lowercasing on miss.
let hits = if self.1 {
self.0.get(&candidate.basename.to_ascii_lowercase())
} else {
self.0.get(candidate.basename.as_bytes())
};
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
749
to
+753
| fn is_match(&self, candidate: &Candidate<'_>) -> bool { | ||
| self.0.contains_key(candidate.path.as_bytes()) | ||
| if self.1 { | ||
| self.0.contains_key(&candidate.path.to_ascii_lowercase()) | ||
| } else { | ||
| self.0.contains_key(candidate.path.as_bytes()) |
Comment on lines
795
to
+799
| fn is_match(&self, candidate: &Candidate<'_>) -> bool { | ||
| if candidate.basename.is_empty() { | ||
| return false; | ||
| } | ||
| self.0.contains_key(candidate.basename.as_bytes()) | ||
| if self.1 { |
Comment on lines
+485
to
+489
| let case_insensitive = p.is_case_insensitive(); | ||
| match MatchStrategy::new(p) { | ||
| MatchStrategy::Literal(lit) => { | ||
| lits.add(i, lit); | ||
| if case_insensitive { | ||
| lits_ci.add(i, lit); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses #1086: case-insensitive glob patterns that are pure literals (no wildcards) — e.g. a bare filename with
.case_insensitive(true)— were always falling through to full regex matching, even though there's no wildcard to justify it. On a large pattern set this is orders of magnitude slower than the literal fast path already used for case-sensitive literals (see the benchmark in the issue).Scope note: this covers the
LiteralandBasenameLiteralstrategies, which is what the issue's ownignore::overridesbenchmark exercises.ext()/prefix()/suffix()are intentionally left as-is (still falling back to regex for case-insensitive patterns) —ExtensionStrategyand the prefix/suffixMultiStrategyBuilderhave no case-insensitive comparison mode, and wiring case-insensitive literals into them without that support would silently produce wrong matches (e.g. a case-insensitive*.TXTpattern failing to match a realphoto.TXTfile). Extending those is a reasonable follow-up but is a larger, separate change — didn't want to bundle a correctness risk into a perf fix.Changes
literal()/basename_literal()no longer bail out early oncase_insensitive; they ASCII-lowercase the extracted literal and let it flow into the existingLiteral/BasenameLiteralmatch strategies instead of forcingMatchStrategy::Regex.LiteralStrategy/BasenameLiteralStrategygained a case-insensitive mode: literals are lowercased on insert, lookups lowercase the candidate path/basename before hashing.GlobSet::new()now builds separate case-sensitive and case-insensitive instances of each strategy and routes each pattern into the right one.to_ascii_lowercase, notto_lowercase) to match this crate's actual case-insensitive semantics —regex-syntaxis built here without theunicode-casefeature, so even the pre-existing(?i)regex fallback can't fold non-ASCII case pairs (confirmed directly:(?i)on a pattern with a non-ASCII literal fails to compile at all in this build withUnicodeCaseUnavailable). Using Unicode-aware lowercasing in the new fast path would have made it behave differently from the regex path it replaces for the same logical pattern.Test plan
cargo test -p globset— 293 passed, 0 failed (added case-insensitive literal/basename correctness tests, plus fixed two pre-existing tests that encoded the old bail-out-to-None behavior as expected)cargo test -p ignore(direct consumer) — 182 unit + 5 integration + 7 doctests, all passcargo fmt -p globset -- --check— cleancargo clippy -p globset --all-targets -- -D warnings— no new warnings introduced by this diff (24 pre-existing warnings elsewhere in the crate, unrelated to this change, left untouched)ripgrep/grep-*crates pin 1.96 — unrelated to this change;globsetitself only requires 1.88 and builds/tests fine)