Set hail!=0.2.139 in requirements.txt to avoid array-literal contains memory regression - #845
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The changes are small, self-contained, and correctly prevent setup.py from ingesting the newly added comment/pinned Hail line while keeping Hail excluded from install_requires.
Pull request overview
This PR updates the development/test dependency set to avoid a known Hail 0.2.139 region-memory regression triggered by using array-literal .contains() inside hl.Table.filter, and adjusts packaging logic so setup.py continues to exclude Hail from install_requires even when the requirements entry includes a version specifier.
Changes:
- Add an upper bound (
hail<0.2.139) inrequirements.txtwith context on the regression and removal criteria. - Update
setup.pyrequirements parsing to (a) skip blank/comment lines and (b) excludehailby package name rather than exact-line match, keeping Hail out ofinstall_requireseven when pinned.
File summaries
| File | Description |
|---|---|
| setup.py | Improves requirements parsing so comment/blank lines don’t break install_requires, and excludes Hail by name even when version-spec’d. |
| requirements.txt | Pins Hail below 0.2.139 and documents the rationale/exit criteria for the temporary constraint. |
Review details
- Files reviewed: 1/2 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Hail 0.2.139 leaks region memory when an array literal's .contains() is used inside a Table filter (e.g. filter_gencode_ht's hl.literal(feature).contains(...). In CI the 1 GB driver JVM runs out of heap during test_filter_to_cds, dies, and pytest hangs on the dead py4j connection until the job is cancelled. Set-based membership and Hail 0.2.138 are unaffected. The exclusion lifts itself on 0.2.140. setup.py has excluded hail from install_requires since 2020 by matching the exact string hail; match on the package name instead, and skip blank and comment lines, so the versioned requirements.txt line stays excluded. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> EOF )
mike-w-wilson
force-pushed
the
mw/pin_hail_below_0.2.139
branch
from
September 10, 2026 18:19
efcf091 to
e5a90e9
Compare
mike-w-wilson
changed the title
ch-kr
approved these changes
Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hail 0.2.139 leaks region memory when an array literal's .contains() is used inside a Table filter (e.g. filter_gencode_ht's hl.literal(feature).contains(...). In CI the 1 GB driver JVM runs out of heap during test_filter_to_cds, dies, and pytest hangs on the dead py4j connection until the job is cancelled. Set-based membership and Hail 0.2.138 are unaffected.
setup.py has excluded hail from install_requires since 2020 by matching the exact string hail, the edits to set up now get around the < version bit. We will revert to an unpinned version once the 139 bugs are fixed.
I chose to update requirements since the test suite will fail locally for anyone trying to run it on 0.2.139 as opposed to updating just the way git actions was installing hail.