feat: implement expressions, cut, and aliases for RNTuple.arrays via formulate - #1705
feat: implement expressions, cut, and aliases for RNTuple.arrays via formulate #1705KaranSinghDev wants to merge 23 commits into
Conversation
…fied along with pre-comits)
…fied along with pre-comits)
Codecov Report❌ Patch coverage is
Additional details and impacted files
|
b8230b3 to
dc6e2fc
Compare
|
@ariostas I was thinking if we should put the virtual arr args first on function, the invalid ones would fail before loading dependencies and parsing |
Yeah, the virtual check should come checks. Expressions won't be supported for virtual arrays (there's already a check there), so it should fail right away. |
|
Also, I see that the tests are failing because numexpr is not installed. I forgot that numexpr is not a dependency of formulate. So we should also add it as a dependency as well. What you have so far looks good to me. I think the one thing it might be worth thinking a bit about is if there should be some simple check that decides whether the cut should be applied before of after evaluation. If the cut throws away most of the data then it's worth applying it before, but if it let's most things through, then it's better to apply it at the end. |
…eck and optimize simple reads
3f5426c to
fa7be54
Compare
|
Hmm I guess Pyodide might be a problem. I'll see what we can do about that |
|
Hi @ariostas I tried to lok in it, since numexpr does not have a WebAssembly wheel, so making it a strict dependency seems to be breaking the Uproot installation for browser users. I think this can be fixed by using an environment marker in pyproject.toml , Also working on the codecov coverage. Let me know your thoughts on this. |
|
Yeah, I think for now we can just add the marker to the pyproject.toml and skip the tests in Pyodide. I'm gonna see how feasible it is to have a numexpr wheel for pyodide. As for codecov. Don't worry too much about it. If the tests ends up needing to be really convoluted just to hit coverage, then they might not be worth it. We might need to reconsider that threshold since we haven't been hitting it for most PRs. |
|
I opened pydata/numexpr#568. We'll see what they say. |
Description
This PR implements initial support for
expressions,cut, andaliasesinRNTuple.arrays()by delegating parsing toformulateand evaluation toak.numexpr, as discussed in #1673.Key Changes
1.formulate.from_root() parses expressions and extracts ast.variables to ensure minimal I/O (we only fetch required fields).
2.Evaluation happens post-materialization via ak.numexpr.evaluate().
3.cut expressions generate a boolean mask applied to the resulting Awkward array.
4.Added regression test in tests/test_1673_rntuple_expressions.py.
@ariostas Let me know your thoughts on this.