Skip to content

Commit 248e147

Browse files
ptomecekptomecek
committed
Generalize distributed partitioning onto a shared read-partition vocabulary
Add io_sources.partitions, a backend-neutral way to describe how to split a read, and route the Ray executor through it. A ReadPartition is one slice of the input, defined by a Polars predicate. A Partitioner derives slices from the predicate pushed down at scan time, with builders for common axes: - by_time: calendar windows, built with Polars' temporal functions - by_value: one slice per discrete value, or per member group - by_range: fixed-width numeric buckets - by_key: equality on caller-enumerated keys - discrete_partitions / cartesian_partitions: explicit member lists and date-window x bucket products execute_on_ray takes a single partitions argument (a partitioner, an explicit list of ReadPartition, or by_key); RayPartition additionally carries per-task remote options. The date_column/time_unit calendar form is retained as a shortcut for by_time. Ordered output streams partitions in spec order by blocking on the next partition while later tasks run ahead, so the number of outstanding results stays bounded by max_concurrency instead of growing with the total output. Signed-off-by: Pascal Tomecek <40371786+ptomecek@users.noreply.github.com>
1 parent 0ef48ea commit 248e147

27 files changed

Lines changed: 1963 additions & 1676 deletions

‎.copier-answers.yaml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,8 @@
11
# Changes here will be overwritten by Copier
2-
_commit: 7522e24
2+
_commit: '4875149'
33
_src_path: https://github.com/python-project-templates/base.git
44
add_docs: false
55
add_extension: rust
6-
add_pyodide: false
76
add_wiki: true
87
email: OpenSource@point72.com
98
github: Point72

‎.github/workflows/build.yaml‎

Lines changed: 84 additions & 135 deletions
Original file line numberDiff line numberDiff line change
@@ -25,144 +25,93 @@ permissions:
2525

2626
jobs:
2727
build:
28-
name: build (${{ matrix.name }})
2928
runs-on: ${{ matrix.os }}
3029

3130
strategy:
3231
matrix:
33-
include:
34-
- name: ubuntu-latest
35-
dist_path: dist
36-
os: ubuntu-latest
37-
python: "3.11"
38-
target: native
39-
- name: ubuntu-24.04-arm
40-
dist_path: dist
41-
os: ubuntu-24.04-arm
42-
python: "3.11"
43-
target: native
44-
- name: macos-latest
45-
dist_path: dist
46-
os: macos-latest
47-
python: "3.11"
48-
target: native
49-
- name: windows-latest
50-
dist_path: dist
51-
os: windows-latest
52-
python: "3.11"
53-
target: native
54-
32+
os: [ubuntu-latest, macos-latest, windows-latest]
33+
python-version: ["3.11"]
5534

5635
steps:
57-
- name: Checkout
58-
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
59-
with:
60-
persist-credentials: false
61-
62-
- name: Setup Python
63-
uses: actions-ext/python/setup@6e1b91a408ea89f49bc8aff8724f83826f7b5446
64-
with:
65-
version: ${{ matrix.python }}
66-
67-
- name: Setup Rust
68-
uses: actions-ext/rust/setup@2db8e3e4c56beb7320c7d730bd2c2d6a6447bf6b
69-
70-
- name: Install dependencies
71-
run: make develop
72-
if: matrix.target == 'native'
73-
74-
- name: Lint
75-
run: make lint
76-
if: matrix.target == 'native'
77-
78-
- name: Checks
79-
run: make checks
80-
if: matrix.name == 'ubuntu-latest'
81-
82-
- name: Build
83-
run: make build
84-
if: matrix.target == 'native'
85-
86-
- name: Test
87-
run: make coverage
88-
if: matrix.target == 'native'
89-
90-
- name: Build and test Pyodide wheel
91-
run: make test-pyodide
92-
if: matrix.target == 'pyodide'
93-
94-
- name: Upload test results (Python)
95-
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
96-
with:
97-
name: test-results-${{ matrix.name }}
98-
path: '**/junit.xml'
99-
if: always() && matrix.target == 'native'
100-
101-
- name: Publish test results
102-
uses: EnricoMi/publish-unit-test-result-action@d0a4676d0e0b938bc201470d88276b7c74c712b3 # v2.24.0
103-
with:
104-
files: '**/junit.xml'
105-
if: matrix.name == 'ubuntu-latest'
106-
107-
- name: Upload coverage
108-
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
109-
with:
110-
token: ${{ secrets.CODECOV_TOKEN }}
111-
if: matrix.target == 'native'
112-
113-
- name: Free disk space
114-
run: |
115-
sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc /usr/local/.ghcup /usr/share/swift
116-
df -h /
117-
shell: bash
118-
if: matrix.target == 'native' && runner.os == 'Linux'
119-
120-
- name: Build distributions (Linux x86_64)
121-
run: |
122-
rm -rf dist
123-
make dist-rs
124-
make dist-py-sdist
125-
make dist-py-wheel
126-
make dist-check
127-
shell: bash
128-
env:
129-
CIBW_ARCHS_LINUX: native
130-
if: matrix.name == 'ubuntu-latest'
131-
132-
- name: Build distributions (Linux ARM64)
133-
run: |
134-
rm -rf dist
135-
make dist-py-wheel
136-
make dist-check
137-
shell: bash
138-
env:
139-
CIBW_ARCHS_LINUX: native
140-
if: matrix.name == 'ubuntu-24.04-arm'
141-
142-
- name: Build distributions (macOS/Windows)
143-
run: |
144-
rm -rf dist
145-
make dist-py-wheel
146-
make dist-check
147-
shell: bash
148-
env:
149-
CIBW_ENVIRONMENT_MACOS: MACOSX_DEPLOYMENT_TARGET=11.0
150-
if: matrix.target == 'native' && runner.os != 'Linux'
151-
152-
- name: Test wheel
153-
uses: actions-ext/python/test-wheel@6e1b91a408ea89f49bc8aff8724f83826f7b5446
154-
with:
155-
module: polars_io_tools
156-
if: matrix.target == 'native' && runner.os == 'Linux'
157-
158-
- name: Test source distribution
159-
uses: actions-ext/python/test-sdist@6e1b91a408ea89f49bc8aff8724f83826f7b5446
160-
with:
161-
module: polars_io_tools
162-
if: matrix.name == 'ubuntu-latest'
163-
164-
- name: Upload distributions
165-
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
166-
with:
167-
name: dist-${{ matrix.name }}
168-
path: ${{ matrix.dist_path }}
36+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
37+
38+
- uses: actions-ext/python/setup@main
39+
with:
40+
version: ${{ matrix.python-version }}
41+
42+
- uses: actions-ext/rust/setup@main
43+
44+
- name: Install dependencies
45+
run: make develop
46+
47+
- name: Lint
48+
run: make lint
49+
50+
- name: Checks
51+
run: make checks
52+
if: matrix.os == 'ubuntu-latest'
53+
54+
- name: Build
55+
run: make build
56+
57+
- name: Test
58+
run: make coverage
59+
60+
- name: Upload test results (Python)
61+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
62+
with:
63+
name: test-results-${{ matrix.os }}-${{ matrix.python-version }}
64+
path: '**/junit.xml'
65+
if: ${{ always() }}
66+
67+
- name: Publish Unit Test Results
68+
uses: EnricoMi/publish-unit-test-result-action@d0a4676d0e0b938bc201470d88276b7c74c712b3 # v2.24.0
69+
with:
70+
files: '**/junit.xml'
71+
if: matrix.os == 'ubuntu-latest'
72+
73+
- name: Upload coverage
74+
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
75+
with:
76+
token: ${{ secrets.CODECOV_TOKEN }}
77+
78+
- name: Set up QEMU
79+
uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4.2.0
80+
with:
81+
platforms: all
82+
if: runner.os == 'Linux' && runner.arch == 'X64'
83+
84+
- name: Make dist (Linux)
85+
run: |
86+
rm -rf dist
87+
make dist-rs
88+
make dist-py-sdist
89+
make dist-py-wheel
90+
make dist-check
91+
shell: bash
92+
if: matrix.os == 'ubuntu-latest'
93+
94+
- name: Make dist (Macos / Windows)
95+
run: |
96+
rm -rf dist
97+
make dist-py-wheel
98+
make dist-check
99+
shell: bash
100+
env:
101+
CIBW_ENVIRONMENT_MACOS: MACOSX_DEPLOYMENT_TARGET=11.0
102+
if: matrix.os != 'ubuntu-latest'
103+
104+
- uses: actions-ext/python/test-wheel@main
105+
with:
106+
module: polars_io_tools
107+
if: matrix.os == 'ubuntu-latest'
108+
109+
- uses: actions-ext/python/test-sdist@main
110+
with:
111+
module: polars_io_tools
112+
if: matrix.os == 'ubuntu-latest'
113+
114+
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
115+
with:
116+
name: dist-${{matrix.os}}
117+
path: dist

‎.github/workflows/copier.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
name: Copier Updates
2+
3+
on:
4+
workflow_dispatch:
5+
schedule:
6+
- cron: "0 5 * * 0"
7+
8+
jobs:
9+
update:
10+
permissions:
11+
contents: write
12+
pull-requests: write
13+
runs-on: ubuntu-latest
14+
steps:
15+
- uses: actions-ext/copier-update@main
16+
with:
17+
token: ${{ secrets.WORKFLOW_TOKEN }}

‎.github/workflows/wiki.yaml‎

Lines changed: 3 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -20,15 +20,8 @@ jobs:
2020
deploy:
2121
runs-on: ubuntu-latest
2222
steps:
23-
- name: Checkout
24-
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
25-
with:
26-
persist-credentials: false
27-
28-
- name: Prepare wiki
29-
run: cp README.md docs/wiki/Home.md
30-
31-
- name: Publish wiki
32-
uses: Andrew-Chen-Wang/github-wiki-action@1bbb4280446f9630e8e21a18012cbacf3b0f992e # v5.0.6
23+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
24+
- run: cp README.md docs/wiki/Home.md
25+
- uses: Andrew-Chen-Wang/github-wiki-action@1bbb4280446f9630e8e21a18012cbacf3b0f992e # v5.0.6
3326
with:
3427
path: docs/wiki

‎.gitignore‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,6 @@ __pycache__/
1313
*.exp
1414
*.lib
1515

16-
# vcpkg
17-
vcpkg/
18-
vcpkg_installed/
19-
2016
# Rust
2117
target
2218
target-capi
@@ -146,7 +142,6 @@ polars_io_tools/labextension
146142

147143
# Emscripten SDK (locally installed)
148144
emsdk
149-
.pyodide_build/
150145

151146
# Mac
152147
.DS_Store

0 commit comments

Comments
 (0)