Commit f50710e
committed
feat(sql): add opt-in partitioned reads to scan_db
Add a `partitions=` argument to `scan_db` that splits a read into independent
slices pulled over parallel connections and concatenated in order, speeding up
large scan-like extracts whose single-cursor transfer is the bottleneck.
`scan_db` consumes the shared read-partition vocabulary from
`io_sources.partitions` (the same `ReadPartition` / `by_time` / `by_value` /
`by_range` used by the distributed executor). Each slice's predicate is
translated to a SQL `WHERE` and injected as an innermost subquery so it
survives outer projection; a predicate that does not translate to SQL raises
rather than silently reading the whole slice. Concurrency is capped by the
Polars thread pool (`POLARS_MAX_THREADS`), with an optional `max_concurrency`
to throttle below it. The path is fully opt-in and falls back to a single query
when a partitioner cannot derive a bounded split.
By default slices are concatenated in partition order, matching an unpartitioned
read. Pass `preserve_partition_order=False` to yield each slice as it finishes
instead, lowering time-to-first-row and avoiding head-of-line blocking behind a
slow slice, at the cost of a nondeterministic row order.
Add unit tests for the partitioned reader and a how-to section in the Reading
and Writing Data wiki.
Signed-off-by: Pascal Tomecek <40371786+ptomecek@users.noreply.github.com>1 parent 7e83ded commit f50710e
3 files changed
Lines changed: 510 additions & 45 deletions
File tree
- docs/wiki
- polars_io_tools
- io_sources
- tests/io_sources
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
70 | 116 | | |
71 | 117 | | |
72 | 118 | | |
| |||
0 commit comments