Repository navigation
Commit 8ff6341
feat(scalar): deliver ConstParam as a typed Arrow scalar when annotated as one
A ConstParam was always converted to a Python value via as_py(), so workers that
feed it back into pyarrow.compute (e.g. pc.multiply(value, factor)) make pyarrow
re-infer an Arrow scalar on every batch. That inference is ~40-50x slower on
Windows than passing a typed pa.scalar (and dominates per-batch cost for scalar
UDFs — measured: 148us vs 3us for a 2048-row int64 multiply).
Make delivery follow the annotation: when a ConstParam is annotated with a
pyarrow scalar class (Annotated[pa.Int64Scalar, ConstParam(...)]), deliver the
typed pa.Scalar straight through to compute(); Python-type annotations
(int/str/float/...) still get as_py(), so existing workers are unaffected.
- arguments.py: SCALAR_CLASS_TO_DATATYPE (scalar class -> Arrow type), exported.
- scalar_function.py: _is_scalar_annotation(); _const_param_to_arg infers the
Arrow type from the scalar class; per-class _const_scalar_params drives whether
compute() receives the scalar or the Python value.
- _test_fixtures/scalar/arithmetic.py: MultiplyFunction.factor is now
Annotated[pa.Int64Scalar, ...] — the idiomatic fast pattern.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 4384825 commit 8ff6341
3 files changed
Lines changed: 96 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
54 | | - | |
| 54 | + | |
55 | 55 | | |
56 | | - | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
57 | 65 | | |
58 | 66 | | |
59 | 67 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
109 | 109 | | |
110 | 110 | | |
111 | 111 | | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
112 | 152 | | |
113 | 153 | | |
114 | 154 | | |
| |||
158 | 198 | | |
159 | 199 | | |
160 | 200 | | |
| 201 | + | |
161 | 202 | | |
162 | 203 | | |
163 | 204 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| 51 | + | |
51 | 52 | | |
52 | 53 | | |
53 | 54 | | |
| |||
212 | 213 | | |
213 | 214 | | |
214 | 215 | | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
215 | 235 | | |
216 | 236 | | |
217 | 237 | | |
| |||
235 | 255 | | |
236 | 256 | | |
237 | 257 | | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
238 | 267 | | |
239 | 268 | | |
240 | 269 | | |
| |||
791 | 820 | | |
792 | 821 | | |
793 | 822 | | |
| 823 | + | |
794 | 824 | | |
795 | 825 | | |
796 | 826 | | |
| |||
866 | 896 | | |
867 | 897 | | |
868 | 898 | | |
| 899 | + | |
869 | 900 | | |
870 | 901 | | |
871 | 902 | | |
| |||
940 | 971 | | |
941 | 972 | | |
942 | 973 | | |
| 974 | + | |
| 975 | + | |
943 | 976 | | |
944 | 977 | | |
945 | 978 | | |
| |||
966 | 999 | | |
967 | 1000 | | |
968 | 1001 | | |
| 1002 | + | |
969 | 1003 | | |
970 | 1004 | | |
971 | 1005 | | |
| |||
1051 | 1085 | | |
1052 | 1086 | | |
1053 | 1087 | | |
1054 | | - | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
1055 | 1093 | | |
1056 | 1094 | | |
1057 | 1095 | | |
1058 | 1096 | | |
1059 | 1097 | | |
1060 | | - | |
1061 | | - | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
1062 | 1104 | | |
1063 | 1105 | | |
1064 | 1106 | | |
| |||
0 commit comments