Repository navigation
Commit 1b5ba09
committed
fix(cypher): count DISTINCT rows against the row cap (DeusData#1364)
RETURN DISTINCT projected up to the engine row cap (max_rows, 100k by
default) and only then removed duplicates in rb_apply_distinct. A value
that first appeared past the cap was never seen. On a 150k-edge index
`MATCH ()-[r]->() RETURN DISTINCT type(r)` returned 21 of 22 types:
HAS_BRANCH, a single edge, was missing, and the result was flagged
truncated:true / engine_limit.
The projection loops (plain RETURN, RETURN *, RETURN * after WITH) now
de-duplicate while projecting, through a hash set of length-prefixed row
keys. The cap therefore counts distinct rows. Memory stays bounded by the
cap: only admitted rows are remembered. Once the cap is full, the rest of
the bindings are only probed (no insert), to decide whether a new
distinct row exists, which is when `truncated` is set. An explicit LIMIT
on DISTINCT now also counts distinct rows and can stop the projection
early. ORDER BY and SKIP/LIMIT keep running after DISTINCT on the
already-distinct rows. The O(n^2) post-pass remains only for aggregate
rows, which are distinct by group key.
Proof on django (55,377 nodes / 294,528 edges, 23 edge types in
get_graph_schema), same index, before vs after:
before: 22 types, HAS_BRANCH missing, truncated:true engine_limit
after: 23 types, truncated:false (same 5.3 s wall time)
Test: cypher_return_distinct_counts_distinct_rows_against_cap_issue1364
(rare type past a max_rows=5 cap; ORDER BY, SKIP/LIMIT and LIMIT variants;
more distinct rows than the cap still reports truncated).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>1 parent 64c23fa commit 1b5ba09
2 files changed
Lines changed: 226 additions & 18 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
15 | 17 | | |
16 | 18 | | |
17 | 19 | | |
| |||
2874 | 2876 | | |
2875 | 2877 | | |
2876 | 2878 | | |
| 2879 | + | |
| 2880 | + | |
| 2881 | + | |
| 2882 | + | |
| 2883 | + | |
| 2884 | + | |
| 2885 | + | |
| 2886 | + | |
| 2887 | + | |
| 2888 | + | |
| 2889 | + | |
| 2890 | + | |
| 2891 | + | |
| 2892 | + | |
| 2893 | + | |
| 2894 | + | |
| 2895 | + | |
| 2896 | + | |
| 2897 | + | |
| 2898 | + | |
| 2899 | + | |
| 2900 | + | |
| 2901 | + | |
| 2902 | + | |
| 2903 | + | |
| 2904 | + | |
| 2905 | + | |
| 2906 | + | |
| 2907 | + | |
| 2908 | + | |
| 2909 | + | |
| 2910 | + | |
| 2911 | + | |
| 2912 | + | |
| 2913 | + | |
| 2914 | + | |
| 2915 | + | |
| 2916 | + | |
| 2917 | + | |
| 2918 | + | |
| 2919 | + | |
| 2920 | + | |
| 2921 | + | |
| 2922 | + | |
| 2923 | + | |
| 2924 | + | |
| 2925 | + | |
| 2926 | + | |
| 2927 | + | |
| 2928 | + | |
| 2929 | + | |
| 2930 | + | |
| 2931 | + | |
| 2932 | + | |
| 2933 | + | |
| 2934 | + | |
| 2935 | + | |
| 2936 | + | |
| 2937 | + | |
| 2938 | + | |
| 2939 | + | |
| 2940 | + | |
| 2941 | + | |
| 2942 | + | |
| 2943 | + | |
| 2944 | + | |
| 2945 | + | |
| 2946 | + | |
| 2947 | + | |
| 2948 | + | |
| 2949 | + | |
| 2950 | + | |
| 2951 | + | |
| 2952 | + | |
| 2953 | + | |
| 2954 | + | |
| 2955 | + | |
| 2956 | + | |
| 2957 | + | |
| 2958 | + | |
| 2959 | + | |
| 2960 | + | |
| 2961 | + | |
| 2962 | + | |
| 2963 | + | |
| 2964 | + | |
| 2965 | + | |
| 2966 | + | |
| 2967 | + | |
| 2968 | + | |
| 2969 | + | |
| 2970 | + | |
| 2971 | + | |
| 2972 | + | |
| 2973 | + | |
| 2974 | + | |
| 2975 | + | |
| 2976 | + | |
2877 | 2977 | | |
2878 | 2978 | | |
2879 | 2979 | | |
| |||
4348 | 4448 | | |
4349 | 4449 | | |
4350 | 4450 | | |
4351 | | - | |
| 4451 | + | |
| 4452 | + | |
| 4453 | + | |
4352 | 4454 | | |
4353 | 4455 | | |
4354 | 4456 | | |
4355 | 4457 | | |
4356 | 4458 | | |
4357 | | - | |
| 4459 | + | |
| 4460 | + | |
| 4461 | + | |
| 4462 | + | |
4358 | 4463 | | |
| 4464 | + | |
4359 | 4465 | | |
4360 | 4466 | | |
4361 | 4467 | | |
| |||
4370 | 4476 | | |
4371 | 4477 | | |
4372 | 4478 | | |
4373 | | - | |
4374 | | - | |
4375 | | - | |
| 4479 | + | |
| 4480 | + | |
| 4481 | + | |
4376 | 4482 | | |
4377 | 4483 | | |
4378 | | - | |
| 4484 | + | |
| 4485 | + | |
| 4486 | + | |
4379 | 4487 | | |
4380 | 4488 | | |
4381 | | - | |
4382 | | - | |
4383 | | - | |
4384 | | - | |
| 4489 | + | |
| 4490 | + | |
| 4491 | + | |
| 4492 | + | |
4385 | 4493 | | |
| 4494 | + | |
4386 | 4495 | | |
4387 | 4496 | | |
4388 | 4497 | | |
| |||
4651 | 4760 | | |
4652 | 4761 | | |
4653 | 4762 | | |
4654 | | - | |
4655 | | - | |
| 4763 | + | |
| 4764 | + | |
4656 | 4765 | | |
4657 | 4766 | | |
4658 | | - | |
| 4767 | + | |
| 4768 | + | |
| 4769 | + | |
4659 | 4770 | | |
4660 | 4771 | | |
4661 | 4772 | | |
4662 | 4773 | | |
4663 | 4774 | | |
4664 | 4775 | | |
4665 | | - | |
4666 | | - | |
4667 | | - | |
4668 | | - | |
| 4776 | + | |
| 4777 | + | |
| 4778 | + | |
| 4779 | + | |
4669 | 4780 | | |
| 4781 | + | |
4670 | 4782 | | |
4671 | 4783 | | |
4672 | 4784 | | |
| |||
5005 | 5117 | | |
5006 | 5118 | | |
5007 | 5119 | | |
5008 | | - | |
| 5120 | + | |
| 5121 | + | |
| 5122 | + | |
5009 | 5123 | | |
5010 | 5124 | | |
5011 | 5125 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1230 | 1230 | | |
1231 | 1231 | | |
1232 | 1232 | | |
| 1233 | + | |
| 1234 | + | |
| 1235 | + | |
| 1236 | + | |
| 1237 | + | |
| 1238 | + | |
| 1239 | + | |
| 1240 | + | |
| 1241 | + | |
| 1242 | + | |
| 1243 | + | |
| 1244 | + | |
| 1245 | + | |
| 1246 | + | |
| 1247 | + | |
| 1248 | + | |
| 1249 | + | |
| 1250 | + | |
| 1251 | + | |
| 1252 | + | |
| 1253 | + | |
| 1254 | + | |
| 1255 | + | |
| 1256 | + | |
| 1257 | + | |
| 1258 | + | |
| 1259 | + | |
| 1260 | + | |
| 1261 | + | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
| 1265 | + | |
| 1266 | + | |
| 1267 | + | |
| 1268 | + | |
| 1269 | + | |
| 1270 | + | |
| 1271 | + | |
| 1272 | + | |
| 1273 | + | |
| 1274 | + | |
| 1275 | + | |
| 1276 | + | |
| 1277 | + | |
| 1278 | + | |
| 1279 | + | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
| 1283 | + | |
| 1284 | + | |
| 1285 | + | |
| 1286 | + | |
| 1287 | + | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
| 1295 | + | |
| 1296 | + | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
| 1321 | + | |
| 1322 | + | |
| 1323 | + | |
| 1324 | + | |
| 1325 | + | |
1233 | 1326 | | |
1234 | 1327 | | |
1235 | 1328 | | |
| |||
4689 | 4782 | | |
4690 | 4783 | | |
4691 | 4784 | | |
| 4785 | + | |
4692 | 4786 | | |
4693 | 4787 | | |
4694 | 4788 | | |
| |||
0 commit comments