Repository navigation
Commit 26cbccd
committed
fix(pipeline): spill before the post-extraction phases when they would cross the budget (DeusData#2184)
Spill mode was entered only DURING extraction, when the charge crossed
budget - budget/16, and the final sweep at extraction end ran only if that
latch had already fired. A run that ended extraction just under the latch
kept every result in memory, and the phases that cannot spill -- registry
build, cross-LSP prepare (all_defs, per-language registries, surface rows,
module index) and resolve (edges into the graph buffer, cross-LSP appends
into the cached results) -- then grew the process well past the budget.
openclaw at 8 workers with a 4079 MB budget: 3678 MB charged at extraction
end, 4076 after registry build, 4933 after cross-LSP prepare, 5829 in
resolve (+43 %). The same repo at 18 workers latched during extraction and
stayed under budget.
Extraction end now projects the post-extraction growth from the result
counts (defs, calls + usages, files) and enters spill mode when charged +
growth would cross the latch, so the existing final sweep parks every
cached result before registry build. The per-unit costs (1280 B/def,
160 B/ref, 8 KiB/file) are fitted on the measured charge growth from the
parallel_extract to the parallel_resolve mark on five local corpora
(go, django, kotlin, rust, php) and cover openclaw; the model under-reads by
at most 7 % (inside the budget/16 margin) and over-reads by up to 46 %, the
safe side: a spill never changes graph content. The decision is a pure
function of counts (no RSS-window sampling) and is logged as
mem.post_extract.projection.
A charge test seam (cbm_mem_set_charged_for_tests) pins cbm_mem_charged(),
which otherwise reads the process footprint, so the "ended just under the
latch" case is testable deterministically.
Proof (release build, M5 Pro, CBM_WORKERS=8, peak charged = the budget
metric's high-water mark):
go, CBM_MEM_BUDGET_MB=2000 before: extract 1348 -> registry 1497 ->
lsp_prepare 2168 -> resolve 2660 MB (+33 % over budget, no spill)
after: projection charged 1335 + growth 1909 > line 1875 -> spill,
21,882 results parked; extract 413 -> registry 558 -> lsp_prepare 1151
-> resolve 1297, semantic 1746 MB peak: every phase under budget.
Node/edge dumps byte-identical (335,191 nodes / 1,813,857 edges); wall
29.2 s vs 29.0 s.
go, 40000 MB (generous): decision=keep, no spill, identical dumps, wall
27.4 s vs 27.5 s.
django, 400 MB: resolve mark 455 -> 277 MB after the spill, dumps
identical; the remaining peak (476 MB) is the semantic pass's retained
set on top of the graph floor, which this change does not address.
Refs DeusData#2184
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>1 parent 9d65f4c commit 26cbccd
4 files changed
Lines changed: 246 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
594 | 594 | | |
595 | 595 | | |
596 | 596 | | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
597 | 601 | | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
598 | 606 | | |
599 | 607 | | |
600 | 608 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
93 | 93 | | |
94 | 94 | | |
95 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
96 | 104 | | |
97 | 105 | | |
98 | 106 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
940 | 940 | | |
941 | 941 | | |
942 | 942 | | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
943 | 1025 | | |
944 | 1026 | | |
945 | 1027 | | |
| |||
1485 | 1567 | | |
1486 | 1568 | | |
1487 | 1569 | | |
| 1570 | + | |
1488 | 1571 | | |
1489 | 1572 | | |
1490 | 1573 | | |
| |||
1493 | 1576 | | |
1494 | 1577 | | |
1495 | 1578 | | |
1496 | | - | |
| 1579 | + | |
| 1580 | + | |
| 1581 | + | |
1497 | 1582 | | |
1498 | 1583 | | |
1499 | 1584 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
616 | 617 | | |
617 | 618 | | |
618 | 619 | | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
619 | 762 | | |
620 | 763 | | |
621 | 764 | | |
| |||
4437 | 4580 | | |
4438 | 4581 | | |
4439 | 4582 | | |
| 4583 | + | |
4440 | 4584 | | |
4441 | 4585 | | |
4442 | 4586 | | |
| |||
0 commit comments