QA: JET package analysis exceeds the 60-minute root QA budget

[ChrisRackauckas-Claude]

Reproduction

Current master: 754b9679a017aae5d3a632d1d95dfd0c82229b41
Julia: 1.12.6
SciMLTesting: 2.8.0
JET: 0.11.6

From a clean detached master worktree with an otherwise empty, dedicated depot, run:

timeout 3600 env GROUP=QA JULIA_DEPOT_PATH=$PWD/.julia_depot_rootqa_master_20260813 JULIA_NUM_PRECOMPILE_TASKS=8 julia +release --startup-file=no --project=. -e 'using Pkg; Pkg.test(; coverage = false)'\n```\n\nPackage setup/precompilation completed in 809 seconds. The selected QA body then spent the remaining time in `JET.test_package(ModelingToolkit; target_defined_modules = true)` and was terminated by the 3600-second timeout before producing a test result. The timeout stack identifies:\n\n```text\nJET.collect_callee_reports!\nJET.report_package\nJET.test_package\nSciMLTesting.run_qa  (test/qa/qa.jl:363)\n```\n\nAt termination, JET reported `Allocations: 3851649683` and `GC: 262`. A native stack sample while it was active was in `JET/src/abstractinterpret/typeinfer.jl:12`, `collect_callee_reports!`.\n\nThe timeout signal reached the root Pkg process and QA runner; the runner required explicit cleanup afterward, so this also leaves an orphan process when invoked under plain `timeout`.\n\nThe InputOutput public-API smoke test runs before `run_qa` and does not alter the existing JET configuration. `test/qa/qa.jl` used the same `jet = true` and `target_defined_modules = true` configuration before that work.\n\n## Expected\n\nStrict root QA should complete within its configured one-hour local budget, or JET analysis needs a maintainable, evidence-based reduction/fix rather than silently disabling the check.

[ChrisRackauckas-Claude]

Follow-up measurements from a clean MTK worktree (Julia 1.12.6, JET 0.11.6, SciMLTesting 2.8.0):

• Removing MTK's jet_kwargs override (target_defined_modules=true) and using SciMLTesting's standard defaults did not solve the lane. The focused QA process was terminated after 18 minutes at 28.36 GB RSS without producing a test summary.
• A per-signature comparison isolated the expensive path to linearization_function: with aggressive_constant_propagation=true, it took 152 s, allocated 31.35 GB, and reached 7.38 GB live / 9.16 GB max RSS; with it false, it took 16 s and about 2 GB.
• A package-level typo-mode run with aggressive_constant_propagation=false completed in 87 s at 2.73 GB, but produced 20 JET reports. The reports are generated-closure UndefVarErrorReport entries in analysis_points.jl, sccnonlinearproblem.jl, alias_elimination.jl, and structural_transformation/symbolics_tearing.jl, plus a generated BuiltinErrorReport. They are not a clean passing QA result.

Therefore removing the deprecated MTK override is useful cleanup but is not a fix for this issue, and changing the global default to disable aggressive constant propagation would merely trade the timeout for 20 reports. No workaround PR was opened. The remaining blocker is JET's analysis of the generated Moshi/symbolic closures; it needs an upstream JET/Moshi fix or a narrowly justified upstream-supported configuration.

[ChrisRackauckas-Claude]

Follow-up from a fresh clone of master d4a73e1f12 (Julia 1.12.6, JET 0.11.6, Moshi 0.3.12, SciMLTesting 2.8.0), QA env built the way run_qa builds it (test/qa/Project.toml sandboxed, root + lib/ModelingToolkitBase Pkg.developed). No repo change is proposed; the blockers are upstream. Evidence:

1. What CI actually does. The QA lane on master (tests / QA (julia 1)) has been red since JET was added on 2026-06-15 (#4610); the latest failure is https://github.com/SciML/ModelingToolkit.jl/actions/runs/31862677442/job/94958687187. It is not the 120-minute job timeout: precompile takes ~23 min, JET starts at 04:44 and at 04:58 the log ends with The runner has received a shutdown signal — the 8 GB self-hosted runner is being killed under JET, consistent with the RSS numbers below.

2. Memory/time is JET's aggressive constant propagation, and it is not configurable going forward. JET.report_package(ModelingToolkit; target_modules=(ModelingToolkit,), mode=:typo) (SciMLTesting's default) went from 0.9 GB to 16.6 GB RSS in 9 min and was still climbing when I stopped it. With aggressive_constant_propagation=false the same call finishes in 94 s at 4.3 GB (basic mode: 129 s, 3.1 GB, 46 reports). JET 0.12 hard-codes aggressive_constant_propagation = true in JETAnalyzer and rejects compiler-parameter keywords with JETConfigError (release notes for v0.12.0, "config: Hide compiler parameter settings from users" aviatesk/JET.jl#845), so that knob is not a path forward. JET 0.12 also removed target_defined_modules, so the current jet_kwargs = (; target_defined_modules = true) in test/qa/qa.jl will hard-error once test/qa/Project.toml allows JET 0.12; that override also drops SciMLTesting's mode = :typo default (JET's default is :basic).

3. The 20 typo-mode reports are all Moshi @match codegen, not MTK-authored code. Every one is local variable ##And#NNN#x#1/##Call#NNN#x#1 may be undefined at Moshi.Match.@match sites (if_lifting.jl:80, analysis_points.jl:301, alias_elimination.jl:292/542, symbolics_tearing.jl:69, sccnonlinearproblem.jl:710). Moshi expands A(; f) && if guard end => body to

if begin v = x; begin isa_variant(v, A) && begin ff = getfield(v, :f); true end end end && guard

The inner conjunction is wrapped in a begin … end block, so Julia lowers it through a value merge (φ (#2 => %4, #3 => #undef)) and emits throw_undef_if_not for ff; Julia's own compiler treats the binding as maybe-undefined and JET reports exactly that. At runtime ff is always assigned on the only path that reaches the guard/body, so it is a false positive plus a dead runtime undef check. Standalone reproducer, no SciML packages, reproduces on JET 0.11.6 and 0.12.1:

using JET
struct A; f; end
function g(x)
    if begin v = x; begin v isa A && begin ff = v.f; true end end end && (ff isa Int)
        return ff
    end
    return nothing
end
occursin("throw_undef_if_not", string(first(code_typed(g, (Any,)))))  # true
JET.report_call(g, (Any,); mode = :typo)  # 2 reports: `ff` may be undefined
# flat chain: `x isa A && begin ff = x.f; true end && (ff isa Int)` -> no undef check, no reports

So the fix is in Moshi's @match codegen (emit a flat && chain rather than nesting the conjunction in a block); MTK has 7 such guard sites in src/ (and MTKBase 56), and rewriting them by hand only until Moshi is fixed would still leave the item-2 blowup.

What I did not do: no MTK PR, because (a) removing the deprecated target_defined_modules override alone leaves the lane red (item 2), and (b) suppressing the item-3 reports would be an allowlist over false positives. Next steps are upstream: a Moshi codegen fix (reproducer above), and a JET-side answer to the const-prop blowup on linearization_function; once both land, test/qa/qa.jl should drop jet_kwargs and take SciMLTesting's default.

[ChrisRackauckas-Claude]

Fresh clean-master reproduction confirms the QA failure is not introduced by the rendered-doc cleanup:

• Base: ModelingToolkit master d4a73e1
• Julia: 1.12.4
• Command: env GROUP=QA julia --project=. -e 'using Pkg; Pkg.test()'
• Result: QA reached 501 passed, 1 error, 502 total after 23m48s; JET interrupted while analyzing.
• RSS reached 27.4 GB without completing.
• target_modules=(ModelingToolkit,) still reached 25.3 GB; SciMLTesting 2.8 typo mode reached 17.8 GB.
• aggressive_constant_propagation=false completed in 9m47s at 5.1 GB but produced 46 JET reports, so it is not a valid green fix.

A standalone environment containing only Moshi and JET reproduces the generated short-circuit pattern-guard false positive: runtime behavior is correct, but JET reports generated ##And locals as possibly undefined. The relevant MTK sites include analysis_points.jl:301, if_lifting.jl:79, and structural_transformation/symbolics_tearing.jl:69.

Conclusion: there is no focused MTK-only fix that makes this green without suppressing reports or changing tests. The remaining blocker is upstream JET/Moshi handling of generated pattern guards plus package-wide JET constant-propagation cost.

[ChrisRackauckas-Claude]

Fresh audit of the exact failed job and current master (2026-08-15):

• The failed job at https://github.com/SciML/ModelingToolkit.jl/actions/runs/31869818100/job/94987763937 ran d4a73e1 on self-hosted-4vcpu-16gb. GitHub finalized the job after 64 minutes with the test step still marked in_progress; the job-logs endpoint returns 404, so the runner disappeared without finalizing or uploading logs.
• A clean isolated Julia 1.12.6 reproduction at that exact commit completed locally with approximately 40 GB peak RSS. Root QA reported 508 pass / 1 fail / 2 errors: 20 JET reports from generated Moshi pattern guards, plus two ExplicitImports UnanalyzableModuleExceptions caused by the dynamic include at src/precompile.jl:1.
• 76e76ea (the rendered-API cleanup) retained the same 20 JET reports, the same two ExplicitImports errors, and exceeded 42 GB RSS. It did not affect this failure.
• Current master f324831 changes jet = true to jet = false. A clean exact-current control therefore no longer runs the failing analysis, but root QA is still red at 50 pass / 2 errors because both ExplicitImports checks remain blocked by the dynamic include. The QA body took 4m26.7s; the full clean invocation including precompilation took 21m34.95s and peaked at 6,152,344 KB.

The generated-guard report reproduces without any SciML package in clean environments containing Moshi 0.3.12 and either JET 0.11.6 or JET 0.12.1. This runtime-correct function returns 1, while lowered code contains throw_undef_if_not and both JET versions report the same two possibly-undefined generated ##And#...#f#1 locals:

using JET
using Moshi
using Moshi.Data: @data

@data Expr begin
    struct A
        f
    end
end

function moshi_guard(x)
    return Moshi.Match.@match x begin
        Expr.A(; f) && if f isa Int end => f
        _ => nothing
    end
end

moshi_guard(Expr.A(1)) # 1
occursin("throw_undef_if_not", string(first(code_typed(moshi_guard, (Any,))))) # true
JET.report_call(moshi_guard, (Any,); mode = :typo) # 2 UndefVarErrorReports

The source/config boundary is d647d61: it added the root QA file with jet = true; its parent has no root QA file. Thus the JET lane was not regressed from a previously green root lane—it was failing from the lane's introduction. No workaround PR is proposed here: disabling JET does not fix either the upstream false positives or the package-analysis memory cost.

[ChrisRackauckas-Claude]

MTKBase-specific history audit (2026-08-30):

• Clean upstream/master at f9131984d97460057c62830b60790380ee5fc2f1, Julia 1.12.7, JET 0.12.1, SciMLTesting 2.13.0: GROUP=QA julia +1.12 --project=. -e 'using Pkg; Pkg.test()' reproduced ═════ 263 possible errors found ═════ in JET.report_package(ModelingToolkitBase; target_modules=(ModelingToolkitBase,), mode=:typo). The package-wide QA body took 21m09.6s. Representative reports are the same generated ##And#... locals described here and in Package analysis exhausts memory; generated short-circuit guards produce false-positive UndefVarError reports aviatesk/JET.jl#858.
• The master history boundary is 92c27bb, which replaced MTKBase 's Aqua-only body with run_qa(..., jet=true) through Adopt SciMLTesting 2.4 strict QA #4832.
• The very first MTKBase QA CI run for that change already reported the identical 263 JET findings: https://github.com/SciML/ModelingToolkit.jl/actions/runs/31478008118/job/93737918032. Thus this was not regressed from a previously green MTKBase package-JET lane; the lane failed from its introduction.
• The separate four-name public-reexport failure on current master comes from Symbolics 7.38 export metadata and is unrelated to JET.

No JET test suppression or MTK source workaround is proposed. The existing standalone upstream reductions remain the appropriate blockers: aviatesk/JET.jl#858 and JuliaLang/julia#62745.

[ChrisRackauckas-Claude]

Additional MTKBase-specific bisect and fix-attempt evidence:

• Clean 5509368fa6708bc5edac3798aa0c4ab65374591f, Julia 1.12.7, JET 0.12.1: the dedicated-depot QA run passed the call-analysis testset 54/54, then package analysis reported the same 263 possible errors; QA ended 20 pass/1 fail and exited 1 after 38m14s.
• The standalone no-SciML Moshi package from the earlier comments passes its runtime guarded-match assertions. On Julia 1.12.0 and Moshi 0.3.12, JET.test_package exits 0 with JET 0.10.15 and reports three ##And... may be undefined errors with JET 0.11.0. JET 0.11.6, 0.12.0, and 0.12.1 reproduce. This narrows the JET dependency boundary to 0.10.15 → 0.11.0.
• A commit-level JET bisect with that same standalone package is clean at 14a60904 and fails with the three reports at 3fe9d8a0, "simplify report_package analyzer" (aviatesk/JET.jl#774). The preceding Revise-loader and module-filter commits all exit 0 on this reproducer.
• The same standalone failure occurs with Moshi 0.3.0, 0.3.6, 0.3.7, 0.3.11, and 0.3.12 under JET 0.11.6, so it is not a recent Moshi release regression.
• Compatible controls exit 0 on Julia 1.10.12/JET 0.9.18 and Julia 1.11.9/JET 0.9.20. Julia 1.12.0, 1.12.3, and 1.12.6 reproduce with JET 0.11.6.

I also tested the Moshi-side structural fix suggested above: emit the pattern conjunction directly rather than wrapping each && expression in an Expr(:block, ...). The local change is two insertions/four deletions in src/match/emit/logic.jl and ctx.jl. With that patch:

standalone runtime assertions: pass
JET.test_package(MoshiJETMWE): exit 0, no reports
Moshi data tests:            325/325
Moshi match tests:           140/140
Moshi derive tests:            30/30
Moshi perf regression tests:    5/5

No MTK workaround is proposed: pinning JET, disabling package analysis, or allowlisting generated reports would hide the failure. The currently failing downgrade-mtkbase (QA) job is https://github.com/SciML/ModelingToolkit.jl/actions/runs/33413872118/job/99575998833.

[ChrisRackauckas-Claude]

Package-wide validation of the Moshi flattening prototype is complete, and it is not a full fix for MTKBase:

unpatched JET.test_package(ModelingToolkitBase): 263 possible errors, exit 1
flattened-Moshi prototype:                        211 possible errors, exit 1

The patched run used Julia 1.12.6, JET 0.11.6, the clean MTKBase worktree, a separately developed local Moshi checkout, and a fresh writable depot. It completed in about 30 minutes. Thus the two-line flattening change remains a valid fix for the standalone guarded-match reproducer and removes 52 package reports, but additional Moshi emission shapes and non-Moshi MTKBase findings remain. It should not be described as making the MTKBase QA lane green.

The prototype itself remains locally validated with JuliaFormatter unchanged, typos clean, git diff --check clean, the standalone JET reproducer passing, and Moshi's 500 tests passing. No suppression or MTK test/configuration change was made.

[ChrisRackauckas-Claude]

Current-master refresh (2026-08-31): the MTKBase package-JET failure still reproduces on clean e91f13e48fc4f2fdda67e66ad7b2d9c67234ac2b.

Environment:

Julia 1.12.7 (6d172b025e4)
JET 0.12.1
SciMLTesting 2.13.0
ModelingToolkitBase 1.68.2
Linux x86_64, 1 Julia thread

Focused reproduction, after resolving the clean lib/ModelingToolkitBase/test/qa environment into a dedicated writable depot:

using ModelingToolkitBase, JET
rep = JET.report_package(
    ModelingToolkitBase;
    target_modules = (ModelingToolkitBase,),
)
reports = JET.get_reports(rep)
@show length(reports)

Observed output:

[toplevel-info] Analyzed all top-level definitions (all: 5169 | analyzed: 5169 | cached: 0 | took: 1056.555 sec)
length(reports) = 450

The reports are the same broad failure class already documented above. The first six are MethodErrorReports for keyword calls to toexpr(::BasicSymbolic{SymReal}); most of the following reports are generated-local UndefVarErrorReports such as ##And#280#f#1 may be undefined; later reports include invalid inferred constructor calls for SCC nonlinear and optimization problems.

I repeated the repository-history bisect with the focused predicate "does the MTKBase QA body invoke package-wide JET analysis?" The first bad commit remains 92c27bb9fdcaa024ff190feb3d665452e33f9bd8, which replaced the Aqua-only body with run_qa(ModelingToolkitBase; Aqua, JET, jet = true). Its parent does not invoke package-wide JET. The first CI run for that change already failed with the same diagnostics, as recorded in the previous MTKBase-specific audit.

No MTK suppression/configuration change is proposed. The package-wide Moshi flattening experiment already recorded above reduced, but did not eliminate, the findings; the upstream JET/Moshi blockers remain the appropriate scope for a fix.