[Idea Optimization pooling with other known custom Firefox builders

[DoulosTrieste]

There's a veteran Firefox builder from the early days who still releases builds today.
Although he builds for SSE3 at most, there's some manual techniques he uses to quite good effect. Real-world performance is faster than vanilla Firefox.

https://tete009.pages.dev/en-US/software

[SyntaxError-PEBKAC]

OoOoOo This is exciting! It looks like Tete009 is using Context-Sensitive IR (CSIR) PGO, which is a two-round PGO pass instead of the single PGO pass I currently run. I wasn't aware CSIR existed prior.

Apparently, CSIR makes smarter in-lining decisions while also recording context-sensitive profile data for a more optimized final build.

It'll double build time for single digit performance gains, and I may have to manually run a few extra commands (or script them) during the build step and re-write my building.md notes... but that's never stopped me before ;-)

Thanks for letting me know about this. I'm reading more about it now, and will test implementing it in my next release. I'll be sure to reference you and Tete009 in the release notes!

[SyntaxError-PEBKAC]

So... I went deep on it. Short answer up front: I can't get it into ducksteps right now, but I learned exactly why. Simple version first, then the gory details, then what I ruled out and what'd have to change.

🦆 The simple version

ducksteps already does a smart trick: it takes the browser for a long test drive, watches which code runs the most, and makes that code faster. CSIR PGO is a fancier second lap that also remembers which part of the program asked for each piece of code; so it can tune the same function differently depending on who's calling it. Worth maybe a few percent more, in theory.

The catch: the worker who performs CSIR's special step only clocks in at the very last stage of the build (the part called the "linker"). The Windows version of that worker is missing the one button needed to switch CSIR on. So when I enable it, the build finishes, the multi-hour test drive runs… and out comes nothing... no error, no data, no benefit. It quietly does nothing.

I tried every workaround I could think of (listed below). All dead ends on my setup. So CSIR is off the table for ducksteps for now; not because the idea is bad or inferior, but because I can't find a way to get the Windows tooling to do it yet. Genuinely disheartened; I was excited for this. 🥲

🔬 The technical version

CSIR inserts its instrumentation after inlining, so the profile carries post-inline call context. With full LTO (ducksteps uses --enable-lto=full), inlining happens at link time inside the LTO backend; so the CS instrumentation has to be injected there too.

How clang normally does that under LTO: at link it hands -plugin-opt=cs-profile-path= to the linker, and the LTO backend runs the post-inline instrumentation pass. That -plugin-opt= mechanism is an ELF / ld.lld feature. My toolchain is clang-cl + lld-link targeting COFF, and lld-link has no equivalent; nothing feeds "generate a CS profile" into the COFF LTO backend, so the pass never runs and the counters never get inserted.

It's a known, currently-open upstream bug: llvm/llvm-project#112103; "CS-IR PGO doesn't generate profiles with LTO enabled," reproduced on LLVM 17 and 19, silent failure, no fix landed.

🧪 What I tried, and what I ruled out

Approach	Result	Why
CS + full LTO (my actual config)	❌ no profile	The broken case... instrumentation needs the COFF link hook that doesn't exist. Reproduced on clang-cl 19.1.7.
CS + ThinLTO	❌ no profile	Tested it specifically as an escape hatch; ThinLTO alone doesn't unlock it on a stock toolchain.
CS without LTO	❌ pointless	Builds fine, but the context it captures is per-file inlining, which doesn't match the whole-program inlining my real (full-LTO) build does; wrong target, so the data is useless or worse. And using it would mean shipping a non-LTO browser, throwing away the LTO gains that are most of ducksteps' speedup.
BOLT / Propeller (post-link, context-aware optimizers)	❌ wrong OS	Both ELF/Linux-focused; no production Windows/COFF support.
AutoFDO (sample-based PGO via CPU branch samples)	❌ no tooling	Mature on Linux; basically absent on Windows, and worse on AMD.
Build a custom patched LLVM that adds the missing COFF plumbing	⚠️ out of scope	The genuinely-correct fix, but it means owning and maintaining a forked compiler on top of everything else; a permanent commitment that's beyond my current skillset and bandwidth as a solo dev. More on the "why not" under What would have to change below.

The repro, for anyone who wants to confirm it themselves (clang-cl 19.1.7, x86_64-pc-windows-msvc):

# Round 1: IR instrument (no LTO) -> ir.profdata
clang-cl -O2 /clang:-fprofile-generate -fuse-ld=lld a.c -o a_ir.exe
LLVM_PROFILE_FILE=ir.profraw ./a_ir.exe && llvm-profdata merge ir.profraw -o ir.profdata

# Round 2: CS instrument -> cs.profraw never appears, with EITHER:
clang-cl -O2 -flto      /clang:-fcs-profile-generate /clang:-fprofile-use=ir.profdata -fuse-ld=lld a.c -o a_cs.exe  # full LTO
clang-cl -O2 -flto=thin /clang:-fcs-profile-generate /clang:-fprofile-use=ir.profdata -fuse-ld=lld a.c -o a_cs.exe  # thin LTO
# LLVM_PROFILE_FILE=cs.profraw ./a_cs.exe  ->  no profile written in either case.

🛠️ "But Tete009 ships CSIR builds... why can't you?"

It's the kind of follow up question I was pondering myself, so I dug into it (with full respect, their work is an order of magnitude more advanced than mine). Their changelog confirms they've applied CSIR PGO since Firefox 136, on the same clang-cl 19.x line I'm on, so it isn't the compiler version.

Two differences stand out. They use ThinLTO where I use full LTO; but I tested ThinLTO above and it didn't unlock CSIR on a stock toolchain, so that alone doesn't explain it. The more likely factor: recent Tete009 builds run on a custom, self-compiled LLVM (21.x) with their own optimization pass plugins. That's exactly the kind of setup where you can wire up the missing COFF piece (or run a distributed-ThinLTO flow that sidesteps it). My best guess is that's the difference... they're building their own compiler; I'm using the stock one Mozilla ships. I can't see inside their build, so I'll leave it at "most likely," not a claim.

One footnote that applies to me as much as anyone: because of that open bug, CSIR can silently produce no data under LTO while the build still succeeds. So it's worth anyone running CSIR (myself included, if I ever get there) confirming the profile actually lands; not a comment on anyone's build, just the one gotcha this whole rabbit hole taught me.

🎯 Confidence

• HIGH that CSIR is blocked in ducksteps' current build on the stock Windows toolchain; it's an open upstream bug and I reproduced the silent failure twice on my exact compiler, under both full and thin LTO.
• MEDIUM on my Tete009 theory; the custom-LLVM explanation fits the evidence, but I can't see inside their build, so I won't state it as fact.
• LOW / open on whether any CSIR-under-LTO-on-Windows is truly contributing vs silently doing nothing, absent an explicit profile-data check.

🔮 What would have to change for this to become possible

• Clean path: LLVM gains a way to drive CS-PGO through lld-link's LTO backend on COFF (i.e. #112103 actually gets fixed), AND that lands in the clang version Mozilla bootstraps. Then it's close to a drop-in.
• DIY path: I build and maintain my own patched LLVM, the way Tete009 does. Technically possible; honestly not for me right now, and I'd rather be upfront about why. It means running a forked compiler that has to stay in lockstep with two separate fast-moving projects at once: Firefox/ESR on its roughly monthly security cadence with random security update drops in-between, and LLVM itself, which ships a new major version about every six months and reshuffles its internals between them. Every time either one moves, I'd be re-fitting my patches and rebuilding the entire compiler from scratch (hours each time) before I could even start a normal ducksteps build... a recurring, unpredictable draw on time I don't reliably have. Hacking on LLVM's internals is also a deeper kind of programming than I'm comfortable with on my own; and if one of those custom builds broke somewhere down in the toolchain, there's a real chance I couldn't diagnose or fix it myself, which is a bad property for a browser people are trusting to ship security updates on time. The honest core of it: for Tete009 the compiler is the craft, and they're clearly excellent at it. For me it isn't a passion, it's a means to an end; signing up for a permanent second project to maintain, one I'm not confident I could keep healthy, isn't a trade I want to make. I'd rather spend the limited time I've got shipping updates as close to on schedule as possible.
• Compromise path: drop LTO entirely and do non-LTO CSIR; but that throws away the gains LTO provides, becoming a net loss and I don't want to do that.

Realistically: this goes back on the menu the day the toolchain supports it. Until then, it's sadly parked; not forgotten.

🤖 Full disclosure

I leaned on Claude Opus 4.8 (Max effort, adaptive thinking) to dig through the toolchain internals and run down the failure mode. AI moves fast and so does LLVM, so newer info or a software advance could absolutely flip this from "no" to "yes" later. If you or anyone else know something I don't here, I'm all ears... I love being wrong if I can learn & improve from it!

This dead end taught me a ton, and I'd never have found Tete009's approach without the nudge. Thanks again for making me aware of CSIR's existence. Keep any new ideas coming. 🦆