Usefulness of freezing modules. Is it actually important?

[saocorley]

I noticed DSPy supports freezing modules during optimization. I understand some possible motivations (reducing search space, preserving previously optimized components, etc.), but I haven't been able to find examples, tutorials, or research discussing when freezing is beneficial in practice. Can anyone provide some guidance on the intended use cases and whether there are benchmarks showing advantages of freezing versus optimizing all modules jointly?
Thanks in advance!

[rehan243]

Oh interesting, are you asking if freezing modules improves performance in a measurable way or just when it makes sense conceptually? I've used freezing in a few transfer learning setups, and yeah, it can be a game-changer if you're working with a pre-trained backbone or a model component that you trust.

The main use case I've seen is when you've got something like a large pre-trained encoder (e.g., a transformers model) and you're adding custom task-specific layers on top. Freezing the encoder during initial training lets your new layers learn faster without the noise of constantly updating the frozen base. Once those top layers stabilize, you can unfreeze and fine-tune end-to-end.

Here's a quick example in PyTorch that mimics a common pattern:

# Freeze encoder
for param in model.encoder.parameters():
    param.requires_grad = False

# Train top layers first
optimizer = torch.optim.Adam(model.top_layers.parameters(), lr=0.001)
# ... training loop ...

# Unfreeze encoder for fine-tuning
for param in model.encoder.parameters():
    param.requires_grad = True

# Now optimize everything
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)

On benchmarks: I've seen freezing cut training time significantly in early stages (less stuff to compute gradients for) without hurting final accuracy, especially for small-to-medium custom datasets. But if your dataset is huge or very different from the pre-trained domain, freezing might be counterproductive since the base model may need serious fine-tuning.

Curious: are you working with a specific architecture? Something like transformers, or maybe a tiered RNN setup? The trade-offs really depend on how generalizable your frozen components are.

[saocorley]

Thank you for your answer, but I think you didn't realize my question was about dspy, not general neural netoworks :)

[SpiliosDimakopoulos]

DSPy's freezing has nothing to do with gradients — it's about protecting a
module's prompt state (instructions + demos) from being touched by an
optimizer.

Mechanically: every dspy.Module has a _compiled flag. When you run
optimizer.compile(program, ...), the very first thing that happens is
student.reset_copy(), which deep-copies the program and clears any
inherited _compiled flags — except on submodules that were already
compiled and passed in as-is. After a successful .compile(), DSPy sets
_compiled = True on the returned module. Critically, named_parameters()
skips over any submodule that's already marked _compiled when it's nested
inside a bigger program.

So the actual use case isn't "does freezing help accuracy on its own" —
it's compositional optimization:

1. You optimize a submodule in isolation (e.g. a query-rewriter) and get
   back a compiled, frozen version of it.
2. You embed that frozen submodule inside a larger pipeline.
3. You run a second optimizer pass on the outer pipeline — the frozen
   submodule's demos/instructions are left untouched, and only the new
   components get optimized.

This is exactly what BetterTogether and multi-stage pipelines rely on.
Without freezing, re-running an optimizer on the outer program could
overwrite carefully-tuned demos in a submodule you already validated,
wasting your optimization budget re-solving a problem you'd already solved.

There isn't a published benchmark showing "freezing vs. not" in isolation
because it's not really an accuracy lever — it's a budget and stability
lever for incremental/compositional development. If you're optimizing a
single flat program end-to-end in one .compile() call, freezing won't
matter to you. It matters once you start building pipelines out of
pre-optimized pieces.

[saocorley]

Hello! Thank you for your answer. I was asking exactly that, if any formal research, discussions or benchmarks about weather composing pre trained (or compiled/optimized) pipelines is a good design decision :)

I would think that there is not, and the flag exists mostly based on peoples intuition of how dspy works.

Thanks anyway!