[Discussion] DAG-scoped import errors for factory/driver files (e.g. dag-factory)

[Prajwal7842]

Summary

Hi folks, starting this discussion to discuss “dag-scoped import errors” for driver files like dag-factory.
TL’DR: There is no way to surface import errors as well as healthy dags from a driver (like dag-factory), if some configs passed into the generator are wrong. It is either “all” or “none”. I would like to discuss potential solutions (or would like to know if it already exists or is planned).

Details

When a single Python file generates many DAGs — as dag-factory and similar "driver file" patterns do — Airflow's import-error handling is file-scoped, not DAG-scoped. One bad config in a factory that builds hundreds of DAGs currently forces an all-or-nothing outcome.
If the factory lets the exception propagate, the whole file fails to import and every DAG it would have produced disappears (has_import_error=True, is_stale=True in dag-processor).
If the factory swallows the bad config to keep the others alive, the failure is silent — no import error is surfaced to the author at all.
There is no way to say "these 3 configs are broken, here is why, but the other 497 DAGs are healthy and should keep running." I'd like to discuss a contract that makes that possible.

Current Behavior:

Import errors are keyed by file, one string per file.
Staleness is applied to every DAG sharing the file's fileloc.

Proposed Solution:

A contract between dag-processor and driver file to return healthy dags as well as import errors (per file or per dag, given it is not necessary that dag_id is available)

Questions

Does something like this already exist / is it planned?
Per-file concatenation preserves file level import errors. Is the solution acceptable ?

[hamedrabah]

Airflow's parser unit is the Python file. An uncaught exception means that module did not finish importing, so Airflow cannot reliably treat some DAG objects from that module as healthy while also recording a native import failure for the file. The stale-DAG behavior follows from that boundary.

Today, a factory that must preserve healthy DAGs has two practical choices:

1. Catch errors per configuration, register the valid DAGs, and emit explicit diagnostics for invalid configurations through logs/metrics or a small diagnostic DAG.
2. Generate separate driver files, one per configuration or bounded group. Then Airflow's existing file-level import error maps cleanly to the affected DAGs and leaves the other files healthy.

The second option gives the strongest isolation. Catching exceptions in one large driver keeps 497 DAGs alive, but the failures cannot also appear as native import errors without a new API.

I would not overload the existing file import-error field with partial-success semantics because schedulers currently use that state to decide whether every DAG from the file is stale. A new structured “bundle/factory diagnostics” channel could support partial success without changing what an import error means.