Skip to content

[CONCEPT] config.mk as the Bazel DSL: zero-friction path to incremental builds - #4073

Closed
oharboe wants to merge 8 commits into
The-OpenROAD-Project:masterfrom
oharboe:bazel-orfs
Closed

oharboe wants to merge 8 commits into
The-OpenROAD-Project:masterfrom
oharboe:bazel-orfs

Conversation

@oharboe

@oharboe oharboe commented Mar 29, 2026

Copy link
Copy Markdown
Collaborator

Proof of concept, run this to test:

bazelisk test designs/asap7/gcd/...

When the remainder of the issues are ground down, test all designs by:

bazelisk test ...

Problem

ORFS has two build systems that don't talk to each other:

  • Make: the production system. Every design has a config.mk. CI runs Make. Designers think in Make.
  • Bazel (via bazel-orfs): faster, cached, parallel — but requires hand-written BUILD files that duplicate config.mk content and drift out of sync.

Today, only 5 designs have Bazel BUILD files. The other 76 don't. Maintaining both is a tax nobody wants to pay.

Thesis

config.mk is already a DSL. Parse it, don't replace it.

Instead of asking designers to learn Bazel or maintain BUILD files, we parse config.mk at analysis time and generate orfs_flow() targets automatically. Designers keep writing config.mk — the same file they test with make — and get Bazel's caching and parallelism for free.

What this PR does

  1. Python parser reads config.mk and extracts variables into structured data (platform, design name, verilog files, flow arguments). 84 unit tests.

  2. Repository rule runs the parser on all config.mk files at fetch time. Uses repository_ctx.read() so Bazel watches every config.mk — edits trigger automatic re-parsing.

  3. orfs_design() macro — each design's BUILD.bazel is 3 lines:

    load("//bazel:orfs.bzl", "orfs_design")
    exports_files(glob(["*"]))
    orfs_design()

    The macro loads the parsed config, routes variables to the correct stages via variables.yaml metadata, and calls orfs_flow().

  4. Incremental caching works at the variable level. Changing CORE_UTILIZATION in config.mk rebuilds floorplan→place→route→final but skips synthesis — because variables.yaml says CORE_UTILIZATION doesn't affect synthesis. Verified: 2 sandbox actions, 11 cache hits.

Why this is low-churn

  • config.mk doesn't change. It's the source of truth, not a generated artifact. Designers and CI keep using it with Make exactly as before.

  • No new DSL to learn. The 3-line BUILD.bazel is boilerplate. The real configuration stays in config.mk.

  • Existing CI is unaffected. Make still works. Bazel is additive.

  • 81 designs get Bazel targets immediately. No per-design migration work. The parser handles the existing config.mk files as-is.

  • Tier 2 designs degrade gracefully. Designs with complex Make features (ifeq, $(if $(filter ...))) still get targets — the parser extracts what it can and filters unresolvable labels. The linter (python3 bazel/config_mk_parser.py --lint) shows what to simplify.

Why it works well

For designers: Change config.mk, run bazel build //flow/designs/asap7/gcd:gcd_place. Only the stages affected by your change rebuild. No Docker pull, no full rebuild.

For CI: Build all designs in parallel: bazel build //flow/designs/asap7/.... Bazel handles dependency tracking, caching, and parallelism. Shared remote cache means CI machines don't repeat work.

For maintainers: One source of truth (config.mk), one parser (84 tests), one macro (orfs_design). No hand-written BUILD files to maintain per design. Adding a new design = add config.mk + 3-line BUILD.bazel.

Commits (least churn first)

  1. Remove stale merge_lib/preprocessLib BUILD refs
  2. Add all 6 PDKs to flow/BUILD.bazel
  3. Add exports_files to all src/ directories
  4. config.mk parser + 84 unit tests
  5. orfs_design() macro + repo rule + design BUILD files
  6. README update
  7. Remove old hand-written BUILD files

Remaining work

Before bazelisk test flow/designs/asap7/... works with no caveats

Eliminate docker image dependency

  • Build OpenROAD from source via tools/OpenROAD/ submodule (has its own BUILD.bazel). Wire up orfs.default(openroad = "//tools/OpenROAD:openroad"). Currently uses docker image binary.
  • Build yosys from source via tools/yosys/ submodule. Wire up orfs.default(yosys = "//tools/yosys:yosys"). Currently uses docker image binary.
  • Build klayout from source or use system binary. Currently uses docker image binary.
  • System wrappers for make, tcl, ruby, opengl, qt_plugins, opensta — once OpenROAD/yosys/klayout build from source, these are the remaining docker dependencies. Need system-path wrappers or source builds.
  • Remove image and sha256 from MODULE.bazel once all tools build locally.
  • Switch MODULE.bazel from local_path_override to git_override pointing to bazel-orfs branch/PR.

Fix Tier 2 design analysis errors

  • cva6: multi-glob $(SRC_HOME) verilog files reference deep subdirectories without BUILD files
  • ibex: $(filter-out) pattern in VERILOG_FILES not fully resolved
  • jpeg/jpeg_lvt: VERILOG_INCLUDE_DIRS references include directories (not files)
  • riscv32i: ADDITIONAL_LEFS/LIBS from $(PLATFORM_DIR) inside ifeq conditionals
  • mock-alu/minimal/gcd-ccs: get DESIGN_NAME from include directives or don't set it

Fix designs that fail at build time

These need verification on a machine with enough cores/RAM:

  • aes: synthesis failure (needs investigation)
  • mock-cpu: synthesis failure
  • swerv_wrapper, riscv32i: synthesis failure (possibly missing verilog files)
  • ethmac, ethmac_lvt: OOM during CTS on small machines
  • aes_lvt, aes-block, aes-mbff: terminated during route (resource limits)

Push bazel-orfs changes

  • Create bazel-orfs PR with: zero-docker mode (stub repo), substeps leak fix, configurable variables_yaml, CONFIG_MAKE/CONFIG_OPENSTA in global config

Nice to have

  • Linter in CI — bazel run //bazel:config_mk_lint as informational check
  • Simplify Tier 2 config.mk files — file issues with specific migration instructions for designs using ifeq, $(if $(filter ...)), $(PLATFORM_DIR) in VERILOG_FILES
  • Remote cache — shared Bazel remote cache for CI
  • Auto-generate design BUILD files — instead of 81 committed files with identical 3-line content

oharboe and others added 7 commits March 29, 2026 23:13
These files were deleted in 644ff70 but the BUILD reference was
not updated.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
Add gf180, ihp-sg13g2, nangate45, sky130hs PDK definitions with
correct lib/ext patterns matching docker.BUILD.bazel.

Add synth*.v to makefile_yosys glob for synth_wrap_operators-booth.v.
Export scripts/variables.yaml for local variable metadata.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
Every flow/designs/src/<name>/ directory now has a BUILD.bazel with
exports_files and a :verilog filegroup, enabling both single-file
and glob references from orfs_flow() targets.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
Python parser for the config.mk design DSL that extracts structural
params (PLATFORM, DESIGN_NAME, VERILOG_FILES, SDC_FILE, BLOCKS) and
flow arguments (CORE_UTILIZATION, PLACE_DENSITY, etc.) for generating
orfs_flow() Bazel targets.

Handles all DSL features: export/=/?=/:=, line continuations,
$(wildcard), $(sort), variable resolution. Emits deprecation warnings
for complex features (ifeq, $(if), $(filter)).

84 unit tests including integration tests against all real config.mk
files in the repository.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
config.mk is the single source of truth for design configuration.
A repository rule parses all config.mk files and generates a DESIGNS
dict. Each design's BUILD.bazel calls orfs_design() which creates
orfs_flow() targets with properly routed stage arguments.

Key components:
- bazel/designs.bzl: repo rule that watches config.mk files via
  repository_ctx.read() and re-parses on changes
- bazel/orfs.bzl: orfs_design() macro with BLOCKS support, label
  filtering for Tier 2 designs, rules-base.json auto-detection
- MODULE.bazel: local bazel-orfs, docker image for yosys/klayout,
  local scripts/PDKs, configurable variables_yaml

Incremental caching verified: changing CORE_UTILIZATION rebuilds only
floorplan+place (2 actions), synthesis is cached (11 cache hits).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
Document the value proposition: think in Make, test with Make, get
Bazel caching and parallelism. Add sections on incremental caching
(changing CORE_UTILIZATION skips synthesis), quick start, and the
config.mk DSL.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
Replaced by auto-generated orfs_design() targets from config.mk.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
@oharboe
oharboe marked this pull request as draft March 29, 2026 21:40
@oharboe oharboe changed the title config.mk as the Bazel DSL: zero-friction path to incremental builds [CONCEPT] config.mk as the Bazel DSL: zero-friction path to incremental builds Mar 29, 2026
@oharboe

oharboe commented Mar 29, 2026

Copy link
Copy Markdown
Collaborator Author

@maliberty Thoughts?

Fix several issues preventing `bazelisk test //flow/designs/asap7/...`
from working:

Parser fixes (config_mk_parser.py):
- Add _tokenize_make_expr() to preserve $(...) nesting when splitting
  multi-value VERILOG_FILES (fixes wildcard resolution for ibex, cva6)
- Fix _dir_to_verilog_label() to preserve subdirectory paths instead
  of collapsing all to top-level :verilog label
- Fix _resolve_include_path() to find included files when repo rule
  runs from a different CWD (fixes mock-alu defaults.mk include)
- Skip tab-indented Make recipe lines (fixes mock-alu MOCK_ALU_WIDTH)

Bazel rule fixes (orfs.bzl):
- Add VERILOG_INCLUDE_DIRS directories as stage_data for synth so
  include files are available in the Bazel sandbox

New BUILD.bazel files:
- 3 missing design dirs: aes, ethmac_lvt, swerv_wrapper
- 21 cva6 src subdirectories for nested verilog/include files
- ibex, jpeg include directories

Disable orfs_design() for designs with pre-existing issues:
- cva6, minimal, mock-cpu, riscv32i, swerv_wrapper

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
@oharboe

oharboe commented Mar 30, 2026

Copy link
Copy Markdown
Collaborator Author

I'm still looking for comments on this, but I'll move the PR to bazel-orfs and patch up ORFS from there.

@oharboe oharboe closed this Mar 30, 2026
@maliberty

Copy link
Copy Markdown
Member

config.mk is only part of the configuration. Variables may be set in other files (eg platform, Makefile, etc). You would need to parse all of them including any conditionals. We already dump all the evaluated variables in make_issue so that would seem easier than trying to recreate make in python.

@oharboe

oharboe commented Mar 30, 2026

Copy link
Copy Markdown
Collaborator Author

config.mk is only part of the configuration. Variables may be set in other files (eg platform, Makefile, etc). You would need to parse all of them including any conditionals. We already dump all the evaluated variables in make_issue so that would seem easier than trying to recreate make in python.

Actually, the DSL is limited to designs/**/*.mk, the rest stays in make territory.

It is already working, I have all designs/**/*.mk wired up and working in bazel with linting + normal flow.

so, run the below and all ORFS designs are run and tested in Bazel:

bazelisk test ...

Obviously, I need to run through all of them and fix some nits, but that's mainly compute and a bit of tinkering.

My focus now is to create a standalone bazel-orfs PR demonstration and then queue up single concern PRs, mainly to ORFS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants