Skip to content

Remove need for variables.json in git - #4780

Closed
oharboe wants to merge 3 commits into
The-OpenROAD-Project:masterfrom
oharboe:yaml2json
Closed

oharboe wants to merge 3 commits into
The-OpenROAD-Project:masterfrom
oharboe:yaml2json

Conversation

@oharboe

@oharboe oharboe commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Why: fewer papercuts in CI and locally. flow/scripts/variables.json is generated from variables.yaml and committed to git, CI fails when the two drift apart, and every installer carries PyYAML only to regenerate it. This removes the dependency, the committed .json and the check. Proposed in #4180.

How: the flow pipes variables.yaml through OpenROAD's yaml2json into its readers. variables.json, yaml_to_json.py, the sync step and PyYAML are deleted.

What:

  • docker: ship yaml2json next to openroad — the flow image copies it with openroad; the builder image stops when either is missing.
  • flow: read variables.yaml through yaml2json — $(YAML2JSON) variables.yaml | $(PYTHON_EXE) <reader>.py in variables.mk, util.tcl and the docs CI job; each reader fails on empty input. YAML2JSON defaults to yaml2json next to OPENROAD_EXE and is listed in variables.yaml. Deletes variables.json, its .gitattributes entry, yaml_to_json.py and the "Check variables.json is up to date" step. The variables.yaml CI job builds //src/yaml2json in tools/OpenROAD with bazelisk.
  • etc: drop PyYAML — from the pip requirements and locks (only pyyaml itself; nothing else depended on it), and python3-yaml from DependencyInstaller.sh.

Verified on this branch with yaml2json from #12102:

  • defaults.py prints the same export VAR?=value lines as from variables.json (sorted diff empty).
  • non_stage_variables.py lists the same variables for synth, floorplan, place, cts, grt, route and final.
  • generate-variables-docs.py regenerates FlowVariables.md byte-identical, apart from the new YAML2JSON row.
  • With no yaml2json, make stops with [ERROR][FLOW] Could not read the defaults in .../variables.yaml with YAML2JSON=... and erase_non_stage_variables errors with couldn't execute.

Draft: needs The-OpenROAD-Project/OpenROAD#12102 and an OpenROAD submodule bump that brings it in; this PR does not bump tools/OpenROAD. Until then the Docker builder check and the variables.yaml CI job fail, for want of yaml2json.

Nix is not updated in this PR; Nix users update flake.nix.

🤖 Generated with Claude Code

https://claude.ai/code/session_018PFLaG12qpV6aKy7cdfwXT

OpenROAD's install puts yaml2json in bin/ next to openroad. Copy it into
the flow image with openroad, and stop the builder image when either is
missing, so the flow can read variables.yaml without PyYAML on the host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018PFLaG12qpV6aKy7cdfwXT
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request removes the dependency on pyyaml and the pre-generated variables.json file by utilizing the yaml2json executable to convert variables.yaml to JSON on the fly, which is then parsed using Python's built-in json module. The feedback highlights an issue in flow/scripts/variables.mk where YAML2JSON may resolve incorrectly to ./yaml2json if OPENROAD_EXE is set to a bare command name without a directory path, and suggests a robust fallback logic to handle this case.

Comment thread flow/scripts/variables.mk Resolved
oharboe and others added 2 commits November 3, 2026 08:19
The readers of variables.yaml, defaults.py (variables.mk),
non_stage_variables.py (util.tcl) and generate-variables-docs.py, read
it as JSON on stdin from OpenROAD's yaml2json:

    $(YAML2JSON) variables.yaml | $(PYTHON_EXE) defaults.py

Each fails on empty input, so a missing or failing yaml2json stops make
and the stage instead of running with no defaults.

YAML2JSON defaults to yaml2json next to OPENROAD_EXE, where OpenROAD's
install puts it.

variables.json, yaml_to_json.py and the CI step that kept the two in
sync are deleted. The variables.yaml CI job builds yaml2json from
tools/OpenROAD to regenerate FlowVariables.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018PFLaG12qpV6aKy7cdfwXT
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
Nothing in the flow imports yaml any more. Remove pyyaml from the pip
requirements and their locks, and python3-yaml from
DependencyInstaller.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018PFLaG12qpV6aKy7cdfwXT
Signed-off-by: Øyvind Harboe <oyvind.harboe@zylin.com>
@oharboe oharboe changed the title Read variables.yaml through OpenROAD's yaml2json; drop variables.json and PyYAML Fewer dependencies and CI papercuts: no PyYAML, no committed variables.json Nov 3, 2026
@oharboe oharboe changed the title Fewer dependencies and CI papercuts: no PyYAML, no committed variables.json Remove need for variables.json in git Nov 3, 2026
@oharboe oharboe closed this Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant