Skip to content

[config-setup] Fail fast and loud when 'config load_minigraph' fails - #29293

Open
louis-nexthop wants to merge 1 commit into
sonic-net:masterfrom
nexthop-ai:config-setup-load-minigraph-fail-fast
Open

[config-setup] Fail fast and loud when 'config load_minigraph' fails#29293
louis-nexthop wants to merge 1 commit into
sonic-net:masterfrom
nexthop-ai:config-setup-load-minigraph-fail-fast

Conversation

@louis-nexthop

@louis-nexthop louis-nexthop commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Why I did it

Fixes #29290.

A failure to load /etc/sonic/minigraph.xml is critical:

  1. As explained in ConfigDbConnector can hang forever and ignore SIGINT (e.g. Ctrl+C, or 'kill -SIGINT <pid>' does not work) sonic-swss-common#1244, it will cause any CLIs or services that make connection to CONFIG_DB to hang forever. This includes config save -f command that config-setup tries to run after loading the minigraph.
  2. As a result, other services that suppose to start after config-setup.service do not get a chance to start. This includes pmon, swss, syncd, and etc.
Work item tracking
  • Microsoft ADO (number only):

How I did it

When config load_minigraph fails..

  1. Log critical error. The log also explains the impact of the failure, so operators have easier time to debug.
  2. Exit the script with error, so the config-setup.service fails at least once, so that the error is loud.

How to verify it

Reproduction steps are in #29290.

After the changes:

  • Error is loud and impact & recovery step are communicated as a CRIT log.
admin@humm109:~$ sudo grep config-setup /var/log/syslog
2026 Sep  2 01:00:06.274533 humm109 INFO config-setup[1787]: No config_db.json found but minigraph.xml is available, using minigraph...
2026 Sep  2 01:00:06.274568 humm109 INFO config-setup[1787]: Reloading minigraph...
2026 Sep  2 01:00:06.274832 humm109 INFO config-setup[1867]: Acquired lock on /etc/sonic/reload.lock
2026 Sep  2 01:00:06.274853 humm109 INFO config-setup[1867]: Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json --write-to-db
2026 Sep  2 01:00:06.275124 humm109 INFO config-setup[1873]: Traceback (most recent call last):
2026 Sep  2 01:00:06.275142 humm109 INFO config-setup[1873]:   File "/usr/local/bin/sonic-cfggen", line 580, in <module>
2026 Sep  2 01:00:06.275168 humm109 INFO config-setup[1873]:     main()
2026 Sep  2 01:00:06.275249 humm109 INFO config-setup[1873]:     ~~~~^^
2026 Sep  2 01:00:06.275348 humm109 INFO config-setup[1873]:   File "/usr/local/bin/sonic-cfggen", line 454, in main
2026 Sep  2 01:00:06.275381 humm109 INFO config-setup[1873]:     deep_update(data, parse_xml(minigraph, platform, asic_name=asic_name))
2026 Sep  2 01:00:06.275409 humm109 INFO config-setup[1873]:                       ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026 Sep  2 01:00:06.275436 humm109 INFO config-setup[1873]:   File "/usr/local/lib/python3.13/dist-packages/minigraph.py", line 1979, in parse_xml
2026 Sep  2 01:00:06.275462 humm109 INFO config-setup[1873]:     root = ET.parse(filename).getroot()
2026 Sep  2 01:00:06.275488 humm109 INFO config-setup[1873]:            ~~~~~~~~^^^^^^^^^^
2026 Sep  2 01:00:06.275516 humm109 INFO config-setup[1873]:   File "src/lxml/etree.pyx", line 3590, in lxml.etree.parse
2026 Sep  2 01:00:06.275542 humm109 INFO config-setup[1873]:   File "src/lxml/parser.pxi", line 1958, in lxml.etree._parseDocument
2026 Sep  2 01:00:06.275568 humm109 INFO config-setup[1873]:   File "src/lxml/parser.pxi", line 1984, in lxml.etree._parseDocumentFromURL
2026 Sep  2 01:00:06.275594 humm109 INFO config-setup[1873]:   File "src/lxml/parser.pxi", line 1887, in lxml.etree._parseDocFromFile
2026 Sep  2 01:00:06.275620 humm109 INFO config-setup[1873]:   File "src/lxml/parser.pxi", line 1200, in lxml.etree._BaseParser._parseDocFromFile
2026 Sep  2 01:00:06.275653 humm109 INFO config-setup[1873]:   File "src/lxml/parser.pxi", line 633, in lxml.etree._ParserContext._handleParseResultDoc
2026 Sep  2 01:00:06.275679 humm109 INFO config-setup[1873]:   File "src/lxml/parser.pxi", line 743, in lxml.etree._handleParseResult
2026 Sep  2 01:00:06.275705 humm109 INFO config-setup[1873]:   File "src/lxml/parser.pxi", line 672, in lxml.etree._raiseParseError
2026 Sep  2 01:00:06.275731 humm109 INFO config-setup[1873]:   File "/etc/sonic/minigraph.xml", line 2
2026 Sep  2 01:00:06.275757 humm109 INFO config-setup[1873]: lxml.etree.XMLSyntaxError: Premature end of data in tag DeviceMiniGraph line 1, line 2, column 1
2026 Sep  2 01:00:06.276145 humm109 INFO config-setup[1867]: Released lock on /etc/sonic/reload.lock
2026 Sep  2 01:00:06.276305 humm109 CRIT config-setup[1787]: 'config load_minigraph' failed (rc=1). CONFIG_DB is empty/partial and CONFIG_DB_INITIALIZED is unset: expect most SONiC CLI commands ('show', 'config'), swss/syncd startup, and interface bring-up to fail or hang until this is fixed. Root cause is in the load_minigraph output above (check 'journalctl -u config-setup' and /etc/sonic/minigraph.xml or golden_config_db.json validity), NOT in the downstream services that report errors.
2026 Sep  2 01:00:06.278159 humm109 INFO systemd[1]: Starting config-setup.service - Config initialization and migration service...
2026 Sep  2 01:00:06.278271 humm109 NOTICE systemd[1]: config-setup.service: Main process exited, code=exited, status=1/FAILURE
2026 Sep  2 01:00:06.278300 humm109 WARNING systemd[1]: config-setup.service: Failed with result 'exit-code'.
2026 Sep  2 01:00:06.278324 humm109 ERR systemd[1]: Failed to start config-setup.service - Config initialization and migration service.
2026 Sep  2 01:00:06.388411 humm109 INFO systemd[1]: config-setup.service: Consumed 445ms CPU time, 71.3M memory peak.
2026 Sep  2 01:00:55.652358 humm109 INFO systemd[1]: Starting config-setup.service - Config initialization and migration service...
2026 Sep  2 01:00:55.701981 humm109 INFO config-setup[2256]: No config_db.json found but minigraph.xml is available, using minigraph...
2026 Sep  2 01:00:55.702052 humm109 INFO config-setup[2256]: Reloading minigraph...

Which release branch to backport (provide reason below if selected)

  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202512
  • 202605
  • 202608

Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO):
Failure type:

Tested branch

  • master
  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202512
  • 202605
  • 202608
  • N/A

Test result

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: louis-nexthop <louis@nexthop.ai>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@mssonicbld

Copy link
Copy Markdown
Collaborator

This PR has backport request for branch(es): 202511,202512,202605.
Added label(s) for branch(es) 202511,msft-202512,202605.

---Powered by SONiC BuildBot

@mssonicbld

Copy link
Copy Markdown
Collaborator

This PR has backport request label(s) for branch(es): 202511,202605,msft-202512, but is missing required test information. Please make sure you tick the tested branch(es) in the Tested branch section and provide test evidence (e.g., 202511: <test result>) in the Test result section as well in your PR description.

---Powered by SONiC BuildBot

nh-grecs Bot pushed a commit to nexthop-ai/sonic-buildimage that referenced this pull request Sep 3, 2026
rebuild-source: sonic-net#29293 @ nexthop-ai/sonic-buildimage 220517e [case: upstream:open]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: [config-setup] Failure to load minigraph can hang the service without any clues

2 participants