Skip to content

[mellanox] Add datetime seconds to the sai_failure_dump archive name - #29306

Open
mdhoff-ms wants to merge 1 commit into
masterfrom
dev/matthoffman/sai-failure-dump-seconds
Open

[mellanox] Add datetime seconds to the sai_failure_dump archive name#29306
mdhoff-ms wants to merge 1 commit into
masterfrom
dev/matthoffman/sai-failure-dump-seconds

Conversation

@mdhoff-ms

Copy link
Copy Markdown
Contributor

Why I did it

Mellanox sai_failure_dump name precision is only to the minute, meaning that if two dumps occur in the same minute there is a collision and gzip will refuse to archive the file. I have seen this on multiple devices such as the 4600C and the 4280 which leaves leftover uncompressed .tar files, leading to /var/log partition filling up.

Work item tracking

How I did it

Added seconds to the timestamp used by platform_syncd_dump.sh for example:
Before: sai_sdk_dump_09_02_2026_06_15_AM
After: sai_sdk_dump_09_02_2026_06_15_39_AM

How to verify it

Invoke the SAI failure dump script twice within the same minute:

sudo docker exec -w / syncd /usr/bin/sai_failure_dump.sh   
sudo docker exec -w / syncd /usr/bin/sai_failure_dump.sh

Verify that each invocation creates a uniquely named .tar.gz archive and does not leave a new uncompressed .tar file:

sudo ls -lh /var/log/sai_failure_dump/

Which release branch to backport (provide reason below if selected)

  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202512
  • 202605
  • 202608

Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO):
Failure type:

Tested branch

  • master
  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202512
  • 202605
  • 202608
  • N/A

Test result

  • 202605: Manually validated on a live Mellanox SN4280 DUT using the updated script. Invoked the production SAI failure dump entry point:
sudo docker exec -w / syncd /usr/bin/sai_failure_dump.sh

Confirmed that seconds were included consistently in the generated directory, archive contents, and SDK dump filenames:

 sai_sdk_dump_09_02_2026_06_15_39_AM

SDK generation and tar creation completed using the second-qualified path.

Tested image version: SONiC.202605

Description for the changelog

Added datetime seconds to the sai_failure_dump archive name.

Signed-off-by: Matt Hoffman <matthoffman@microsoft.com>
@mdhoff-ms
mdhoff-ms requested review from dgsudharsan and a lite review from Copilot September 2, 2026 06:41
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Second-level timestamps still allow collisions for same-second/concurrent invocations, so uniqueness (and the original failure mode) is not fully eliminated.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates the Mellanox syncd SAI failure dump naming scheme to reduce same-minute archive name collisions that can leave uncompressed .tar files in /var/log/sai_failure_dump, helping prevent /var/log from filling up.

Changes:

  • Add seconds (%S) to the sai_dump_name timestamp format used by platform_syncd_dump.sh.
File summaries
File Description
platform/mellanox/docker-syncd-mlnx/platform_syncd_dump.sh Adds seconds to the timestamp used for SAI failure dump directory/archive naming to reduce same-minute collisions.
Review details

Suppressed comments (2)

platform/mellanox/docker-syncd-mlnx/platform_syncd_dump.sh:30

  • Unquoted variable expansions here can lead to word-splitting or globbing if DUMPDIR (or the computed path) ever contains whitespace or glob characters. Quoting these paths is a low-risk hardening change.

This issue also appears on line 29 of the same file.

sai_dump_path="${DUMPDIR}/$sai_dump_name"
mkdir -p $sai_dump_path
sai_dump_file="${sai_dump_path}/$sai_dump_name"
saisdkdump -f $sai_dump_file

platform/mellanox/docker-syncd-mlnx/platform_syncd_dump.sh:29

  • Even with a unique dump name, these downstream commands still use partially/unquoted $sai_dump_name arguments; quoting consistently (and using rm --) avoids word-splitting/globbing issues if names/paths ever contain unexpected characters.
sai_dump_file="${sai_dump_path}/$sai_dump_name"
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

# Source the platform specific dump file

sai_dump_name="sai_sdk_dump_$(date +"%m_%d_%Y_%I_%M_%p")"
sai_dump_name="sai_sdk_dump_$(date +"%m_%d_%Y_%I_%M_%S_%p")"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my understanding, Syncd::processNotifySyncd calls swss::exec which blocks until the process exits, i.e. the dump is finished generation. At the highest level Syncd::processEvent holds a mutex while this happens so AFAIK concurrent invocation is not naturally possible.

@mdhoff-ms
mdhoff-ms marked this pull request as ready for review September 2, 2026 07:02
@mdhoff-ms
mdhoff-ms requested a review from lguohan as a code owner September 2, 2026 07:02
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@mdhoff-ms

Copy link
Copy Markdown
Contributor Author

Hi @dgsudharsan as I see you are the original author of this script, could you check this change and the associated issue? Please let me know if there is a better fix for this. Thanks!

@mdhoff-ms

Copy link
Copy Markdown
Contributor Author

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@mssonicbld

Copy link
Copy Markdown
Collaborator

This PR has backport request for branch(es): 202605.
Added label(s) for branch(es) 202605.

---Powered by SONiC BuildBot

@mssonicbld mssonicbld added the Tested for 202605 branch Tested for 202605 branch label Sep 2, 2026
@mssonicbld

Copy link
Copy Markdown
Collaborator

The Tested branch section has been ticked and Test result is provided for branch(es): 202605. Added label(s): Tested for 202605 Branch.

---Powered by SONiC BuildBot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants