[mellanox] Add datetime seconds to the sai_failure_dump archive name - #29306
[mellanox] Add datetime seconds to the sai_failure_dump archive name#29306mdhoff-ms wants to merge 1 commit into
Conversation
Signed-off-by: Matt Hoffman <matthoffman@microsoft.com>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
🟡 Changes recommended
Second-level timestamps still allow collisions for same-second/concurrent invocations, so uniqueness (and the original failure mode) is not fully eliminated.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates the Mellanox syncd SAI failure dump naming scheme to reduce same-minute archive name collisions that can leave uncompressed .tar files in /var/log/sai_failure_dump, helping prevent /var/log from filling up.
Changes:
- Add seconds (
%S) to thesai_dump_nametimestamp format used byplatform_syncd_dump.sh.
File summaries
| File | Description |
|---|---|
| platform/mellanox/docker-syncd-mlnx/platform_syncd_dump.sh | Adds seconds to the timestamp used for SAI failure dump directory/archive naming to reduce same-minute collisions. |
Review details
Suppressed comments (2)
platform/mellanox/docker-syncd-mlnx/platform_syncd_dump.sh:30
- Unquoted variable expansions here can lead to word-splitting or globbing if DUMPDIR (or the computed path) ever contains whitespace or glob characters. Quoting these paths is a low-risk hardening change.
This issue also appears on line 29 of the same file.
sai_dump_path="${DUMPDIR}/$sai_dump_name"
mkdir -p $sai_dump_path
sai_dump_file="${sai_dump_path}/$sai_dump_name"
saisdkdump -f $sai_dump_file
platform/mellanox/docker-syncd-mlnx/platform_syncd_dump.sh:29
- Even with a unique dump name, these downstream commands still use partially/unquoted
$sai_dump_namearguments; quoting consistently (and usingrm --) avoids word-splitting/globbing issues if names/paths ever contain unexpected characters.
sai_dump_file="${sai_dump_path}/$sai_dump_name"
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # Source the platform specific dump file | ||
|
|
||
| sai_dump_name="sai_sdk_dump_$(date +"%m_%d_%Y_%I_%M_%p")" | ||
| sai_dump_name="sai_sdk_dump_$(date +"%m_%d_%Y_%I_%M_%S_%p")" |
There was a problem hiding this comment.
From my understanding, Syncd::processNotifySyncd calls swss::exec which blocks until the process exits, i.e. the dump is finished generation. At the highest level Syncd::processEvent holds a mutex while this happens so AFAIK concurrent invocation is not naturally possible.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Hi @dgsudharsan as I see you are the original author of this script, could you check this change and the associated issue? Please let me know if there is a better fix for this. Thanks! |
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
This PR has backport request for branch(es): 202605. ---Powered by SONiC BuildBot
|
|
The Tested branch section has been ticked and Test result is provided for branch(es): 202605. Added label(s): Tested for 202605 Branch. ---Powered by SONiC BuildBot
|
Why I did it
Mellanox
sai_failure_dumpname precision is only to the minute, meaning that if two dumps occur in the same minute there is a collision and gzip will refuse to archive the file. I have seen this on multiple devices such as the 4600C and the 4280 which leaves leftover uncompressed .tar files, leading to /var/log partition filling up.Work item tracking
How I did it
Added seconds to the timestamp used by
platform_syncd_dump.shfor example:Before:
sai_sdk_dump_09_02_2026_06_15_AMAfter:
sai_sdk_dump_09_02_2026_06_15_39_AMHow to verify it
Invoke the SAI failure dump script twice within the same minute:
Verify that each invocation creates a uniquely named
.tar.gzarchive and does not leave a new uncompressed.tarfile:Which release branch to backport (provide reason below if selected)
Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO):
Failure type:
Tested branch
Test result
Confirmed that seconds were included consistently in the generated directory, archive contents, and SDK dump filenames:
SDK generation and tar creation completed using the second-qualified path.
Tested image version: SONiC.202605
Description for the changelog
Added datetime seconds to the sai_failure_dump archive name.