Skip to content

vulkan: add a barrier before record_clone writes the dst buffer - #7151

Open
kongxinjun wants to merge 1 commit into
Tencent:masterfrom
kongxinjun:vulkan-record-clone-barrier
Open

kongxinjun wants to merge 1 commit into
Tencent:masterfrom
kongxinjun:vulkan-record-clone-barrier

Conversation

@kongxinjun

Copy link
Copy Markdown
Contributor

在 Apple M5 Pro(macOS 26.6.2,MoltenVK 1.4.2)上开 VK_LAYER_KHRONOS_validation 的同步校验(validate_sync)跑 test_mat_batch,test_batch_forward_scale_external 会报两处:

SYNC-HAZARD-WRITE-AFTER-READ
vkCmdCopyBuffer(): WRITE_AFTER_READ hazard detected. vkCmdCopyBuffer writes to VkBuffer 0x390000000039, which was previously read by another vkCmdCopyBuffer command.
Buffer copy region 0: { offset = 24768, size = 64 }

SYNC-HAZARD-WRITE-AFTER-WRITE
vkCmdCopyBuffer(): WRITE_AFTER_WRITE hazard detected. vkCmdCopyBuffer writes to VkBuffer 0x390000000039, which was previously written by another vkCmdCopyBuffer command.
The current synchronization allows VK_ACCESS_2_SHADER_READ_BIT|VK_ACCESS_2_SHADER_STORAGE_WRITE_BIT accesses at VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT, but to prevent this hazard, it must allow VK_ACCESS_2_TRANSFER_WRITE_BIT accesses at VK_PIPELINE_STAGE_2_COPY_BIT.
Buffer copy region 0: { offset = 24832, size = 16 }

record_clone 对 src 会按 access_flags / stage_flags 加 barrier,对 dst 没有加(注释是 barrier device any @ null)。但 dst 不一定是新内存:录制过程中 VkMat 一释放,内存就回到 VkBlobAllocator,同一个命令缓冲里的下一次分配可能拿到同一段,而之前用到它的命令这时还没执行。

把 allocator.cpp 里注释掉的 VkBlobAllocator M/F 日志打开,再在 record_clone 里打印 src 和 dst 的偏移,出问题的那段是这样的(每个 batch 先把 data 和 scale 各 clone 一份,跑 scale,再 clone 到输出):

VkBlobAllocator M 0x390000000039 +24768 64
CLONE src 0x390000000039 +4096 4096 -> dst 0x390000000039 +24768 64      # batch 1 data
VkBlobAllocator M 0x390000000039 +24832 64
CLONE src 0x390000000039 +12288 16 -> dst 0x390000000039 +24832 16       # batch 1 scale,写 +24832
record_pipeline 0xca6fb53a0
CLONE src 0x390000000039 +24768 64 -> dst 0x390000000039 +16576 4096     # 读 +24768
VkBlobAllocator F 0x390000000039 +24832 64
VkBlobAllocator F 0x390000000039 +24768 64
VkBlobAllocator M 0x390000000039 +24768 64
CLONE src 0x390000000039 +8192 4096 -> dst 0x390000000039 +24768 64      # batch 2 data,写 +24768,WRITE_AFTER_READ
VkBlobAllocator M 0x390000000039 +24832 64
CLONE src 0x390000000039 +12288 16 -> dst 0x390000000039 +24832 16       # batch 2 scale,写 +24832,WRITE_AFTER_WRITE

dispatch 之间复用内存一般没问题:后一个 dispatch 读前一个的输出时会有 compute 到 compute 的 barrier,它的执行依赖覆盖了前面所有的 compute 命令。但 vkCmdCopyBuffer 在 transfer 阶段,这些 barrier 覆盖不到,所以两次 copy 落在同一段回收的内存上时,中间没有任何同步。

改动是在 record_clone 写 dst 之前加一个 buffer barrier,src 是 COMPUTE | TRANSFER 阶段的 SHADER_WRITE | TRANSFER_WRITE,dst 是 TRANSFER 阶段的 TRANSFER_WRITE。host 到 device 的 record_clone(const Mat&, VkMat&) 最后也走这里。record_clone(const VkImageMat&, VkMat&) 里是同样的写法,ncnn 里没有调用,就没有动。

改了以后同步校验不再报这两处。不开校验层时测试结果和改之前一样,只有 test_squeezenet 因为 #4742 的路径问题失败。在 MoltenVK 上没有看到输出出错,这个问题是校验层报出来的。

A VkMat released while recording returns its memory to the blob
allocator right away, so the dst of a buffer copy may be memory that
an earlier command in the same command buffer still reads or writes.
record_clone recorded no barrier for dst, and the compute barriers in
between do not cover the transfer stage.

Synchronization validation reports WRITE_AFTER_READ and
WRITE_AFTER_WRITE on vkCmdCopyBuffer in test_mat_batch.
@github-actions github-actions Bot added the core label Oct 20, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T09:52:25.599807Z 8106b2b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@tencent-adm

Copy link
Copy Markdown
Member

CLA assistant check
Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants