PaddleDetection-release_2_7_npu版本基于华为昇腾显卡训练模型，显卡的ai core使用率一直为0,也没有明显的报错

[chaikenny]

问题确认 Search before asking

• 我已经查询历史issue，没有发现相似的bug。I have searched the issues and found no similar bug report.

Bug组件 Bug Component

No response

Bug描述 Describe the Bug

系统信息如下：
系统信息如下：
root@administrator:/work/Paddle20260610/PaddleDetection-release_2_7_npu# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
root@administrator:/work/Paddle20260610/PaddleDetection-release_2_7_npu# npu-smi info
+------------------------------------------------------------------------------------------------+
| npu-smi 25.5.1 Version: 25.5.1 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 0 910B4 | OK | 87.2 51 0 / 0 |
| 0 | 0000:C1:00.0 | 0 0 / 0 2658 / 32768 |
+===========================+===============+====================================================+
| 1 910B4 | OK | 91.3 52 0 / 0 |
| 0 | 0000:C2:00.0 | 0 0 / 0 2658 / 32768 |
+===========================+===============+====================================================+
| 2 910B4 | OK | 81.5 53 0 / 0 |
| 0 | 0000:81:00.0 | 0 0 / 0 2658 / 32768 |
+===========================+===============+====================================================+
| 3 910B4 | OK | 90.2 55 0 / 0 |
| 0 | 0000:82:00.0 | 0 0 / 0 2658 / 32768 |
+===========================+===============+====================================================+
| 4 910B4 | OK | 92.2 56 0 / 0 |
| 0 | 0000:01:00.0 | 0 0 / 0 2658 / 32768 |
+===========================+===============+====================================================+
| 5 910B4 | OK | 90.1 56 0 / 0 |
| 0 | 0000:02:00.0 | 0 0 / 0 2659 / 32768 |
+===========================+===============+====================================================+
| 6 910B4 | OK | 89.0 56 0 / 0 |
| 0 | 0000:41:00.0 | 0 0 / 0 2658 / 32768 |
+===========================+===============+====================================================+
| 7 910B4 | OK | 85.2 57 0 / 0 |
| 0 | 0000:42:00.0 | 0 0 / 0 2659 / 32768 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| No running processes found in NPU 0 |
+===========================+===============+====================================================+
| No running processes found in NPU 1 |
+===========================+===============+====================================================+
| No running processes found in NPU 2 |
+===========================+===============+====================================================+
| No running processes found in NPU 3 |
+===========================+===============+====================================================+
| No running processes found in NPU 4 |
+===========================+===============+====================================================+
| No running processes found in NPU 5 |
+===========================+===============+====================================================+
| No running processes found in NPU 6 |
+===========================+===============+====================================================+
| No running processes found in NPU 7 |
+===========================+===============+====================================================+
root@administrator:/work/Paddle20260610/PaddleDetection-release_2_7_npu#
root@administrator:/work/Paddle20260610/PaddleDetection-release_2_7_npu#
root@administrator:/work/Paddle20260610/PaddleDetection-release_2_7_npu#
root@administrator:/work/Paddle20260610/PaddleDetection-release_2_7_npu#
root@administrator:/work/Paddle20260610/PaddleDetection-release_2_7_npu#
root@administrator:/work/Paddle20260610/PaddleDetection-release_2_7_npu# python -c "import paddle; print(paddle.version)"
I0615 15:57:27.015686 853 init.cc:237] ENV [CUSTOM_DEVICE_ROOT]=/usr/local/lib/python3.10/dist-packages/paddle_custom_device
I0615 15:57:27.015751 853 init.cc:146] Try loading custom device libs from: [/usr/local/lib/python3.10/dist-packages/paddle_custom_device]
I0615 15:57:27.878111 853 custom_device_load.cc:52] Succeed in loading custom runtime in lib: /usr/local/lib/python3.10/dist-packages/paddle_custom_device/libpaddle-custom-npu.so
I0615 15:57:27.878191 853 custom_device_load.cc:59] Skipped lib [/usr/local/lib/python3.10/dist-packages/paddle_custom_device/libpaddle-custom-npu.so]: no custom engine Plugin symbol in this lib.
I0615 15:57:27.883788 853 custom_kernel.cc:63] Succeed in loading 358 custom kernel(s) from loaded lib(s), will be used like native ones.
I0615 15:57:27.884006 853 init.cc:158] Finished in LoadCustomDevice with libs_path: [/usr/local/lib/python3.10/dist-packages/paddle_custom_device]
I0615 15:57:27.884061 853 init.cc:243] CustomDevice: npu, visible devices count: 8
3.0.0
root@administrator:/work/Paddle20260610/PaddleDetection-release_2_7_npu#

PaddleDetection的版本：PaddleDetection-release_2_7_npu

复现环境 Environment

内容都在问题详情里列出来了

Bug描述确认 Bug description confirmation

• 我确认已经提供了Bug复现步骤、代码改动说明、以及环境信息，确认问题是可以复现的。I confirm that the bug replication steps, code change instructions, and environment information have been provided, and the problem can be reproduced.

是否愿意提交PR？ Are you willing to submit a PR?

• 我愿意提交PR！I'd like to help by submitting a PR!

[chaikenny]

这是执行训练的命令：python tools/train.py -c configs/ppyoloe/ppyoloe_plus_crn_s_80e_coco.yml
--eval -o use_npu=true amp=False batch_size=20 num_workers=4 epoch=100000 log_iter=1

[chaikenny]

这是我安装的paddle的命令：

先安装飞桨 CPU 安装包

pip install paddlepaddle -i https://www.paddlepaddle.org.cn/packages/nightly/cpu

再安装飞桨 NPU 插件包

pip install paddle-custom-npu -i https://www.paddlepaddle.org.cn/packages/nightly/npu

执行# 飞桨基础健康检查
python -c "import paddle; paddle.utils.run_check()" 报错如下：
I0616 16:11:36.902240 985 npu_op_runner.cc:413] CreateTensorDesc for OP: ApplyAdamD, data_type: 0, origin_format: 0, storage_format: 0, origin_dims: 1, storage_dims: 1
I0616 16:11:36.902249 985 npu_op_runner.cc:413] CreateTensorDesc for OP: ApplyAdamD, data_type: 0, origin_format: 0, storage_format: 0, origin_dims: 1, storage_dims: 1
I0616 16:11:36.902258 985 npu_op_runner.cc:413] CreateTensorDesc for OP: ApplyAdamD, data_type: 0, origin_format: 0, storage_format: 0, origin_dims: 2, 4, storage_dims: 2, 4
I0616 16:11:36.902272 985 npu_op_runner.cc:413] CreateTensorDesc for OP: ApplyAdamD, data_type: 0, origin_format: 0, storage_format: 0, origin_dims: 2, 4, storage_dims: 2, 4
I0616 16:11:36.902285 985 npu_op_runner.cc:413] CreateTensorDesc for OP: ApplyAdamD, data_type: 0, origin_format: 0, storage_format: 0, origin_dims: 2, 4, storage_dims: 2, 4
I0616 16:11:36.902294 985 npu_op_runner.cc:413] CreateTensorDesc for OP: ApplyAdamD, data_type: 0, origin_format: 0, storage_format: 0, origin_dims: 2, 4, storage_dims: 2, 4
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.10/dist-packages/paddle/utils/install_check.py", line 274, in run_check
_run_dygraph_single(use_cuda, use_xpu, use_custom, custom_device_name)
File "/usr/local/lib/python3.10/dist-packages/paddle/utils/install_check.py", line 120, in _run_dygraph_single
opt.step()
File "/usr/local/lib/python3.10/dist-packages/paddle/base/dygraph/base.py", line 388, in impl
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/paddle/base/wrapped_decorator.py", line 50, in wrapper
return decorated(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/paddle/base/framework.py", line 741, in impl
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/paddle/optimizer/adam.py", line 528, in step
optimize_ops = self._apply_optimize(
File "/usr/local/lib/python3.10/dist-packages/paddle/optimizer/optimizer.py", line 1748, in _apply_optimize
optimize_ops = self._create_optimization_pass(
File "/usr/local/lib/python3.10/dist-packages/paddle/optimizer/optimizer.py", line 1391, in _create_optimization_pass
self._append_optimize_op(
File "/usr/local/lib/python3.10/dist-packages/paddle/optimizer/adam.py", line 387, in _append_optimize_op
_, _, _, _, _, _, _ = C_ops.adam(
OSError: (External) ACL error, the error code is : 500001. (at /paddle/backends/npu/kernels/funcs/npu_op_runner.cc:456)