Merge/ascend dev0824 to main0824 Ascend 合入后修复批次：上游 PR #1349 审查意见 + dev_0824 PR #39/#40 迁移 - #42
Closed
QQzhouQ wants to merge 65 commits into
Closed
Conversation
…pport on different components
…ng-block-size fix(ascend): 为AscendDecodeAttnOp补充缺失的block_size参数
Signed-off-by: menogrey <1299267905@qq.com>
Signed-off-by: menogrey <1299267905@qq.com>
Fix std::regex memory error at runtime
…sion fix(ascend-decode): fix off-by-one context_lens precision bug
1. Prefill: actual_seq_lengths_kv取消cumsum累加，使用原始kv序列长度，多并发kv长度越界报错 2. Decode: context_lens区分阶段取值，优先sequence_lengths+1，规避prefix_lengths空张量相加尺寸不匹配；actual_seq_kv同步移除cumsum
AscendAttn: Fix prefill/decode actual_seq_kv & context_lens compute bug
feat(ascend): adapt BlockPool, MemoryLayoutStrategy, CudaSampleOp and…
新增apply_top_k_top_p_custom算子接入与npu侧sampleGreedy函数实现
- Remove separate_kv_cache mechanism from C++ cache layer (BlockPool, MemoryLayoutStrategy, CacheConfig, OpDefs, ConfigInit, etc.) - Restore combined KV buffer layout [blocks, 2, seq, heads, dim] (BSND) - FIA read path: switch to npu_fused_infer_attention_score_v2 with 3D cache format - KV cache write: use npu_scatter_pa_kv_cache (cache_mode=Norm, SISO path) - Fix key/value non-contiguity from torch.split by adding .contiguous() at source - Fix from_blob NPU format issue: use narrow().view().reshape() instead of torch::from_blob - Fix warmup path: support() and __init__ handle kv_cache=None - op-plugin: change ScatterPaKvCache cache_mode from PA_NZ to Norm for ND format support
…tter_pa_kv_cache - Port aclnn_apply_top_k_top_p_custom AscendC custom op from upstream PR#13 - Replace CPU fallback sampleGreedy with NPU-accelerated implementation using applyTopKTopP for top-k/top-p filtering + multinomial sampling - Switch FIA read path to npu_fused_infer_attention_score_v2 with 3D cache - C++ kv_cache_base reshape to BSND [blocks, 2, seq, heads, dim] - KV cache write using npu_scatter_pa_kv_cache (cache_mode=Norm) - Fix key/value non-contiguity from torch.split with .contiguous() - op-plugin ScatterPaKvCache cache_mode: PA_NZ -> Norm - Add ascend/ops/ bindings (NPUBridge, op_api_common, ascend_apply_top_k_top_p_custom) - Update core/BUILD deps for npu_bridge + aclnn_custom_ops - Fix namespace conflict: close rtp_llm namespace before torch_npu includes
# Conflicts: # .bazelrc # BUILD # deps/git.bzl # deps/pip.bzl # rtp_llm/config/server_config_setup.py # rtp_llm/cpp/cache/BlockPool.cc # rtp_llm/cpp/cache/BlockPool.h # rtp_llm/cpp/cache/BlockPoolConfig.h # rtp_llm/cpp/cache/BlockPoolConfigHelper.h # rtp_llm/cpp/cache/BufferTypes.h # rtp_llm/cpp/cache/CacheConfig.h # rtp_llm/cpp/cache/CacheConfigCreator.cc # rtp_llm/cpp/cache/HybridConfigCreator.cc # rtp_llm/cpp/cache/HybridTypeKVCacheAllocator.cc # rtp_llm/cpp/cache/KVCacheAllocator.cc # rtp_llm/cpp/cache/KVCacheManager.cc # rtp_llm/cpp/cache/MemoryLayoutStrategy.cc # rtp_llm/cpp/cache/MemoryLayoutStrategy.h # rtp_llm/cpp/cache/SingleConfigCreator.cc # rtp_llm/cpp/cache/SingleTypeKVCacheAllocator.cc # rtp_llm/cpp/cache/Types.h # rtp_llm/cpp/config/ConfigModules.cc # rtp_llm/cpp/config/ConfigModules.h # rtp_llm/cpp/models/PyWrappedModel.h # rtp_llm/cpp/pybind/ConfigInit.cc # rtp_llm/models_py/bindings/OpDefs.cc # rtp_llm/models_py/bindings/OpDefs.h # rtp_llm/models_py/bindings/core/CudaSampleOp.cc # rtp_llm/models_py/model_desc/module_base.py # rtp_llm/models_py/modules/factory/attention/ascend_impl/ascend_decode.py # rtp_llm/models_py/modules/factory/attention/ascend_impl/ascend_kv_cache_write_op.py # rtp_llm/models_py/modules/factory/attention/ascend_impl/ascend_prefill.py # rtp_llm/models_py/modules/factory/attention/ascend_impl/ascend_rope_emb.py # rtp_llm/models_py/modules/factory/attention/attn_factory.py
… docs - Restore .bazelrc PYTHON_BIN_PATH to /opt/conda310/bin/python3 (upstream) - Restore BUILD interpreter_path to /opt/conda310/bin/python (upstream) - Restore deps/pip.bzl python_interpreter to /opt/conda310/bin/python3 (upstream) - Remove all added .md documentation files (adaptation plans, case studies, etc.) - Keep all functional code changes (FIA v2, scatter_pa_kv_cache, BSND layout, etc.)
重构：KV cache支持非连续
Merge the Ascend NPU adaptation branch (rollback state, without aclgraph graph-mode and profiler features) into the upstream-synced main branch. Conflict resolution principles: - Keep upstream (main) evolutions: cache topology restructuring, async MTP/speculative machinery, fusedCopy, allocBuf-based tpSync, OpDefs field naming (cu_seqlens_device etc.), new BUILD targets. - Keep dev_0824 Ascend adaptations: ascend build config/registry (using_ascend, arch_select, ascend bindings/modules), FIA v2 attention (ascend_prefill/decode), MEMORY_NPU, privateuse1 device paths. - Fuse both where needed: device-aware cuda_i32/toCuda* helpers, ascend-aware getLayerCache buffers, batch copy fallback, sampler, gatherers (is_cuda() || is_privateuseone() guards). - aclgraph (rtp_llm/cpp/ascend_graph) excluded per rollback; only the default-off capture flag stub remains.
… into remote_main_0824 Build/deps: - Dual-platform ascend pip locks (x86/arm) + using_ascend_arm config branch - Drop ARM march copt from ascend config; exclude flash_attn for ascend multimodal - vocab_prune_kernels: skip .cu srcs on ascend (same as other cuda kernel targets) Compile fixes (ascend branches/stubs for main-introduced code): - arch_select.bzl: close flashinfer_deps select; add ascend_arm branches - CudaSampleOp: drop stale chainSpeculativeSampling stub, add sampleFromProbs/rejectionSampling/mappingDraft2Target ascend stubs - CudaOps: avoid runtimeBatchCopy redefinition, add runtimeApplyPackedMaskLogits on ascend, maybe-unused helpers - NormalModelInputGatherer: drop stray #endif, ascend-aware mm transfers - cuda_graph shims: GraphStreamGuard/toGraphStream/setDevice/graphCaptureBegin ascend variants + graphRecordEvent/graphBlockEvent helpers - PyWrappedModel: ascend-aware device transfers (kPrivateUse1) across forward/embedding/multimodal/micro-batch paths; unconditional host block table assignment for ascend attention params - OpDefs: KVCache assignable (non-const layout member), ascend BSND per-block MHA layout, PyAttentionInputs *_host fields + bindings - BlockPool MEMORY_NPU case; ModelTypes is_privateuseone; AsyncRunner/Mtp default NPU stream ctor on ascend; PrefixToCandidateTokens lambda deleter Link fixes: - th_compute_lib: link exec_ops_srcs + exec_ctx_ops on ascend so librtp_compute_ops.so exports registerExecCtxOps/sampling symbols - registerExecCtxOps default visibility Runtime fixes: - ChatService/OpenaiEndpoint: null chat_render guards (raw HTTP endpoint) - NormalOutputDispatcher: D2H recognizes privateuseone tensors (blocking copy) - ascend_attn_params: direct kv_cache_block_id_host access - .pyi: drop stale separate K/V cache stubs
del: 将自定义AscendC算子迁移至外部独立仓管理
…-dev0824-to-main0824
…ingweikang#40 port) Bazel glob does not match subpackages that contain their own BUILD file, so fused_moe/impl/ascend/strategy was silently dropped from the wheel and the module failed to import at runtime. - models_py/BUILD: explicitly depend on ascend_moe in the using_ascend branch (same treatment as ascend_base) - fused_moe/impl/ascend/BUILD: drop strategy/*.py from glob (subpackage), depend on strategy:ascend_strategies instead Note: the third change of upstream PR ningweikang#40 (cache_mode="Norm" in ascend_kv_cache_write_op) was already present on this branch.
…point Restores ChatService.cc and OpenaiEndpoint.cc to upstream main state. The guards were only needed for the bare backend HTTP endpoint (port start_port+5) whose chat renderer is None by design (chat completions are served by the frontend processes). Normal frontend usage never traverses these paths, so the extra null checks add no value upstream. Eager inference re-verified on Ascend after the revert.
…upstream helpers Review-driven cleanups: - PyAttentionInputs host-tensor naming: adapt ascend Python (4 sites) to the upstream 5466baf rename (kv_cache_*_block_id = host mirror) and drop the C++ bridge fields; pyi/OpDefs.cc now byte-identical to upstream, the unconditional host-table assignments in PyWrappedModel are gone (upstream setupKVCacheForAttentionInputs already populates them). - BlockPool.cc / BlockPoolConfigHelper.h: restore upstream formatting and statements verbatim; only the three required Ascend changes remain (MEMORY_NPU case, buffer device select, where()). - MemoryLayoutStrategy.cc: restore upstream processKVTensor/processScaleTensor structure and route the three from_blob call sites through a small helper that uses at_npu::native::from_blob on Ascend (torch_npu variant preserves the ACL format tag that FIA/scatter kernels require); GPU path unchanged. - PyWrappedModel.cc / MtpBatchStreamProcessor.cc: replace 15+4 scattered #if USING_ASCEND device branches with the upstream getTorchCudaDevice() helper (zero remaining, CUDA behavior identical). - deps: drop the ARM-generated requirements_lock_ascend.txt and the whole pip_ascend_arm chain (pip.bzl/BUILD/WORKSPACE/arch_select/root BUILD/ http.bzl); rename the source to requirements_ascend_x86.txt and the compile target accordingly. requirements_lock_ascend_x86.txt stays a local build artifact. Verified: ascend build; eager inference re-check after each step.
…per in MemoryLayoutStrategy
…cation The Ascend fusedCopy/fusedStridedCopy hardcoded ACL_MEMCPY_HOST_TO_DEVICE. That matches the only reachable call site today (eager-path flush of host-pinned staging buffers into device tensors), but the CUDA kernel is direction-agnostic and the graph-mode call sites feed device-resident sources. Query aclrtPointerGetAttributes on the source pointer and select D2D when it lives on device, H2D otherwise — current behavior unchanged, future D2D reuse safe.
…pstream formatting
Device-select consolidation (review: "反复出现的 #if #else 集中修改"):
Replace 21 more "#if USING_ASCEND kPrivateUse1 #else kCUDA #endif" pairs
with the upstream getTorchCudaDevice() helper across PR-touched files:
- ModelTypes.cc (allocBuf device / gpu_packed)
- ExecOps.cc (execCreateMoeExpertStates moe_device); also drop the three
stale "TODO: Ascend" comments per review
- MtpExecutor.cc (grpc to_cuda lambda, fake SP buffers, five async-runner
ctor streams now built from getTorchCudaDevice())
- NormalModelInputGatherer / NormalSamplerInputGatherer / CudaCopyUtil /
cuda_graph_runner.{h,cc} / KVCacheMemoryConnector / PyWrappedModel.h
(residual tensor) / MockEngine.h / NormalBatchStreamProcessorTest
Kept as #if on purpose:
- RopeCache.cc (2): its bazel target deps are too light for ExecOps.h
- MtpExecutor runner ctor block: real divergence (shared default NPU
stream vs dedicated CUDA pool streams)
- ExecOps.cc helper definition itself
Fixes two artifacts of an earlier scripted fold: the getTorchCudaDevice
body had become self-recursive and a NormalOutputDispatcher statement
was mangled; both restored by hand.
Restore-upstream cleanups (review: keep main formatting/positions):
- CudaOps.cc: drop [[maybe_unused]] from applyPackedMaskLogitsCpuFallback
(all three platform branches call it now that the Ascend overload
exists); the two in CudaSampleOp.cc remain necessary
- linear/factory.py + linear/impl/cuda/__init__.py: byte-identical to
upstream again — keep the cutlass sys.path hack in factory.py where
main has it (dev had moved it during its LinearFactory refactor)
Verified: --config=ascend build; eager inference accuracy re-checked.
…klist items P0: - shims: restore ROCm graph.capture_begin() under #elif USING_ROCM (was silently no-op'd by the Ascend adaptation); restore the upstream single graphCaptureBegin declaration with an at::cuda::CUDAGraph forward decl so Ascend builds never include ATen/cuda headers - deps: commit requirements_lock_ascend_x86.txt (pip_ascend_torch loaded unconditionally from WORKSPACE/arch_select; missing lock broke loading on every config); drop the dangling root requirements_lock_ascend.txt symlink; export the ascend source/lock via exports_files; fix TAB in pip.bzl P1: - base_model: drop the ROCm "hip:N" device string branch (Ascend-only PR) - AsyncRunner: add no-arg sync() (host join + exception rethrow only); MtpExecutor (8 sites) and NormalExecutor now keep the host join on Ascend instead of compiling sync away; NormalExecutor runner binds to local_rank instead of hardcoded device 0 - shims: drop redundant makeGraphEvent Ascend branch; document that record/block are no-ops under single-stream Ascend and that enabling useStreamAsync would race until aclrtEvent-backed impls land; route NormalExecutor event record/block through the shared helpers - start_backend_server: fall back to single rank only when both CUDA and NPU are unavailable; add device_count()/is_ascend() reuse; pass device count explicitly into _get_cuda_device_list (removes the dir() dead branch); rank processes set ASCEND_RT_VISIBLE_DEVICES on Ascend - deps: real sha256 for torch_cpu_ascend/torch_npu_ascend wheels; full 40-char SHA for the aclnn git_repository - requirements_ascend_x86.txt: add torchvision==0.24.0 (unconditional //rtp_llm:torchvision dependency) Checklist: - cuda_graph_hdrs_lib / cuda_copy_util: declare exec_ops_hdr dep - aclnn BUILD: genrule out of cc_library deps (data/runfiles only), add target_compatible_with to the cc_library, extract SoC version constant - cuda_host_utils.h: drop the misleading empty Ascend branch - server_config_setup / start_backend_server: replace inline torch_npu probing with is_ascend()/device_count(); widen set_device except to (ImportError, OSError, AttributeError) with a warning - BUILD: gate -Bsymbolic-functions with if_ascend, fix "Fox"->"For" - drop five using_ascend select branches that equal //conditions:default - getTorchCudaDevice(): carry the current NPU device index - tests: remove the DEVICE_TYPE macro, call getTorchCudaDevice() directly - normalize #if/#endif indentation to column 0 in MtpExecutor/NormalExecutor Verified: --config=ascend build; default-config `bazel query //rtp_llm/...`; aclnn ops target build with artifacts; py_compile on all touched Python.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
承接已合并的 #41（上游 PR alibaba#1349 的审查对象
e9f7c1ed），本 PR 携带后续 9 个修复 commit，覆盖：1. 上游 PR alibaba#1349 审查意见（P0×2 / P1×6 / checklist 12 项）—
1be8a49aecapture_begin()恢复（Ascend 适配误伤修复）；ascend pip lock 提交 + 悬空软链删除 + 命名统一（全平台 loading 解除）hip:N；AsyncRunner 无参sync()（host join/异常回抛在 Ascend 保留）+ local_rank 落卡；makeGraphEvent 冗余分支删除 + 事件竞态告警；多卡 NPU 启动修复（device_count()/ASCEND_RT_VISIBLE_DEVICES）；http_archive sha256 + 完整 40 位 SHA；torchvision 补齐is_ascend()、Bsymbolic 平台 gate、空 select 分支清理、getTorchCudaDevice带设备 index、DEVICE_TYPE 宏删除、缩进归一等2. dev_0824 PR #39/#40 迁移 —
5a7f121d/4c47487e3. 中间审查轮次（fork 内 PR #41 的 review）—
437d52884/043af6535/57a0a9265/d86adb0b6验证
bazel build //rtp_llm:rtp_llm --config=ascend（Ascend 950）bazel query //rtp_llm/...全平台 loading