Skip to content

Commit f3c8f2f

Browse files
authored
fix(deepseek-v4): use full-prefix inference (#5509)
Signed-off-by: Chen Cui <chcui@nvidia.com>
1 parent 8a84d71 commit f3c8f2f

1 file changed

Lines changed: 3 additions & 0 deletions

File tree

‎examples/models/deepseek_v4/inference.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,13 +46,15 @@ if [[ -z "${PP:-}" ]]; then
4646
fi
4747
TP=1
4848

49+
# DSv4 hybrid attention does not support inference contexts yet, so use full-prefix generation.
4950
# Inference directly from the HF checkpoint (Bridge dequantises in-flight).
5051
uv run python -m torch.distributed.run --nproc_per_node=$((PP * EP)) \
5152
examples/conversion/hf_to_megatron_generate_text.py \
5253
--hf_model_path "${HF_MODEL_ID}" \
5354
--prompt "${PROMPT}" \
5455
--max_new_tokens 100 \
5556
--tp ${TP} --pp ${PP} --ep ${EP} \
57+
--legacy-full-prefix \
5658
--trust-remote-code
5759

5860
# Inference from a previously-imported Megatron checkpoint (faster cold start).
@@ -65,5 +67,6 @@ if [[ -d "${MEGATRON_DIR}/iter_0000000" ]]; then
6567
--prompt "${PROMPT}" \
6668
--max_new_tokens 100 \
6769
--tp ${TP} --pp ${PP} --ep ${EP} \
70+
--legacy-full-prefix \
6871
--trust-remote-code
6972
fi

0 commit comments

Comments
 (0)