Skip to content

fix(docs): clarify scaled dot-product attention rationale - #840

Open
lucasxing1 wants to merge 1 commit into
datawhalechina:mainfrom
lucasxing1:docs/clarify-attention-scaling
Open

fix(docs): clarify scaled dot-product attention rationale#840
lucasxing1 wants to merge 1 commit into
datawhalechina:mainfrom
lucasxing1:docs/clarify-attention-scaling

Conversation

@lucasxing1

Copy link
Copy Markdown

Summary

  • clarify why scaled dot-product attention divides logits by sqrt(d_k)
  • explain the link between larger unscaled dot products, Softmax saturation, and very small gradients
  • keep the Chinese and English Chapter 3 explanations aligned

Rationale

The previous wording stated that scaling prevents gradients from becoming too small, but did not explain the mechanism. The revised text follows the rationale in Attention Is All You Need: as d_k grows, unscaled dot products can grow in magnitude and push Softmax into saturated regions; scaling controls the score magnitude and stabilizes training.

Reference: https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf

Validation

  • reviewed the Chinese and English changes side by side
  • confirmed the branch contains only the two intended documentation files
  • ran git diff --check
  • no runtime tests were needed for this documentation-only change

Fixes #835

Explain how unscaled logits can saturate Softmax in both Chinese and English.

Refs datawhalechina#835
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[问题/Issue] 章节3.1.2:对自注意力公式的解释有疑问

1 participant