Skip to content

Commit 14cb6b1

Browse files
committed
hicache and comments
Signed-off-by: Zhi Yiliu <2584074296@qq.com>
1 parent a0c91be commit 14cb6b1

10 files changed

Lines changed: 980 additions & 486 deletions

File tree

docs/llamacpp/img/load_weight.png

85.7 KB
Loading

docs/llamacpp/server_profile.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Nsys Profiler
2+
## Load Model
3+
- cudamemgetinfo 执行了 373 s,应该是 Orin 的内存探测有问题,可能是其他系统组件影响了这个函数的性能。
4+
5+
- 使用文件进行 mmap,所以会出现 cudamemasync 之后进行 cuda sync 的情况,导致性能下降。
6+
7+
![](img/load_weight.png)
8+
9+
## Forward
10+
**也会出现大量的 H2D 拷贝**
11+
主要拷贝的不是模型权重,也不是大块 KV cache,而是这些每个 ubatch 都会变化的输入:
12+
13+
- tokens: 当前 prefill micro-batch 的 token id
14+
- pos: 每个 token 的 position
15+
- k_idxs / v_idxs: 当前 token 要写入 KV cache 的 row/index
16+
- kq_mask: attention mask,和 n_tokens x n_kv 相关,prefill 时可能比较大
17+
- out_ids: 哪些 token 需要输出 logits
18+
- sampler/backend sampling 相关输入,如果启用了 backend sampler
19+
- 某些模型会有额外输入,比如 M-RoPE position、attention temperature、Hadamard rotation 表等
20+
- prefill: 2.57 s
21+
- D2H: 返回的 logits
22+
- H2D:这一轮的 batch 元数据
23+
- decode: 36.8 ms 左右
24+
- 使用了 CUDA Graph ,第一次 decode 需要捕获 graph,后续 decode 就不需要了,所以第一次 decode 的时间会比较长,后续 decode 的时间会比较短。
25+
- first decode: 39.351 ms
26+
- cuda graph decode: 36.8 ms
27+
1.55 MB
Loading

docs/paperreadings/llm/models/deepseekv4.md

Lines changed: 213 additions & 112 deletions
Large diffs are not rendered by default.

docs/sglang/hicache.md

Lines changed: 692 additions & 281 deletions
Large diffs are not rendered by default.

docs/sglang/img/hicahe_state.png

917 KB
Loading
1.17 MB
Loading

mkdocs.yml

Lines changed: 8 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -185,25 +185,13 @@ extra_css:
185185
# - css/ai-summary.css
186186

187187
extra:
188-
twikoo:
189-
# Tencent Cloudbase: envId can be the cloud environment ID.
190-
# Self-hosted: envId can be your deployed Twikoo service URL, e.g. https://twikoo.example.com
191-
env_id: REPLACE_WITH_TWIKOO_ENV_ID_OR_URL
192-
# region is optional and mainly used with Tencent Cloudbase deployments.
193-
# region: ap-shanghai
194-
giscus:
195-
repo: tom-jerr/tom-jerr.github.io
196-
repo_id: R_kgDONZrjcA
197-
category: General
198-
# Replace this with the real General category ID from https://giscus.app
199-
category_id: REPLACE_WITH_GENERAL_CATEGORY_ID
200-
mapping: pathname
201-
strict: 0
202-
reactions_enabled: 1
203-
emit_metadata: 1
204-
input_position: top
205-
theme: preferred_color_scheme
206-
lang: en
188+
utterances:
189+
repo: tom-jerr/blog-comments
190+
issue_term: pathname
191+
label: comments
192+
theme: preferred-color-scheme
193+
light_theme: github-light
194+
dark_theme: github-dark
207195
social:
208196
- icon: /fontawesome/brands/github
209197
link: https://github.com/tom-jerr/
@@ -384,7 +372,7 @@ nav:
384372
# - DiT Generate Model in SGLang: notes/diffusion/DiT Video Generate.md
385373
# - Masked Diffusion Model: notes/diffusion/Masked Diffusion LLM.md
386374
- SGLang 专题:
387-
- HiCache: sglang/hicahce.md
375+
- HiCache: sglang/hicache.md
388376
- PD Disaggregation in SGLang: sglang/PD Disaggregation in SGLang.md
389377
- EAGLE2 in SGLang: sglang/Eagle2 in SGLang.md
390378
- 一条 Request 在 SGLang 的前世今生: sglang/一条 Request 在 SGLang 的前世今生.md

overrides/partials/comments.html

Lines changed: 33 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -1,78 +1,57 @@
11
{% if page.meta.comments is not defined or page.meta.comments != false %}
2-
{% set twikoo = config.extra.twikoo %}
3-
{% set giscus = config.extra.giscus %}
4-
{% if twikoo and twikoo.env_id and not twikoo.env_id.startswith("REPLACE_WITH_") %}
5-
<div id="twikoo-comments"></div>
6-
<script src="https://cdn.jsdelivr.net/npm/twikoo@1.6.42/dist/twikoo.all.min.js"></script>
7-
<script>
8-
twikoo.init({
9-
envId: "{{ twikoo.env_id }}",
10-
el: "#twikoo-comments",
11-
{% if twikoo.region %}
12-
region: "{{ twikoo.region }}",
13-
{% endif %}
14-
path: "{{ page.url | default('') }}"
15-
})
16-
</script>
17-
{% elif giscus and giscus.repo and giscus.repo_id and giscus.category and giscus.category_id and not giscus.category_id.startswith("REPLACE_WITH_") %}
2+
{% set utterances = config.extra.utterances %}
3+
{% if utterances and utterances.repo and utterances.issue_term %}
184
<script
19-
src="https://giscus.app/client.js"
20-
data-repo="{{ giscus.repo }}"
21-
data-repo-id="{{ giscus.repo_id }}"
22-
data-category="{{ giscus.category }}"
23-
data-category-id="{{ giscus.category_id }}"
24-
data-mapping="{{ giscus.mapping | default('pathname') }}"
25-
data-strict="{{ giscus.strict | default('0') }}"
26-
data-reactions-enabled="{{ giscus.reactions_enabled | default('1') }}"
27-
data-emit-metadata="{{ giscus.emit_metadata | default('1') }}"
28-
data-input-position="{{ giscus.input_position | default('top') }}"
29-
data-theme="{{ giscus.theme | default('preferred_color_scheme') }}"
30-
data-lang="{{ giscus.lang | default('en') }}"
31-
data-loading="lazy"
5+
src="https://utteranc.es/client.js"
6+
repo="{{ utterances.repo }}"
7+
issue-term="{{ utterances.issue_term | default('pathname') }}"
8+
{% if utterances.label %}
9+
label="{{ utterances.label }}"
10+
{% endif %}
11+
theme="{{ utterances.theme | default('preferred-color-scheme') }}"
3212
crossorigin="anonymous"
3313
async>
3414
</script>
3515
<script>
36-
var giscus = document.querySelector("script[src*=giscus]")
16+
var utterances = document.querySelector("script[src*=utteranc]")
3717

38-
// Set palette on initial load
39-
var palette = __md_get("__palette")
40-
if (giscus && palette && typeof palette.color === "object") {
41-
var theme = palette.color.scheme === "slate"
42-
? "transparent_dark"
43-
: "light"
18+
function getUtterancesTheme() {
19+
if (typeof __md_get !== "function") {
20+
return "{{ utterances.theme | default('preferred-color-scheme') }}"
21+
}
4422

45-
// Instruct Giscus to set theme
46-
giscus.setAttribute("data-theme", theme)
23+
var palette = __md_get("__palette")
24+
if (palette && typeof palette.color === "object") {
25+
return palette.color.scheme === "slate"
26+
? "{{ utterances.dark_theme | default('github-dark') }}"
27+
: "{{ utterances.light_theme | default('github-light') }}"
28+
}
29+
30+
return "{{ utterances.theme | default('preferred-color-scheme') }}"
31+
}
32+
33+
if (utterances) {
34+
utterances.setAttribute("theme", getUtterancesTheme())
4735
}
4836

49-
// Register event handlers after documented loaded
5037
document.addEventListener("DOMContentLoaded", function () {
5138
var ref = document.querySelector("[data-md-component=palette]")
5239
if (!ref) {
5340
return
5441
}
5542

5643
ref.addEventListener("change", function () {
57-
var palette = __md_get("__palette")
58-
if (palette && typeof palette.color === "object") {
59-
var theme = palette.color.scheme === "slate"
60-
? "transparent_dark"
61-
: "light"
62-
63-
// Instruct Giscus to change theme
64-
var frame = document.querySelector(".giscus-frame")
65-
if (frame && frame.contentWindow) {
66-
frame.contentWindow.postMessage(
67-
{giscus: {setConfig: {theme}}},
68-
"https://giscus.app"
69-
)
70-
}
44+
var frame = document.querySelector(".utterances-frame")
45+
if (frame && frame.contentWindow) {
46+
frame.contentWindow.postMessage(
47+
{type: "set-theme", theme: getUtterancesTheme()},
48+
"https://utteranc.es"
49+
)
7150
}
7251
})
7352
})
7453
</script>
7554
{% else %}
76-
<!-- Comments are disabled until extra.twikoo.env_id or extra.giscus.category_id is configured. -->
55+
<!-- Comments are disabled until extra.utterances.repo and extra.utterances.issue_term are configured. -->
7756
{% endif %}
7857
{% endif %}

zensical.toml

Lines changed: 7 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -289,25 +289,13 @@ vins = "lucide/map"
289289

290290
[project.extra]
291291

292-
[project.extra.twikoo]
293-
# Tencent Cloudbase: env_id can be the cloud environment ID.
294-
# Self-hosted: env_id can be your deployed Twikoo service URL, e.g. https://twikoo.example.com
295-
env_id = "REPLACE_WITH_TWIKOO_ENV_ID_OR_URL"
296-
# region is optional and mainly used with Tencent Cloudbase deployments.
297-
# region = "ap-shanghai"
298-
299-
[project.extra.giscus]
300-
repo = "tom-jerr/tom-jerr.github.io"
301-
repo_id = "R_kgDONZrjcA"
302-
category = "General"
303-
category_id = "REPLACE_WITH_GENERAL_CATEGORY_ID"
304-
mapping = "pathname"
305-
strict = 0
306-
reactions_enabled = 1
307-
emit_metadata = 1
308-
input_position = "top"
309-
theme = "preferred_color_scheme"
310-
lang = "en"
292+
[project.extra.utterances]
293+
repo = "tom-jerr/blog-comments"
294+
issue_term = "pathname"
295+
label = "comments"
296+
theme = "preferred-color-scheme"
297+
light_theme = "github-light"
298+
dark_theme = "github-dark"
311299

312300
[project.extra.tags]
313301
"C++" = "cpp"

0 commit comments

Comments
 (0)