[feat] Add Ring Attention support - #1694

Draft

klhhhhh wants to merge 49 commits into

hao-ai-lab:mainfrom

klhhhhh:ring-atten

Commits on Sep 3, 2026

[feat]add ring-atten from yunchang long context
klhhhhh
committed
dae1444
[feat] implement ring-atten func in fastvideo
klhhhhh
committed
72c29b6
add test scripts
klhhhhh
committed
7da06af
add test scripts
klhhhhh
committed
af20720
Update test script for ring atten config
klhhhhh
committed
4cb38a3
Fix sp size auto test logic
klhhhhh
committed
f1ae950
update test script
klhhhhh
committed
f3ef1c2
add test for single gpu
klhhhhh
committed
1d48efa
[test] fix multi-gpu test
klhhhhh
committed
d1967ca
[test]update docstr in multi-gpu test
klhhhhh
committed
92c3f88
pre-commit
klhhhhh
committed
6c40985
pre-commit
klhhhhh
committed
1f412ce
[feat]adjust layer for usp
klhhhhh
committed
38f4d9a
[feat] usp distributed implementation
klhhhhh
committed
b0a872a
[test]add test for usp
klhhhhh
committed
53469a3
[test]add distributed test for usp
klhhhhh
committed
962c323
[feat]add args for usp
klhhhhh
committed
3726c39
[refactor]adjust import package
klhhhhh
committed
ccaf451
pre-commit
klhhhhh
committed
5a0403f
[refactor]refactor attention kernel
klhhhhh
committed
8ab0597
[fix]fix ring-atten-bugs
klhhhhh
committed
cc4268e
pre-commit
klhhhhh
committed
6ca2f16
pre-commit
klhhhhh
committed
0305153
pre-commit
klhhhhh
committed
bc8d5ca
add temp test script
klhhhhh
committed
0dfa4fb
[log] add logs for ring-atten
klhhhhh
committed
eced929
delete test script
klhhhhh
committed
f23fed1
pre-commit
klhhhhh
committed
2b000ee
pre-commit
klhhhhh
committed
40a266d
delete some files which were not used
klhhhhh
committed
7687681
pre-commit
klhhhhh
committed
a34882d
[refactor]update ring-atten kernel and delete un-call funcs
klhhhhh
committed
661ef3e
add get ring group func
klhhhhh
committed
d14aed3
refactor with main
klhhhhh
committed
9eab39d
add test for usp
klhhhhh
committed
a5ac4e8
[fix]parse flash_attn version instead of lexicographic string compare
klhhhhh
committed
7eb745e
[fix]preserve q/k/v dtype in ring attention backward
klhhhhh
committed
1cfcdae
[fix]throw error when not dispatch ring-atten in subclass overrides and skip all-to-all when ulysses_size == 1
klhhhhh
committed
525cd9d
[fix]restore hsdp_replicate_dim/hsdp_shard_dim validation
klhhhhh
committed
8507ca8
[fix]fix distributed ring attention bug
klhhhhh
committed
849346e
split usp from distributed-atten
klhhhhh
committed
35d2f78
update usp calling function
klhhhhh
committed
a420016
update ring-atten test
klhhhhh
committed
5429113
update ring-atten test
klhhhhh
committed
2dc0769
split ring-atten from distributed atten
klhhhhh
committed
f5d3c35
call ring-atten using new funcs
klhhhhh
committed
291c678
update docstr
klhhhhh
committed
986a9e7
change attention dir and add ring-attem docs
klhhhhh
committed
1ae9653
avoid finding free port causing race window
klhhhhh
committed
0d3ad18