Skip to content

step35.md

Latest commit

 

History

History
388 lines (380 loc) · 21.8 KB

step35.md

File metadata and controls

388 lines (380 loc) · 21.8 KB
Raw

Step-3.5-Flash

Step-3.5-Flash is a MoE language model from StepFun. Megatron Bridge supports checkpoint conversion, inference, and continued pretraining through a dedicated Step-3.5 bridge, provider, and recipe.

Verified configurations

Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.

Run a configuration

Choose a workflow, precision, and exact recorded combination. The command and expected result update below.

Import & Export Pretrain Benchmark SFT LoRA Long Context
Precision All BF16 FP8 MX NVFP4
GPU All H100
Import · CPU ✓ Verified BF16 Import · GPU ✓ Verified BF16 Export · CPU ✓ Verified BF16 Export · GPU ✓ Verified BF16 Pretrain · H100 ○ Unverified BF16 SFT · H100 ○ Unverified BF16 Long Context · H100 ○ Unverified BF16 LoRA · H100 ○ Unverified BF16

Import · CPU

✓ Verified
Hardware
not specified
Precision
BF16
Last verified
2026-08-05
Exact command
Command Copy 
./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model stepfun-ai/Step-3.5-Flash --hf-revision ab446a3de5e171ea341227e24bb1f090e1b771f7 --megatron-path work/model-verification/step35-flash/cpu-megatron --torch-dtype bfloat16 --trust-remote-code
Expected result

CPU import completed all mappings for the immutable 44-shard source, constructed 199,997,706,240 parameters, and created iter_0000000. The checkpoint reloaded successfully for CPU export; its subsequent strict audit covered all 804 source tensors.

Import · GPU

✓ Verified
Hardware
not specified
Precision
BF16
Last verified
2026-08-05
Exact command
Command Copy 
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model stepfun-ai/Step-3.5-Flash --hf-revision ab446a3de5e171ea341227e24bb1f090e1b771f7 --megatron-path work/model-verification/step35-flash/imported-megatron --torch-dtype bfloat16 --tp 1 --pp 1 --ep 8 --etp 1 --distributed-timeout-minutes 110 --trust-remote-code --low-memory-save
Expected result

Distributed import completed at TP1/PP1/EP8/ETP1, constructed 33,525,780,480 parameters per rank, and created iter_0000000. The persisted checkpoint then reloaded successfully for both distributed export and checkpoint-backed generation.

Export · CPU

✓ Verified
Hardware
not specified
Precision
BF16
Last verified
2026-08-05
Exact command
Command Copy 
./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model stepfun-ai/Step-3.5-Flash --hf-revision ab446a3de5e171ea341227e24bb1f090e1b771f7 --megatron-path work/model-verification/step35-flash/cpu-megatron/iter_0000000 --hf-path work/model-verification/step35-flash/cpu-hf-export --torch-dtype bfloat16 --trust-remote-code
Expected result

CPU export reloaded the imported checkpoint and wrote 44 indexed BF16 shards with all 804 source keys, shapes, and dtypes. Of 398,768,626,944 serialized bytes, 395,600,878,848 bytes across 801 tensors matched the immutable source bitwise. The three published MTP output entries occupy 3,167,748,096 bytes and were intentionally materialized from Megatron's shared lm_head representation; each exported entry matched lm_head bitwise and used independent safetensors storage. A Transformers reload through the pinned checkpoint's metadata/RoPE compatibility view ignored only the three published inference-unused MTP layers, predicted the same next token as the source (' Paris'), and reached cosine similarity 0.997260 with maximum/mean absolute logit differences 1.320312/0.209649.

Export · GPU

✓ Verified
Hardware
not specified
Precision
BF16
Last verified
2026-08-05
Exact command
Command Copy 
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model stepfun-ai/Step-3.5-Flash --hf-revision ab446a3de5e171ea341227e24bb1f090e1b771f7 --megatron-path work/model-verification/step35-flash/imported-megatron/iter_0000000 --hf-path work/model-verification/step35-flash/hf-export --torch-dtype bfloat16 --tp 1 --pp 1 --ep 8 --etp 1 --distributed-timeout-minutes 110 --distributed-save --save-every-n-ranks 1 --no-progress --trust-remote-code
Expected result

Distributed export reloaded the TP1/PP1/EP8/ETP1 checkpoint and wrote 44 indexed BF16 shards with all 804 source keys, shapes, and dtypes. The exact audit found 801 tensors and 395,600,878,848 bytes bit-identical to the source. Megatron-Core shares the output head with MTP, so the three source MTP output tensors were explicitly normalized to lm_head.weight; all three exported copies matched that tensor bitwise and had independent storage. The generic grouped-expert export path recovered from CUDA stack pressure by moving only the affected merge to CPU. A Transformers reload through the pinned checkpoint's metadata/RoPE compatibility view ignored only the three published inference-unused MTP layers, predicted the same next token as the source (' Paris'), and reached cosine similarity 0.997260 with maximum/mean absolute logit differences 1.320312/0.209649.

Pretrain · H100

○ Unverified
Hardware
H100
Precision
BF16
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command

No runnable command is recorded for this status.

Expected result

The public Step-3.5-Flash H100 recipe must complete a bounded 100-step run with finite loss, no skipped or NaN iterations, all five metrics, and a reloadable final checkpoint. Training is deferred by this card.

SFT · H100

○ Unverified
Hardware
H100
Precision
BF16
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command

No runnable command is recorded for this status.

Expected result

A pinned-data 100-step full-SFT run must finish with finite loss, no skipped or NaN iterations, all five metrics, and a reloadable final checkpoint. Training is deferred by this card.

Long Context · H100

○ Unverified
Hardware
H100
Precision
BF16
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command

No runnable command is recorded for this status.

Expected result

A dedicated packed long-context SFT run must complete 100 steps with finite loss, no skipped or NaN iterations, all five metrics, and a reloadable checkpoint. Training is deferred by this card.

LoRA · H100

○ Unverified
Hardware
H100
Precision
BF16
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command

No runnable command is recorded for this status.

Expected result

A Step-3.5-Flash PEFT recipe with an audited adapter target set must complete 100 steps with finite loss, all five metrics, and a reloadable adapter checkpoint. Training is deferred by this card.