Repository navigation

feat(data): add text chat collate for unified HF datasets - #4307

Merged

yaoyu-33 merged 56 commits into

mainfrom

kant/pr4169-unified-hf-dataset

Jun 24, 2026

Merged

feat(data): add text chat collate for unified HF datasets#4307
yaoyu-33 merged 56 commits into
mainfrom
kant/pr4169-unified-hf-dataset

Commits on Jun 2, 2026
- docs(data): map VLM data pipeline refactor state
  yaoyu-33
  committed
  864dbf0
- refactor(data): unify VLM processing helpers
  yaoyu-33
  committed
  145459a
- chore(data): remove VLM phase report
  yaoyu-33
  committed
  cf0b3cf
- refactor(data): add VLM source adapters
  yaoyu-33
  committed
  68bfd59
- fix(data): preserve Qwen Energon assistant masking
  yaoyu-33
  committed
  0d3acb2
- refactor(data): remove VLM collate wrapper helpers
  yaoyu-33
  committed
  6584a2e
- refactor(data): share VLM collate in Energon encoders
  yaoyu-33
  committed
  2d2ff97
Commits on Jun 3, 2026
- refactor(data): rename HF Energon task encoder
  yaoyu-33
  committed
  b2dd093
- refactor(data): clarify VLM collate steps
  yaoyu-33
  committed
  916a6a8
- refactor(data): remove partial vlm collate wrappers
  yaoyu-33
  committed
  d358f01
- fix(data): restore vlm energon safety checks
  yaoyu-33
  committed
  886b6a1
Commits on Jun 4, 2026
- refactor(data): unify VLM Energon collation path
  yaoyu-33
  committed
  a72a0cc
- fix(data): thread HF task encoder collate options
  yaoyu-33
  committed
  4a4a093
- refactor(data): name Gemma3 VLM collator explicitly
  yaoyu-33
  committed
  8e21828
- refactor(data): move VLM collators to model modules
  yaoyu-33
  committed
  e72d2c4
Commits on Jun 5, 2026
- chore(data): remove stale Kimi VL debug comments
  yaoyu-33
  committed
  4f73bd8
- fix(data): anchor VLM assistant loss masks to chat templates
  yaoyu-33
  committed
  7176a1d
Commits on Jun 8, 2026
- Merge origin/main into mira/issue4041-vlm-assistant-mask
  yaoyu-33
  committed
  f9b6b47
- fix(data): make assistant mask role ends explicit
  yaoyu-33
  committed
  9e87f67
- Apply suggestions from code review
  
  yaoyu-33
  and
  claude[bot]
  authored
  d9896d8
Commits on Jun 10, 2026
- fix(data): require explicit VLM assistant mask boundaries
  yaoyu-33
  committed
  98db230
- fix(data): preserve Kimi assistant mask before padding
  yaoyu-33
  committed
  6fda95c
- fix(data): clarify VLM loss role naming
  yaoyu-33
  committed
  2b02235
- fix(data): separate VLM loss roles and parts
  yaoyu-33
  committed
  58cc94e
- fix(data): keep VLM boundary masks role based
  yaoyu-33
  committed
  69ce12d
- fix(data): align VLM assistant masks with padding
  yaoyu-33
  committed
  2a78fa8
Commits on Jun 11, 2026
- Merge remote-tracking branch 'origin/main' into mira/issue4041-vlm-assistant-mask
  yaoyu-33
  committed
  b8a5496
- fix(data): update Qwen VLM padding mask test
  yaoyu-33
  committed
  7693b4d
- feat(data): add text chat collate for HF conversations
  yaoyu-33
  committed
  26f48a2
- Merge branch 'main' into mira/issue4041-vlm-assistant-mask
  yaoyu-33
  authored
  06fdfb0
- refactor(data): unify hf dataset paths
  yaoyu-33
  committed
  427d61f
- fix(data): forward VLM tool schemas to chat templates
  yaoyu-33
  committed
  003f43a
- fix(data): resolve HF dataset refactor conflicts
  yaoyu-33
  committed
  24c4b15
- refactor(data): rename HF conversation dataset provider
  yaoyu-33
  committed
  7a419c3
- fix(data): let hf conversation dataloader control order
  yaoyu-33
  committed
  314060b
- fix(data): restore packed hf text sft path
  yaoyu-33
  committed
  222c19e
- refactor(data): split sequence packing config switches
  yaoyu-33
  committed
  2353c8c
Commits on Jun 12, 2026
- fix(data): pack text HF batches in collate
  yaoyu-33
  committed
  7797f4d
- fix(data): align text in-batch packing metadata
  yaoyu-33
  committed
  d113c10
- fix(data): anchor VLM assistant loss masks to chat templates
  yaoyu-33
  committed
  d01ca27
- fix(data): unwrap megatron tokenizers for text chat collate
  yaoyu-33
  committed
  da973fe
- fix(data): resolve stacked base conflicts
  yaoyu-33
  committed
  ac9fee8
Commits on Jun 22, 2026
Commits on Jun 23, 2026
- Merge branch 'main' into mira/issue4041-vlm-assistant-mask
  yaoyu-33
  authored
  2c95842
- Merge branch 'mira/issue4041-vlm-assistant-mask' into kant/pr4169-unified-hf-dataset
  yaoyu-33
  committed
  e0c6239
Commits on Jun 24, 2026
- fix(data): update VLM token utils imports
  yaoyu-33
  committed
  77d4b06
- Merge branch 'main' into kant/pr4169-unified-hf-dataset
  yaoyu-33
  committed
  1e2723e
- fix(data): update nemotron ultra packing specs field
  yaoyu-33
  committed
  6a2eba2
- fix(recipes): update nemotron openmath provider assertion
  yaoyu-33
  committed
  b3336e7
- Merge branch 'main' into kant/pr4169-unified-hf-dataset
  yaoyu-33
  committed
  3c08231

Commits

Commits on Jun 2, 2026

Commits on Jun 3, 2026

Commits on Jun 4, 2026

Commits on Jun 5, 2026

Commits on Jun 8, 2026

Commits on Jun 10, 2026

Commits on Jun 11, 2026

Commits on Jun 12, 2026

Commits on Jun 22, 2026

Commits on Jun 23, 2026

Commits on Jun 24, 2026