Session 16 of HTGM v2 Hindi LLM training is now completed.
This session focused on stable checkpoint continuation, stronger validation improvement, and pushing the model beyond the important 37K+ optimizer steps milestone.
After 145+ total training hours, the model is showing better consistency, lower validation loss, improved perplexity, and stronger long-session reliability.
This repository documents the real progress of building a Hindi LLM from scratch using a GPT-style Transformer architecture.
- Session Number: 16
- Total Training Time: 145+ Hours
- Total Sessions Completed: 16
- Major Milestone: 37K+ Optimizer Steps Crossed
- Model Name: HTGM v2
- Type: Hindi Large Language Model (Hindi LLM)
- Architecture: GPT-style Transformer
- Model Size: 163.4M Parameters
- Sequence Length: 2048
- Vocabulary Size: 100,000
- Dataset Size: ~41GB
- Dataset Source: AI4Bharat Sangrah + Manual Cleaning
- Training Platform: Kaggle
- GPU: 2x T4
- Tokenizer: Hugging Face BPE-based Tokenizer
- Total Planned Optimizer Steps: 923,382
- Resumed From Checkpoint Step: 1,118,962
- Current Training Step: 37,512+
- Mid Checkpoints Saved: ckpt_s35000.pt, ckpt_s36000.pt, ckpt_s37000.pt
- Final Checkpoint Saved: ckpt_final_s1200400.pt
- chunk_1.txt
- chunk_2.txt
- chunk_3.txt
- Early Session Loss: 3.7838
- Mid Training Loss: 4.8249
- Later Training Loss: 4.8023
- Latest Observed Loss: 5.0656
- Final Validation Loss: 4.9147
- Final Validation Perplexity (PPL): 136.27
- Learning Rate: 7.63e-05
- Initial Learning Rate: 9.90e-07
- Tokens Per Second: Stable throughout training
Training resumed from a previous checkpoint instead of restarting from scratch.
- Safe long training workflow
- Stable continuation across sessions
- Hyperparameter testing without losing progress
- Reliable optimizer and scheduler recovery
This makes long Hindi LLM experiments practical and efficient.
- Validation loss improved to 4.91
- Perplexity improved to 136
- Stronger checkpoint reliability
- GPU performance stayed strong
- No major crashes
- W&B monitoring active
- Smooth checkpoint saving
Crossing 37K+ steps is important, but improving validation quality matters even more.
This session proves that better optimization is more valuable than simply training longer.
- Training is stable
- Validation is improving
- Model learning quality is getting better
- Long-session reliability is strong
Still not production-ready, but the foundation is becoming much stronger.
- Apply Supervised Fine-Tuning (SFT)
- Improve instruction-following capability
- Better dataset structuring
- Continue longer training sessions
- Improve response quality and coherence
Mahesh Editor
India AI Official
This project is shared publicly to document real AI development, not just final results.
Every checkpoint, problem, fix, and improvement is part of the journey.