LLM Training Hub

A technical reference for training large language models — covering pre-training, fine-tuning, distributed training, PEFT, and optimization techniques.

Use the sidebar to explore:

• Training Phases — pre-training objectives, mid-training domain adaptation, SFT, and reasoning/thinking model training
• Distributed Training — data parallelism, tensor/pipeline parallelism, ZeRO/FSDP
• PEFT — LoRA, QLoRA, adapters, and the bitsandbytes library
• Optimization — mixed precision, gradient checkpointing, memory-efficient optimizers, quantization