Efficient LLM

RWKV: Receptance Weighted Key Value for Efficient Language Modeling

RWKV combines transformer parallel training with RNN efficient inference. Learn how this architecture achieves linear scaling while matching transformer performance.

Sparse Mixture of Experts: Scaling Language Models Efficiently

SMoE activates only a subset of parameters per token, enabling massive model capacity with constant compute. Learn about routing mechanisms, load balancing, and deployment.