Gated Linear Attention: Efficient Transformers with Data-Dependent Gating
GLA combines linear attention efficiency with learned gating for expressivity. Learn how it achieves RNN-like inference with transformer-like training.
Linear Attention
RWKV: Receptance Weighted Key Value for Efficient Language Modeling
RWKV combines transformer parallel training with RNN efficient inference. Learn how this architecture achieves linear scaling while matching transformer performance.
State Space Models: Mamba and the Post-Transformer Architecture
Mamba-3 achieves 4% better performance than Transformers with 7x faster inference. Learn SSM foundations, selective mechanisms, and hybrid architectures for efficient inference.
Sparse Attention Algorithms: Efficient Transformers at Scale
Master sparse attention algorithms that reduce Transformers quadratic complexity to linear, enabling efficient processing of long sequences in modern AI systems.