Among the technical highlights of MiniMax's 01 series released last year is Lightning Attention, with a straightforward core goal: reducing the computational complexity of the attention mechanism.
The Problem with Traditional Attention
Standard Transformer attention has O(n²) complexity — doubling the input length quadruples the computation. This is why early models had short context windows (512 or 1024 tokens); longer contexts would cause computational costs to explode.
Although various optimizations have since emerged (FlashAttention, PagedAttention, etc.), the fundamental O(n²) constraint remains unchanged.
The Lightning Attention Approach
Lightning Attention follows a linear attention variant path. By changing how attention is computed, it reduces complexity from O(n²) to near O(n).
Specifically, it involves an approximate decomposition of the attention matrix, significantly reducing floating-point operations while preserving the model's expressive power. MiniMax claims this optimization delivers a several-fold speed improvement in long-context scenarios.
Real-World Performance
The MiniMax-01 series performs well in million-token long-context tests, showing clear speed advantages over similarly sized models using standard attention. It also maintains high accuracy in Needle-in-a-Haystack retrieval tests.
However, the trade-off with linear attention is precision. On tasks requiring precise long-range attention, the linear approximation may lose information. This is why mainstream models have not fully abandoned O(n²) attention — it is more accurate, albeit more expensive.
Industry Trends
MiniMax is not alone in this effort. Reducing attention complexity is a shared challenge across the industry:
- Google's various attention variants
- Meta's GQA (Grouped Query Attention) in Llama
- Mamba/SSM takes the route of abandoning attention entirely
The eventual direction may be a hybrid architecture — using standard attention for layers that require precise focus and linear attention or SSM for the rest, striking a balance between accuracy and efficiency.
Source: CocoLoop, MiniMax official technical report