Mamba 3 Architecture

Quick front-warning: I am not an expert in either the math or architecture for LLMs. It is something I am interested in, but I am not an expert in either.

The Mamba 3 paper could be the future of sub-quadratic LLM inference. Nvidia Nemotron already uses some Mamba2 layers with the classic transformer layers.

The pitch of Mamba 3 is that it is designed inference-first. Most earlier sub-quadratic models (including Mamba 2) were designed training-first, and that left decoding memory-bound. Your GPU spends time moving memory around instead of computing. The Mamba 3 paper frames itself as a fix for exactly that, and the results are good enough to pay attention to.

Three things are new in Mamba 3, and I will try to explain them like I understand them:

• A more expressive recurrence by a discretization trick. The math is over my head, but the practical effect is that the recurrence itself can represent more than Mamba 2’s recurrence could.
• Storing the state in a complex value format, very similar to RoPE in transformers. Hopefully this makes an improvement in state tracking, which was a failure point in previous state space models.
• Multi input multi output, which allows the state space model to apply the recurrence to a vector at a time instead of a state at a time. This increases training cost slightly, but gives the model more representational power.

The numbers from the paper are decent. At the 1.5B scale, Mamba 3 improves downstream accuracy by about 0.6% over the next best sub-quadratic model, which is Gated DeltaNet. The MIMO variant adds another 1.2%, for a total gain of about 1.8%. Mamba 3 also matches Mamba 2 perplexity with about half the state size, which matters because state size is a big chunk of memory for these models.

The big practical claim is that Mamba 3 decodes at roughly half the cost of a comparable transformer, while staying competitive on quality. That is the part that matters for inference-heavy workloads like agentic coding, long-context retrieval, and reasoning models.

Nvidia has been shipping hybrid Mamba-Transformer models for a while, and we could likely see them ship Nemotron 4 with these Mamba3 layers. In my mind, these layers plus DFlash could release a very powerful combo of efficient inference in the near future.