# What is a state space model like Mamba, and why do hybrid models use less memory on long context?

A layer that reads text in order and carries a fixed-size summary forward instead of keeping every past token, so its memory stays the same however long the conversation gets. Mamba is the best-known kind.

- Page: https://tecyfy.com/concepts/state-space-model
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: advanced
- Also called: SSM, Mamba, Mamba-2, Hybrid Mamba-Transformer

A [Transformer](https://tecyfy.com/concepts/transformer) remembers by keeping. Each of its [attention](https://tecyfy.com/concepts/self-attention) layers stores a key and a value for every token it has seen, the [KV cache](https://tecyfy.com/concepts/kv-cache), and looks back over all of them for each new token. That is why it recalls a detail from 100,000 tokens ago so well, and why its memory grows with every token.

## What a state space layer does instead

A state space layer reads the text in order and keeps one running state of a fixed size. Each new token updates the state and then the token is gone; nothing is stored per token. Memory stays flat whether the conversation is 1,000 tokens or 1 million, and the work per new token stays the same too.

Older state space models updated their state the same way whatever the token was, and they lost to attention on language. Mamba, published by Albert Gu and Tri Dao in 2023, made the update depend on the token, so the layer can choose what to keep and what to forget. Its authors measured 5 times the throughput of a Transformer of similar size. Mamba-2, in 2024, made the core layer 2 to 8 times faster again.

## Why most new models are hybrids

A fixed-size state is a summary, and a summary can drop the one exact detail you later ask about. Hybrid models mix the 2: mostly Mamba layers for cheap reading, plus a few attention layers that keep every token for exact recall. NVIDIA's Nemotron 3.5 Super VL and Bolt 74B each have 40 Mamba-2 layers and 8 attention layers, with [mixture-of-experts](https://tecyfy.com/concepts#mixture-of-experts) layers between them.

## What it means for memory

For someone running a model on their own GPU, a hybrid's cache is far smaller. Only the attention layers grow with the context; the Mamba layers add a fixed state for each conversation, about 170 MB for each conversation of Nemotron 3.5 Super VL, whatever its length. It is part of why NVIDIA can list Super VL with a 1-million-token [context window](https://tecyfy.com/concepts/context-window) on a single RTX PRO 6000. The [GPU memory calculator](/ai-agents/gpu-memory-calculator) counts the fixed state and the growing cache separately for every model it lists.

## Related

- [Transformer](https://tecyfy.com/concepts/transformer): The design behind almost every modern language model. It uses attention to let each word in a passage look at every other word, all at once.
- [Self-Attention](https://tecyfy.com/concepts/self-attention): The step in a transformer where every token looks at the other tokens in the same text and takes in what is relevant to it, so meaning depends on context.
- [KV Cache](https://tecyfy.com/concepts/kv-cache): Work a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it.
- [Context Window](https://tecyfy.com/concepts/context-window): Everything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out.
- [Mixture of Experts](https://tecyfy.com/concepts#mixture-of-experts): A neural architecture that routes tokens through different expert subnetworks.
- [Inference](https://tecyfy.com/concepts/inference): Using a trained model to produce an answer. Training builds the model once; inference is every time someone asks it something, and it is what you pay for per token.

## Sources

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://arxiv.org/abs/2405.21060)
- [NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16)
- [NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 config.json](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/blob/main/config.json)
- [NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4)