A new memory approach could reduce one of the biggest hardware limits affecting large artificial intelligence models by separating model weights from the frequently updated KV cache.
High Bandwidth Flash, or HBF, is being developed as a stacked NAND memory technology for AI systems. Its proposed role is to store large model weights at a lower cost and with much higher capacity than conventional High Bandwidth Memory.
HBM would remain responsible for KV cache and other data that requires frequent writes and very low latency. This division could allow future AI accelerators to support larger models and longer context windows without adding more expensive HBM to every processor.
HBF is still under development, and its final cost, performance, and commercial availability remain uncertain.
AI Models Consume HBM in Two Different Ways
Large language models use memory for both their model weights and their KV cache.
Model weights contain the numerical values learned during training. They represent the model’s stored knowledge and normally remain unchanged while the model generates a response.
KV cache is temporary working memory. It stores information from earlier tokens so the model does not need to recalculate the entire conversation each time it produces a new token.
| Memory workload | Main purpose | Access pattern |
|---|---|---|
| Model weights | Stores learned model parameters | Read frequently, written rarely |
| KV cache | Holds context from the current session | Read and written continuously |
| Activations | Stores temporary calculation results | Frequent reads and writes |
| Training data | Supplies examples during training | Large sequential transfers |
These workloads have different requirements. Model weights need large capacity and strong read bandwidth, while KV cache needs fast writes and low latency.
HBM currently performs both jobs because it is the fastest practical memory available beside modern AI accelerators. That makes the system effective, but also expensive and capacity constrained.
A 100 Billion Parameter Model Needs About 200GB
The memory required for model weights grows directly with the number of parameters and the precision used to store them.
A model with 100 billion parameters stored at FP16 precision requires two bytes per parameter. That produces a basic weight storage requirement of about 200GB.
| Model size | Precision | Approximate weight storage |
|---|---|---|
| 7 billion parameters | FP16 | 14GB |
| 70 billion parameters | FP16 | 140GB |
| 100 billion parameters | FP16 | 200GB |
| 400 billion parameters | FP16 | 800GB |
Quantisation can reduce these requirements, but large models still consume substantial memory.
HBM4 stacks are expected to provide high bandwidth, but individual stacks still have limited capacity. A 12 layer stack may hold around 36GB, while a 16 layer version could reach approximately 48GB.
Storing hundreds of gigabytes of model weights therefore requires several HBM stacks and often multiple accelerators.
HBF Uses Stacked NAND for Higher Capacity
HBF applies a structure similar to HBM but replaces stacked DRAM with stacked NAND flash.
Several NAND dies are placed vertically and connected using Through Silicon Vias. A logic die manages the memory array and schedules large numbers of parallel read operations.
A proposed HBF device could offer 512GB of capacity and bandwidth between 0.4TB/s and 3TB/s.
| Feature | HBF | HBM |
|---|---|---|
| Memory technology | NAND flash | DRAM |
| Main proposed role | Model weight storage | KV cache and active data |
| Capacity | Potentially very high | Lower per stack |
| Write endurance | Limited | Much stronger |
| Read latency | Higher | Lower |
| Write performance | Slower | Faster |
| Expected cost | Lower than HBM | Very expensive |
| Packaging | Advanced stacked design | Advanced stacked design |
The high capacity is important because one HBF array could potentially hold the weights of a large model that would otherwise require several HBM stacks.
Parallelism Helps Overcome Slow NAND Reads
Individual NAND cells are much slower than DRAM.
Typical SRAM access may take around one nanosecond, while DRAM can require roughly 100 nanoseconds. NAND access can extend to around 100 microseconds.
That makes an individual NAND read approximately 1,000 times slower than a DRAM read.
HBF attempts to compensate by reading thousands of NAND cells at the same time. The logic controller coordinates these parallel operations, producing a high combined bandwidth even though each individual memory cell remains slow.
This design is well suited to model weights because inference repeatedly reads large blocks of mostly static data.
The stated bandwidth figures should be interpreted carefully. Comparing cumulative HBF bandwidth with a single interface or stack can produce misleading results unless capacity, stack count, workload, and measurement conditions are identical.
KV Cache Still Needs HBM
HBF is not expected to replace HBM for every AI workload.
NAND flash has limited write endurance. Its cells wear out after repeated programming and erasing, making it unsuitable for data that changes constantly.
KV cache is updated every time the model processes or generates new tokens. It therefore needs memory that supports continuous, rapid writes.
HBM remains a better choice because DRAM can handle repeated updates with much lower latency and stronger endurance.
| Requirement | Model weights | KV cache |
|---|---|---|
| Large capacity | Very important | Important |
| High read bandwidth | Very important | Very important |
| Fast writes | Less important | Essential |
| Write endurance | Less important | Essential |
| Low latency | Helpful | Critical |
| Data changes frequently | No | Yes |
This creates a natural division of responsibilities. HBF stores the mostly fixed model weights, while HBM handles KV cache and other dynamic information.
Freeing HBM Could Support Longer Context Windows
Moving model weights away from HBM would leave more high speed memory available for KV cache.
KV cache grows as context length increases. Long conversations, large documents, codebases, and multi-step reasoning tasks can consume large amounts of memory.
When HBM fills, a system may need to reduce the context size, move data to slower memory, or distribute the workload across more accelerators.
HBF could reduce that pressure by removing one of the largest static memory demands.
| Possible benefit | Effect on AI systems |
|---|---|
| More HBM available for KV cache | Supports longer context |
| Higher model capacity | Larger models fit on fewer accelerators |
| Lower memory cost | Reduces inference expense |
| Fewer GPUs required | Simplifies some deployments |
| Better accelerator utilisation | More compute time spent processing data |
| Larger local model storage | Reduces transfers from slower storage |
Longer context does not automatically improve model accuracy. Models must still be trained to use large context windows effectively. However, additional memory gives developers more flexibility.
HBF Will Not Be as Cheap as an SSD
HBF uses NAND, but it should not be confused with ordinary solid state storage.
The technology requires advanced packaging, vertically stacked dies, precision bonding, Through Silicon Vias, and a dedicated logic controller. These features add significant cost.
An HBF device will likely be more expensive than a conventional SSD with the same capacity.
Its value comes from providing far greater bandwidth and tighter integration with AI processors.
| Storage technology | Capacity | Bandwidth | Expected cost |
|---|---|---|---|
| Consumer SSD | High | Moderate | Lowest |
| Enterprise SSD | High | Higher | Moderate |
| HBF | Very high | Up to several TB/s | Higher |
| HBM | Limited | Extremely high | Highest |
The commercial case depends on whether HBF is sufficiently cheaper than the quantity of HBM or additional GPUs it replaces.
A Split Memory Architecture Could Lower AI Costs
AI hardware currently relies heavily on HBM because it combines strong bandwidth with low latency. Its limited capacity and high price have made it one of the most important constraints in accelerator design.
HBF proposes a more specialised architecture.
Static model weights could move to large, read-optimised NAND stacks. Dynamic workloads such as KV cache would stay in fast DRAM-based HBM.
This would not remove the memory wall entirely. AI systems would still need careful software scheduling, fast interconnects, efficient controllers, and enough HBM for active workloads.
The approach could nevertheless reduce the need to use expensive HBM for data that rarely changes.
HBF remains an emerging standard rather than a finished commercial product. Its success will depend on real bandwidth, latency, endurance, packaging yield, software support, and price.
If those challenges are solved, future AI accelerators may no longer treat all model memory in the same way. HBF could become the high capacity home for model weights, while HBM concentrates on KV cache and the rapidly changing data that needs its speed most.
Discussion (0)
Be the first to comment.