NVIDIA Proposes Full-Stack AI Memory Design, Citing Up to 30x Inference Throughput

NVIDIA has proposed a coordinated approach to AI memory design as demand for GPU memory rises alongside larger models, longer context windows, and wider use of AI agents. According to Odaily, the approach coordinates GPUs, CPUs, networking, storage, and software to improve memory efficiency rather than simply increase GPU memory capacity. The proposal uses the NVFP4 low-precision format to compress model weights and KV cache, moves temporarily inactive context to lower-cost storage, and uses multi-GPU deployments to free up HBM. NVIDIA reported that in SemiAnalysis AgentX tests, the GB300 NVL72 delivered up to 10 times the inference throughput per unit cost of the H200 platform on certain workloads. The Vera Rubin NVL72 delivered up to 30 times the throughput per unit cost of the GB300 NVL72, while increasing HBM capacity per GPU by about 3%.