NVIDIA moves a custom memory controller into the HBM base die with NVHBM

NVIDIA has added NVHBM, a custom high-bandwidth-memory approach, to its NVLink Fusion program. The architectural change moves NVIDIA's memory controller into the HBM base die instead of keeping it on the accelerator's compute die. Amazon's Annapurna Labs is the first named collaborator.

The product is aimed at hyperscalers and chip designers building custom XPUs while connecting them to NVIDIA rack-scale systems. NVIDIA also says it is defining a standard implementation that multiple memory providers can offer. The event is therefore both a component design and an ecosystem proposal for semi-custom AI infrastructure.

Compared with standard HBM4E, NVIDIA claims NVHBM can provide up to 30% greater memory bandwidth, 15% lower HBM power consumption and up to 25% more area on the XPU compute die. Each figure is a vendor comparison and uses “up to” where stated.

The three metrics address different bottlenecks. Bandwidth affects how quickly processors can access model data, power affects system efficiency, and compute-die area can be allocated to additional logic. A design can benefit in one dimension without delivering the maximum claim in all three simultaneously.

Traditional HBM designs place the memory controller on the XPU die. Moving it into the base die changes where silicon area and power are spent. NVIDIA's standardization plan is intended to reduce the engineering work needed to qualify memory from more than one supplier.

NVLink Fusion is the surrounding platform. It gives partners access to interconnect chiplets, switches and rack designs so their custom CPUs or XPUs can participate in NVIDIA's scale-up architecture. NVHBM adds memory technology to that menu rather than functioning as a standalone consumer memory product.

Large AI models can be constrained by data movement even when compute throughput is high. More bandwidth and lower memory power can improve the balance of a rack, while reclaimed die area gives a custom accelerator designer more room for its own logic. Those are system-level advantages rather than a simple increase in memory capacity.

The commercial significance will depend on adoption beyond one collaborator and on manufacturability across providers. A standard can lower integration risk only if suppliers deliver compatible parts and customers validate them. NVIDIA's announcement establishes the design direction, not volume shipment or independent benchmark results.

The immediate users are cloud providers, custom-silicon teams, memory suppliers and system builders. Annapurna Labs plans to work with the technology and support NVLink Fusion with Trainium chips beginning with Trainium4. That creates a concrete first integration path between Amazon accelerators and NVIDIA's rack architecture.

Enterprise buyers are one step removed. They may eventually see the technology through cloud instances or managed AI systems rather than purchasing NVHBM directly. Procurement decisions should wait for product specifications, availability and workload testing.

NVIDIA has not disclosed shipment dates, memory capacities, pricing, participating memory providers or production yields in the announcement. It also does not provide a benchmark system demonstrating the maximum bandwidth, power and area claims together.

Compatibility across future XPU designs and the degree of supplier interchangeability remain to be demonstrated. The evidence supports the architecture, claimed upper-bound improvements and Annapurna collaboration. It does not support a claim that NVHBM has already replaced HBM4E in deployed systems.