High Bandwidth Memory

It is used in conjunction with high-performance graphics accelerators, network devices, high-performance datacenter AI ASICs, as on-package cache in CPUs[1] and on-package RAM in upcoming CPUs, and FPGAs and in some supercomputers (such as the NEC SX-Aurora TSUBASA and Fujitsu A64FX).

[10] The stack is often connected to the memory controller on a GPU or CPU through a substrate, such as a silicon interposer.

[15] AMD and Nvidia have both used purpose-built silicon chips, called interposers, to connect the memory and GPU.

However, as semiconductor device fabrication is significantly more expensive than printed circuit board manufacture, this adds cost to the final product.

[16] The second generation of High Bandwidth Memory, HBM2, also specifies up to eight dies per stack and doubles pin transfer rates up to 2 GT/s.

[21] Up to 307 GB/s per stack (2.5 Tbit/s effective data rate) is now supported in the official specification, though products operating at this speed had already been available.

[27][28][29][30] In mid 2021, SK Hynix unveiled some specifications of the HBM3 standard, with 5.2 Gbit/s I/O speeds and bandwidth of 665 GB/s per package, as well as up to 16-high 2.5D and 3D solutions.

The basic bus widths for HBM3 remain unchanged, with a single stack of memory being 1024-bits wide.

The stacks consist of 8 or 12 16 Gb DRAMs that are each 30 μm thick and interconnected using Through Silicon Vias (TSVs).

[34] According to Chris Mellor of The Register, with JEDEC not yet having developed its HBM3 standard, might mean that SK Hynix would need to retrofit its design to a future and faster one.

[36] In June 2022, SK Hynix announced they started mass production of industry's first HBM3 memory to be used with Nvidia's H100 GPU expected to ship in Q3 2022.

[38] On 30 May 2023, SK Hynix unveiled its HBM3E memory with 8 Gbit/s/pin data processing speed (25% faster than HBM3), which is to enter production in the first half of 2024.

[40] Micron HBM3E memory is a high-performance HBM that uses 1β DRAM process technology and advanced packaging to achieve the highest performance, capacity and power efficiency in the industry.

[46] In September 2024, SK Hynix announced the mass production of its 12-layered HBM3E memory[47] and in November the 16-layered version.

A DRAM-optimised AI engine is placed inside each memory bank to enable parallel processing and minimise data movement.

[53] JEDEC first released the JESD229 standard for Wide IO memory,[54] the predecessor of HBM featuring four 128 bit channels with single data rate clocking, in December 2011 after several years of work.

[3] HBM was adopted as industry standard JESD235 by JEDEC in October 2013, following a proposal by AMD and SK Hynix in 2010.

[6] High volume manufacturing began at a Hynix facility in Icheon, South Korea, in 2015.

[58][59] In June 2016, Intel released a family of Xeon Phi processors with 8 stacks of HCDRAM, Micron's version of HBM.

At Hot Chips in August 2016, both Samsung and Hynix announced a new generation HBM memory technologies.

Samsung also announced a lower-cost version of HBM under development targeting mass markets.