In March 2017, Intel announced the first of a new range of persistent storage and memory products based on Intel Optane technology. Optane is the Intel brand name for 3D XPoint, a persistent memory solution developed in conjunction with Micron Technologies. Intel sells Optane as an SSD and in a DIMM form factor. This post examines both technologies and their use in the Enterprise.
Figure 1 shows a version of the classic storage hierarchy, generally drawn as a pyramid. The fastest volatile memory devices sit at the top of the pyramid, with each lower layer representing persistence, increasing capacity and latency. As described in this post, classic computing architectures read and write data from external (and persistent) storage into DRAM where the CPU processes it. Faster external storage translates to better application responses.
The progression of Moore’s Law has seen a transition towards multi-core processors with the current generation supporting up to 28 cores (Intel Xeon Platinum 8380HL). DRAM hasn’t scaled quite so well, offering up to 128GB capacities and in the case of the Xeon processor quoted, a theoretical maximum of 4.5TB supported, although that would require a motherboard with 36 DIMM slots! As a result, multi-core processors see a reducing average availability of DRAM per core.
As the price of computing has dropped, in-memory computing has become a more viable solution for specific application processing models, including ML/AI and relational databases like SAP HANA. Increasing the amount of directly addressable memory increases application efficiencies. All this sets the scene for a new type of memory to answer these challenges.
3D XPoint technology and specifically in this instance, Intel Optane provides a new tier of persistent memory to address both the needs of external storage and DRAM. The characteristics of 3D XPoint are different from existing NAND and DRAM products, spanning both categories, as summarised in the following table. Although Intel hasn’t released specific details, it is believed that 3D XPoint is a form of resistive RAM (ReRAM) where electrical resistance determines the value of a bit, rather than measuring a trapped charge or capacitance.
|Endurance||High||Variable (low on QLC)||High|
|Update in place||Yes||No||Yes|
|Latency||~100ns (DDR4)||100µs||10µs (SSD), 350ns (DIMM)|
|Throughput||25GB/s (DDR4-3200)||7GB/s read, 3.8GB/s write||500K+ IOPS (SSD), 8GB/s (read DIMM) 3GB/s (write DIMM) 2.5GB/s (SSD)|
|Scalability||Up to 128GB per DIMM||Up to 16/32TB, depending on vendor & Interface||SSD up to 1.5TB, DIMM up to 512GB|
These figures offer a guide to the differences between each of the media types. We can see that 3D XPoint (as an SSD) provides a 10x improvement in latency over NAND flash and another order of magnitude again when deployed as a DIMM.
A critical characteristic of 3D XPoint and Optane is how data is written to the media. 3D XPoint provides byte-addressability in place, allowing any individual byte to be updated at any time, similar to DRAM. NAND flash writes data in blocks and pages, which require erasure before writing, introducing both complexity and latency into I/O. This design is also why NAND flash has a lower endurance than DRAM or 3D XPoint. The ability to update at the byte level makes 3D XPoint suitable as a memory replacement. As a result, Intel offers Optane as a persistent memory technology in a DIMM form-factor as well as an SSD.
Intel Optane Persistent Memory
Intel Optane Persistent Memory (Optane PMEM) is Optane technology implemented in a DIMM form factor that plugs directly into motherboard system memory slots and is pin-compatible with DDR4. The solution uses a modified interface called DDR-T, so can’t be plugged into just any motherboard. Optane PMEM requires processor and chipset support, specifically Intel second-generation Xeon Scalable Processors or later.
Intel recently released the second generation of Optane Persistent Memory based on generation two 3D XPoint with four-layer technology, improving the power consumption and performance compared to generation 1 devices. Capacities remain the same at 128GB, 256GB and 512GB versions. Although increased capacities could be delivered (density doubled from two to four layers between generations), Intel has stated that customer demand is driving capacity requirements and so devices greater than 512GB per DIMM aren’t needed yet.
Storage as Memory
Optane PMEM introduces an exciting challenge for applications and operating system designers as the contents of Optane DIMMs retain their contents across power cycles. Standard DRAM resets system memory at power-up, and operating systems expect DRAM to be volatile and unused at boot time.
To ensure compatibility and ease adoption, Intel has introduced two operating models for Optane PMEM.
- Memory Mode – Optane DIMMs are volatile and act as part of main memory, with DRAM used as an L4 cache by the processor and not directly addressable by the O/S. In Memory Mode, applications and operating systems can run unmodified, simply seeing a larger capacity of DRAM than usual, which is effectively volatile Optane and a physical DRAM cache.
- App Direct Mode – DRAM acts as standard memory, while Optane PMEM is exposed to the O/S as a special block device that can be treated as memory or used for a local file system.
Both modes may also be operated together, using DRAM and Optane PMEM in a combination of both methods at the same time.
How will these technologies be adopted? The simplest option is to use Memory Mode and increase the available logical DRAM capacity. Naturally, this will be slower than an all-DRAM solution but benefit from increased in-memory capacity and lower cost. We recently recorded a Storage Unpacked podcast with Nikita Ivanov from GridGain Systems, where we discuss the challenges of in-memory computing. The Memory Mode option would potentially fit within the Apache Ignite architecture.
App Direct Mode requires application modifications unless of course, you can put a memory manager in place to optimise between DRAM and Optane. This process is how the Big Memory solution developed by MemVerge works. Using the LD_PRELOAD feature in Linux, MemVerge can to virtualise DRAM and Optane to provide a much larger logical DRAM capacity. More interesting is how this technology can be used to create image snapshots of running applications. You can learn more from this Storage Unpacked podcast episode.
The Architect’s View
Intel Optane is at the cusp of development with so much to offer in the future. Today’s products implement only four 3D layers, so if the promise of 3D-NAND can be reflected in Optane, we can expect multi-terabyte devices soon.
Like all new technologies, demand drives development, which, in turn, drives further demand. As recently reported by Jim Handy, Intel could now be making a profit from Optane, which bodes well for the market. It means we may see much wider adoption and the economies of scale that make Optane viable for an even wider number of applications.
Copyright (c) 2007-2020 – Post #6675 – Brookend Ltd, first published on https://www.architecting.it/blog, do not reproduce without permission.