Storage Predictions for 2023 and Beyond (Part VII - Protocols)

This is the seventh post in a series looking at predictions for the storage industry in 2023. Previous posts:

Over the six posts in this series, we’ve looked at a broad range of issues in data storage. One topic that doesn’t get much coverage is the lower-level protocols. In this final post, we look at advancements in PCI Express, CXL, and storage protocols to determine what these solutions will do for the future of enterprise data storage.

Background

The term “protocols” covers a wide range of topics. At the component level, devices such as SSDs and HDDs connect to the host through external peripheral interconnects such as PCIe/NVMe and SAS/SATA. Further up the stack, hosts connect via storage protocols that include Fibre Channel, iSCSI, NVMe-oF and SCSI. At each level, there is a division between the physical connection and the storage protocol that overlays it. For example, NVMe is a new storage protocol using PCI Express interconnects and both Ethernet and Fibre Channel networks (NVMe over Fabrics).

For the last four decades, we’ve been using variations of the peripheral interface developed by Shugart Associates, specifically SCSI and the modern variants SAS and Fibre Channel Protocol (not the transport, confusingly, both are similarly named). SATA was developed in the mid-1980s from ATA/IDE as a low-cost connection for PC clones. Today we still see SAS and SATA in use across the industry, although NVMe has quickly become dominant.

NVMe

The development of NVMe was a big step forward in remediating many of the issues of the SCSI interface. The protocol brings in a high degree of parallelism that wasn’t needed with spinning disks or tape drives. The performance of NVMe is much higher than SAS/SATA, while the connectivity is simplified, with devices plugging straight onto the PCI Express bus. The standardisation and development of NVMe resolved the issue of custom drivers for add-in cards that had been developed by companies like Fusion-io.

Evolution

While NVMe continues to evolve, the SAS/SATA protocols have moved on too. We’ve covered the emergence of 24Gb and 24Gb+ SAS variants, both of which have many new features focused on the hyper-scaler market. Typically, these enhancements are centred on exposing functionality to the host that enables more device and data controls. We’ve extensively covered NVMe and SATA/SAS evolution in podcasts, which are listed here.

PCI Express

As we’ve already highlighted, NVMe devices plug directly into the PCI Express bus. In the last 18 months, we’ve seen the announcement of both PCI Express 6.0 and 7.0 protocols, which we’ve covered in recent podcasts. The gap between PCIe 3.0 and 4.0 standards introduction was seven years. However, the PCI-SIG has picked up the pace and expects to have 7.0 ratified by 2025. This equates to a shortened cycle of three years between major releases. This improvement is crucial because it affects the adoption of CXL (Compute Express Link), a new standard for high-speed CPU-to-peripheral communication. We covered CXL in a podcast broadcast in late 2021.

CXL

CXL promises to revolutionise system architectures with cache-coherent communications between the CPU and devices connected to the PCI Express bus. This could include expanding system memory, the enhanced use of accelerator cards and high-performance clusters. CXL has the capability to make software-composable infrastructure a viable technology for widespread data centre use.

Status Quo

At the front end of systems, not much has changed in the world of APIs (like S3), NFS or block-based protocols. These interfaces are pretty stable and unlikely to change much because they’re so embedded into our infrastructure ecosystem.

Predictions

So how will protocols develop over the next decade?

NVMe dominates, SAS/SATA enter severe decline. NVMe becomes the primary protocol for SSDs and (potentially) HDDs. As the cost of flash gains closer TCO alignment to HDDs, servers will drop SATA connectivity and eventually, SAS will become an option and not a default. Some of this change will be driven by new form factors and the desire for more efficient server operations. The only remaining opportunity for SAS is scalability in high-density storage systems. We see this being a niche requirement, increasingly only used by hyper-scalers and for inactive data in enterprises.

NVMe expands to support data-centric protocols. Vendors have tried and partially failed to move on from traditional block-based protocols for persistent media. Remember Seagate Kinetic drives? Kioxia has recently revived this idea with the EM6, which uses NVMe-oF and 25Gb Ethernet. Rather than put drives onto a network, we think a short-term gain can be achieved with enhancements to NVMe that read and write at the data level, not block. Pliops has just announced a new series of data services that push the database storage engine into the accelerator card, with a significant increase in performance. NVMe looks to be the best route for this technology (so far).

CXL opens up a world of new technologies. We’re only on the cusp of what CXL will offer data centre architects. Cache coherency provides a wealth of capabilities that offload tasks or cooperate with the core processor. While Intel and AMD continue to push greater functionality into the core CPUs (with power and cost implications), CXL could expand the functionality of solutions like ARM with discrete capabilities that are enabled on demand.

DPUs evolve quickly due to CXL. The DPU (IPU, SmartNIC or whatever you want to call it) will see increased traction with CXL but will need some standardised interfaces and APIs. Much of this work will need to address security concerns as data processing becomes more distributed, even within the confines of a single server.

The Architect’s View®

There’s still a lot of work going on inside the server, transforming what started out as PC technology into enterprise mainframe equivalents. New standards enhance the scalability of server architecture, moving us closer to rack-scale computing. In the data centre, SAS and SATA will become legacy connectivity and could be superseded by Ethernet-connected devices if the BOM for manufacture make sense. It’s still a fascinating time in the world of hardware, one that’s constantly driven by the need for greater performance and scalability.