Storage Predictions for 2024 and Beyond (Part III

This is the third of a multi-part series looking at technology infrastructure predictions for 2024 and beyond. It builds on discussions from previous posts to predict what we might expect in the segment of software-defined storage from both a short and long-term perspective.

Background

Software-defined Storage, or SDS, is now a well-defined part of the infrastructure market, as the evolution of the public cloud has transformed storage into a feature as much as it is an appliance. Over the last decade or so, the x86 architecture has become dominant for delivering storage solutions, with a gradual demise in bespoke hardware platforms.

The Arm instruction set architecture is used in some solutions (notably SoftIron and some smaller midrange or SMB solutions), while the use of GPUs has been used tactically to provide complex computing offload for RAID and erasure coding calculations (Nyriad and GRAID Technology are examples of this).

We previously highlighted six relatively distinct phases through which the evolution of software-defined storage can be tracked. We now think there is a seventh phase, developing from the use of SDS in the public cloud. Here is the list.

Hardware separation – the unbundling of the connection between hardware and software that enabled end users to take commodity servers and build storage solutions.
Bespoke SDS solutions – mainly driven by the object and file storage solutions, this phase saw the rise of storage software specifically designed to be run on commodity hardware.
Bespoke SDS appliances – as the market matured, storage software vendors worked with hardware manufacturers to build reference architectures and designs that deliver more reliability, improved predictability, and scalable performance.
Abstracted SDS solutions – a new wave of products where Quality-of-Service (QoS) and the delivery of storage were based on abstracted metrics rather than being a direct function of the underlying hardware.
Partner model – vendors taking their SDS products and selling as part of an integrated stack with solutions and infrastructure providers.
Container-friendly storage – also known as container-native storage (CNS), which runs in a Kubernetes cluster. We discuss this category in a separate post.
Cloud-friendly storage – as the public cloud continues to dominate, many vendors have evolved, rewritten, or developed storage solutions that work within the public cloud, turning virtual instances into storage appliances or HCI platforms.

These phases aren’t distinct transitions but overlap and don’t have specific timescales. The last phase (cloud-friendly storage) is one we’ve mentioned previously but is now gaining much more traction with the availability of bare-metal instances and locally attached NVMe storage with very low latencies.

Diversification

One beneficial aspect of SDS is the ability to create storage solutions that address a wide range of requirements. We mostly think of SDS for storage systems, but the technology encompasses file systems, unstructured data stores (for both object and file), databases and the containerised implementations classified as CNS. We will highlight open source and CNS in separate posts.

One area to watch is the cloud-friendly storage model. Silk, Weka, Volumez and Lightbits Labs all have solutions that deploy in the public cloud and are capable of delivering microsecond-level latencies. We will cover these in a separate Pathfinder report in February 2024.

Predictions

So, what can we expect for SDS in 2024 and beyond?

Everything SDS. We’ve already discussed how x86 has become the primary platform for software-defined storage, and it may seem evident that SDS will be everywhere. As silicon evolves, new processors now have dozens of cores, acceleration instructions that assist erasure coding calculations and access to high-speed interconnects like PCI Express. Arm will offer an alternative for some storage requirements, potentially those where performance, cost and environmental overheads are important.
Increased Integration. We highlighted this aspect previously, initially focusing on the additional APIs exposed to manage SMR hard drives and NVMe SSDs (namespaces). We can see this integration going further as media evolves and diversifies. As an example, Solidigm has addressed the metadata issue for large SSDs with a software solution called CSAL that caches data on a smaller SLC device. The introduction of CXL will see storage integrated with high-performance memory devices, potentially allowing more complex storage solutions to develop.
Improved Data Management. SDS offers more than just simple data storage. We’ve seen companies like Hammerspace introduce distributed file storage while solutions from Quobyte and Qumulo enable data access anywhere. Komprise and Datadobi are good examples of companies with solutions that can address ILM and tiering needs.
SANs in the Cloud. High-performance virtual instances, fast networking and low-latency NVMe storage are driving the market for distributed storage in the public cloud. It may seem odd to build storage into platforms that have storage already available, but these solutions can offer improved performance, availability, and cost savings over native storage features.
Improved Data Mobility. Increasingly, there will be a need to move data between fast and slow media, as the growth of AI demands ever-increasing volumes of unstructured data to process. One fundamental tenet of the storage industry is the need for tiering as data volumes grow. AI and HPC processing are no exceptions, where data will need to be moved in and out from cheap media, either HDD or tape. Weka already enables a file system to be offloaded to AWS S3 storage, while Hammerspace recently announced support for tape. Replication in and out of the public cloud already exists and is likely to become even more critical as data volumes increase.
Storage as a Service. Storage is now a feature, as well as being part of infrastructure. Within a dynamic application environment, storage resources are consumed and released on-demand. CSI provides some of this in the public cloud, while APIs for storage platforms have extended automation capabilities. We can see the requirements and capabilities being developed to the point where entire file systems are moved around on demand.

Although we’ve seen only limited capabilities, we think that SDS has a role to play in expanding data protection and ransomware prevention. SDS stores and retrieves data at a fundamental level where snapshots and rate-of-change detection are possible (if not trivial to implement). It should be possible and practical, therefore, to build in additional protection capabilities that expand on traditional snapshots and leverage some of the capabilities of CDP (continuous data protection), which seems to have waned in the industry.

The Architect’s View®

SDS is the core of data storage in modern IT. The proliferation of cheap and reliable hardware has driven the market towards standardisation at the component level that enables storage software to be run anywhere x86 computing exists.

Interestingly, the improvements in software-defined storage are being driven by the evolutions in hardware. Specifically, at the device level this is being achieved with extended APIs and with new technologies such as CXL. Even as we think of the IT world as software-defined, hardware still plays a significant part – hardware and software are always entwined.

In the next few years, we believe that storage in the public cloud will expand significantly as SAN-like solutions pervade. Once this happens, data mobility for traditional applications will be more practical. This transition could see a new age of solutions that enable mobility into (or out of) the public cloud, which has been traditionally a challenge to achieve. Software-defined is the future of all storage as solutions continue to diversify and expand.