Storage Predictions for 2023 and Beyond (Part III - SDS)

This is the third post in a series looking at predictions for the data storage industry in 2023 and beyond. You can find the first two posts in this series here:

Software-defined Storage, or SDS, has evolved from an option to stop vendors price gouging hardware sales to one where hardware and software are once again closely combined. How will SDS evolve over the next decade?

SDS Defined

In previous discussions, we attempted to put a definition around SDS. As we enter 2023, it’s clear that almost every storage solution is software-defined in some way. We previously noted in part two of this review that storage had evolved to the point where the vast majority of solutions are built on commodity x86 architectures (with some Arm creeping in). There may be some bespoke designs in there, but the basic concept is the same in that most functionality is now delivered in software with a degree of hardware acceleration.

Since its inception, SDS has moved through six relatively distinct phases.

Hardware separation – the unbundling of the connection between hardware and software that enabled end users to take commodity servers and build storage solutions.
Bespoke SDS solutions – mainly driven by the object and file storage solutions, this phase saw the rise of storage software specifically designed to be run on commodity hardware.
Bespoke SDS appliances – as the market matured, storage software vendors worked with hardware manufacturers to build reference architectures and designs that deliver more reliability, improved predictability, and scalable performance.
Abstracted SDS solutions – a new wave of products where quality of service (QoS) and the delivery of storage were based on abstracted metrics rather than being a direct function of the underlying hardware.
Partner model – vendors taking their SDS products and selling as part of an integrated stack with solutions and infrastructure providers.
Cloud-friendly storage – as the public cloud continues to dominate, many vendors have evolved, rewritten, or developed storage solutions that work within the public cloud, turning virtual instances into storage appliances or HCI platforms.
Container-friendly storage – also known as container-native storage, which runs in a Kubernetes cluster. We will discuss this category in a separate post.

Each of these solutions isn’t one following on from the next. The timeline of each phase is intertwined and more complex than a simple progression. It’s fair to say that once a new model emerges, it doesn’t disappear but forms part of the continuum of options. SDS has been a quiet transformation and a big success story in the industry over the last 15 years.

Open Source

One of the significant drivers of storage software has been the success of the Open Source movement. We’ll cover this area in a separate post. However, we should highlight here that Open Source and the standardisation on the Linux O/S have enabled developers to create new solutions relatively easily. Initiatives like the Cloud-Native Computing Foundation help promote products through a development phase that encourages experimentation. Take a quick look at the Cloud Native Landscape for Storage, and you’ll see a mix of solutions for block, file and object, data protection and file systems.

Perhaps the only downside of the explosion in storage solutions is the time and effort required to keep track of them all. Inevitably, some SDS will be successful, and others will fall away, making it challenging to pick the winners out from the losers.

Predictions

What can we see for SDS in 2023 and beyond?

Everything SDS. Possibly the most blindingly obvious prediction of them all. Intel (and AMD) x86 is powerful enough to deliver to the most I/O intensive requirements. There’s a proliferation of cores available to developers, with new instruction sets, faster DRAM, faster I/O channels, faster networking, and storage. The Arm market offers options for building more efficient solutions, so we could see SDS gaining more traction on this platform for specific use cases, like object and file storage.
Increased Integration. Hardware vendors continue to expose more functionality to the server and operating system. Hyper-scalers are already capitalising on these features and anticipating more options in the future. We’ll cover this area in a separate post, but we can already see features like ZNS and NVMe namespaces offering more control over hardware. SmartNICs could extend this further, with software features offloaded to pre-process data, detect anomalies, and implement security functions.
Improved Data Management. SDS is now more than just the storage software itself. Vendors pull metrics from systems that use complex AI models to predict failures, manage capacity and perform in-place upgrades. We’ve still yet to see real traction in ILM (most solutions are add-on vendor tools), except perhaps Hammerspace, which has a rich set of functionalities at the file layer. SDS is used to provide the additional data management tasks and could be used to do more around challenges like ILM and tiering.
Cloud-Native Expansion. Cloud-native solutions will continue to grow, with HCI and dedicated “storage appliances” built into and on the public cloud. These solutions will be infrastructure-aware, and be capable of using the storage hardware capabilities of virtual instances and scaling up/down on demand. We’ll cover cloud-native storage in a separate post.
Improved Data Mobility. We’ve seen some improvements in data mobility, for example, Hammerspace (already mentioned) and Weka, with the ability to move data between clusters and object stores. NetApp, for example, now allows on-premises storage to replicate into AWS FSx for ONTAP. We need more data mobility capabilities if hybrid cloud is to succeed.
Storage as a Service. By this, we mean not purchasing on-demand but the ability for developers to consume and return storage resources as required by an application. CSI already provides this capability for Kubernetes, while cloud hyper-scalers enable storage to be provisioned by API. We need more removal of the human factor from storage consumption and more AIOps to optimise technology (part of the data management requirement).

The Architect’s View®

Software-defined storage has become a bedrock of the data storage industry. In less than two decades, the market has evolved to make storage a feature in application architectures rather than a hardware product. This positioning is where we should expect to see storage in the future. Storage needs to be a consumable resource in every aspect of the terminology, from the vendor-deployed solutions in the data centre to the consumption of storage in the public cloud.