Storage Predictions for 2021 and Beyond (Part III - SDS)

This is the third post in a series looking at predictions for the storage industry in 2021. The first two posts are here:

Software-Defined Storage or SDS was hailed as the saviour of enterprises looking to get away from price-gouging vendors and expensive proprietary appliances. In the last decade, has SDS delivered?

SDS Defined

There have been many attempts to put a definition around software-defined storage. The basic concept of being “software-defined” implies an abstraction away from hardware and the features of storage media. I like to think of SDS as delivering storage based on standardised metrics, including IOPS, bandwidth and latency. These metrics are aligned through Quality-of-Service functionality to the extent that changing the underlying hardware should have no direct impact on I/O performance.

The Stages of SDS

Initial SDS solutions were nothing better than hardware and software separation. Vendors sold software to run on commodity hardware and priced the software either on a capacity or hardware-based licence (e.g. cores or sockets). While the ability to use cheap hardware is attractive, it comes with challenges of compatibility, bugs and supply chain availability. This is why many SDS solutions are packaged as appliances or sold with hardware. I like to quote the stages of SDS as follows:

Hardware separation – unbundling of pricing and the independent deployment of storage software and hardware.
Bespoke SDS solutions – these include object and file storage, plus a wealth of block-based products/vendors.
Bespoke SDS appliances – software-based solutions sold as packaged appliances with certified and tested configurations. These offer predictable and scalable performance.
Abstracted SDS solutions – software products that deliver storage resources based on metrics and QoS, independent of the hardware used.
Partner Model – vendors taking their SDS products and selling as part of integrated solutions with solutions and infrastructure providers.

Arguably there is a further category evolving, and that’s container-attached storage, which uses containers to deliver storage resources, generally in the form of a Kubernetes cluster. We’ll touch on those solutions in another post.

Adoption

Software-defined Storage has been widely adopted as a standard deployment model across the industry. Even vendors with bespoke appliance-based solutions will implement new features in software. Many offer virtual appliances or cloud-based versions of on-premises storage hardware. It’s fair to say that SDS has been a quiet success across the industry. With the exception of high-end platforms that support mainframe attach, SDS is the dominant development and deployment model.

Circle of Storage Life

Over the past ten years, we’ve seen significant improvements in processing power, generally through the addition of cores to a single processor. Performance of DRAM has increased; bus speeds have increased, and new storage media has emerged. SDS has been in a strong position to exploit these advances.

As software evolves to take advantage of current hardware, so the cycle moves around for hardware to take centre stage once again. Enter SmartNICs, a technology to offload storage, networking and some security tasks to dedicated hardware. The SmartNIC movement is driven by public cloud vendors and the desire to optimise every component of the hardware landscape. This currently includes using ARM for general-purpose processing and SmartNICs with ARM cores and FPGAs.

The “SmartNIC revolution” is in the early days of development so we’ll need to wait and see how far down the stack (from big enterprise to SME) this technology will stretch.

Open Source

One of the biggest drivers of SDS over the last decade has been the success of Open Source. Enterprise computing is moving steadily away from proprietary operating systems to Linux (which does include proprietary distributions). There is now a wide range of open-source software products from file systems to entire platforms that encompass all types of data storage. This development is so advanced that there’s no requirement to purchase commercial products if you don’t need them. There are open-source object stores, filers and block storage readily available for both modern and slightly older hardware.

Predictions

What can we predict for SDS over the coming decade?

Everything SDS. Probably the most obvious evolution is that (almost) all storage resources will be software-defined. General-purpose CPUs are powerful enough to deliver to most I/O requirements across practically all businesses. Other than the SmartNIC revolution, there’s no logic in developing bespoke and tightly integrated hardware for storage, when commodity hardware can meet 90% of requirements.
Deeper Complexity and Integration. New media demands greater understanding from the software layer, whether that’s to mitigate the challenges of NAND flash, exploit persistent memory or use archive HDDs efficiently. This integration will result in new storage APIs that translate structured data into I/O primitives. Some of this is already coming with Ethernet SSDs, EBOFs (Ethernet Bunch of Flash) and solutions like MCAS. Linux already supports SMR drives with APIs and abstraction layers to optimise performance. Storage solution vendors are taking advantage of technology advancements to develop new storage architectures.
Better Data Management. Storage solutions have focused mainly on providing access to data over standard protocols. In the long-term, this strategy will be insufficient to deliver to the needs of AI and analytics, data mobility and challenges around data security. SDS needs to resolve the problems of ongoing data management through extensions to file systems and new APIs. For example, data protection and ILM should be functions of the file system.
Better Data Mobility. One of the biggest challenges of our age is the issue of how we manage access to data across geographic and technology boundaries. Today, data sits in silos, either on-premises or in the public cloud, with limited data mobility to provide access to data across platforms. There are point-solutions to the data mobility challenge but no winner in this market. SDS solutions need to evolve and address the mobility challenge. Both data mobility and data management challenges will be resolved through data awareness – understanding the structure and the needs of data and applications.

The Architect’s View

SDS has transformed the storage industry with a quiet revolution that means almost everything is software-defined. The winning storage solutions of the future will be those with flexible consumption models, zero dependencies on hardware and features that enable data management and mobility. As always, there are leaders and laggards, some of whom depend on being incumbent but will continue to see market share erosion. The next decade of storage is evolving quicker than ever.