Storage Predictions for 2023 and Beyond (Part VI - Public Cloud)

This is the sixth post in a series looking at predictions for the storage industry in 2023. Previous posts:

The public cloud has claimed a substantial part of enterprise IT spending since the first object storage platform was introduced in 2006. In less than 20 years, cloud has become the dominant (and in many cases preferred) choice for new applications and a repository for massive data storage. What can we envisage for the future of cloud-based data storage throughout the rest of the decade?

Background

The history of the modern era public cloud started with data storage and specifically object storage. Amazon Web Services introduced the Simple Storage Service (S3) in 2006, followed in August 2008 by the Elastic Block Store (EBS) and then the Elastic File System (EFS) in April 2015. Other hyper-scalers have followed a similar pattern of product releases, which offer the traditional block, file, and object protocols across a mix of performance, capacity, and cost metrics.

Cloud-Native

Of course, what we’ve discussed so far are cloud-native solutions, those directly offered by the cloud platform and integrated with APIs, security/credentials and billing delivered by the hyper-scaler. Many other cloud storage solutions exist, some using cloud services and some simply running in virtual instances and purchased through marketplace platforms.

One very early example is Nasuni, a virtual filer appliance (also available as a physical appliance) that uses cloud object storage for both long-term retention and the flexibility of global mobility.

Another example is Backblaze, which started initially as a data protection company and has subsequently expanded into online object storage. Most recently, we saw NetApp announce FSx for NetApp ONTAP, an integrated solution that effectively implements the ONTAP operating system in AWS.

Protocol

Looking across the cloud storage landscape, the typical three storage protocols operate slightly differently from each other. Both object and file protocols expose resources that can be consumed outside of the native cloud platform, while block storage is generally restricted for use on virtual instances. This design continues to be the norm as hyper-scalers introduce locally attached NVMe devices to improve performance. In this respect, the public cloud mirrors a traditional data centre, with data replication and resiliency built into the platform but not exposed directly to the customer. End users can, of course, use native data protection features that range from simple snapshots to dedicated backup services.

Cost Efficiency

In the early days of object storage, hyper-scalers were keen to reduce end-user costs, as we highlighted in a 2019 blog post. These reductions then stopped and evolved into new features and functionality to reduce prices through a reduction in service. Typically, this means reducing access times through tiering or having fewer mirrored copies of data.

From the hyper-scaler’s perspective, this change in policy shouldn’t be a surprise. Vendors can’t continue to deploy disk-based storage ad infinitum because, as we know, most data in object stores is inactive. Tactical use of tape and possibly optical media make data storage more cost-effective and reduce overheads such as power consumption. This practice is likely to continue, especially with a focus on sustainability (more on this in our predictions).

Improvements

Storage in the public cloud continues to evolve. As we recently highlighted in this blog post, hyper-scalers continue to improve reliability and performance. All hyper-scalers, for example, have introduced some form of local NVMe storage that delivers improved consistency.

From a sustainability perspective, AWS is leading the way with automated features (lifecycle management, storage class analysis) for cost reductions and analytics. We’re also starting to see caching services implemented to improve performance for the most active data. We will review these services as part of a new report on cloud storage that will be due out later in 2023.

Predictions

What developments can we expect from the public cloud in the next few years?

Significant focus on efficiency and sustainability. The ball is already rolling with efficiency savings. Cloud vendors will continue to optimise costs, optimise performance, and provide customers with opportunities to save money. This isn’t an altruistic stance but is used as a marketing and sales strategy to generate and retain business. Customers, for example, who save 10-20% on storage are more likely to use the saved budget to store more data rather than spend less because the budget is already accounted for. We expect greater transparency on the environmental cost of storing data to be exposed to customers, aligning cost savings with reduced power consumption and CO₂ emissions (or the use of green energy).
Continued improvements in performance and reliability. The public cloud must improve reliability in line with increasing scale. For example, every “9” of improved reliability allows 10x the volume of data to be stored with the same absolute volume of failures or rebuilds. Reliability is essential to reduce network traffic rebuilds where customers implement scale-out storage solutions. There are arguably gaps in the feature offerings of the public cloud, although these issues are being addressed.
Increased use of proprietary “cloud SANs”. Vendors such as Lightbits Labs now offer their software solutions as “cloud SANs”, connecting multiple virtual instances together to provide resilient block storage. This model already exists with Pure Storage Cloud Block Store and scale-out filesystems from Weka. For customers operating at scale, the virtual storage layer can provide improved resiliency and reduced costs but is unlikely to be cost-effective for smaller customers. One additional aspect of this approach is to enable greater data mobility in and out of the public cloud with greater efficiency.
Extended support for popular storage platforms. NetApp and Microsoft have native file system implementations in AWS. We can envisage further popular platforms being integrated into the public cloud. The tipping point for the hyper-scalers is when the volume of data being stored in “cloud SANs” justifies the engineering work to adopt that platform natively (with a mutually acceptable business model).
Tactical use of the cloud to mitigate costs and supply chain. As supply chain issues persist, we envisage the increased use of the public cloud for tactical purposes, including mitigating supply chain issues and managing peaks in demand. NetApp’s FSx for ONTAP facilitates the use of SnapMirror to move data into the public cloud, making it relatively easy to incrementally update cloud copies. As applications become more portable, we see data mobility and agility improving.
Increased use of event-driven processing. AWS already offers Lambda processing for S3 data. We can see this process extending to data in and out of unstructured data stores, especially where that content is created and delivered from edge locations.
Cloud Storage competition will increase. The hyper-scalers will face further competition from bespoke solutions such as Zadara, Backblaze, Wasabi and CloudFlare. Currently, this market is focused on unstructured content and is likely to remain this way, as pricing remains the differentiating factor. As data protection moves to be more cloud-focused, large-scale unstructured data stores will become significant players.

The Architect’s View®

Throughout these discussions, there are two areas we haven’t covered. The first is online databases, and the second is the cloud “sync-n-share” market. Looking at the former, solutions such as FaunaDB and MongoDB Atlas have the opportunity to make database APIs the “fourth cloud storage protocol”. This is an early market but one that could be big. We think that the trust factor that needed to be gained for widespread S3 adoption will be the same cycle required for cloud database growth. For the latter topic, solutions such as Dropbox seem to be on a hiatus in features and functionality. This is clearly an area for more investigation.

In this blog post from 2022, we suggested a model for data management, which at the lower end, focuses more on storage than data. In this article, it’s clear that the lines between data storage and data management are becoming increasingly blurred, or to view it another way, intrinsically linked. Hyper-scalers already cover the four key areas we highlighted. As a result, future predictions need to include a data management discussion to align with data storage.