Research Note: NetApp teases new disaggregated storage architecture for ONTAP at Insight 2024

At Insight 2024, NetApp discussed a future disaggregated storage design for ONTAP systems that will be a radical move away from the current clustered architecture introduced following the Spinnaker acquisition in 2003. We discuss (and speculate) what a future architecture might look like.

Background

During the Insight 2024 Day 2 Keynote session, SVP for Shared Platform Krish Vitaldevara introduced the concept of a disaggregated storage architecture as the next generation of hardware for ONTAP. As we can see from the relatively generic schematic shown on-stage (see below), instead of storage enclosures being dedicated to ONTAP controllers, the storage resources will be shared by any and all of the controller heads in what could be termed a “shared everything” architecture.

Cluster Mode

The current NetApp design originates from the architecture developed following the Spinnaker Networks acquisition in November 2003. The Spinnaker design, which was initially branded as Data ONTAP GX and released in 2006, is described in a paper linked here. Each node in the cluster comprises a server with dedicated storage. The nodes are linked together as HA pairs and scaled out to form a virtual namespace of storage capacity.

Storage resources, either file shares or LUNs, are allocated through volumes, and a multi-tenant entity called a storage virtual machine (SVM), previously known as a vserver. The distribution of file resources enables a single namespace to comprise data from multiple nodes in a cluster through a mechanism called junctions.

The cluster mode architecture overcomes many of the issues with the original ONTAP filer design, which was limited by the performance of the controller heads and the amount of storage capacity these controllers could support. Cluster mode provides the capability to expand to 24 nodes (or 12 HA-pairs), each with storage enclosures attached to them.

NetApp proposed disaggregated architecture

Fragmentation

Unfortunately, the cluster design has some drawbacks, all of which result from an architecture that isn’t fully shared. For example, spare capacity on one node pair can’t be used by another HA pair. If one node pair is at its performance/throughput peak but hasn’t used all its storage capacity, then that unused capacity could be wasted.

Similarly, as the failover relationship between nodes is in dedicated pairs, NetApp recommends that nodes run at no more than 50% of available controller performance. Logically this means that an entire cluster will only run at 50% of the available performance of the controllers, even if there are no failures or issues present.

Shared Everything

In a “shared everything” architecture, every node could theoretically have access to all storage resources. With this design, no storage would be wasted as it could be made accessible to any node.

The performance issue becomes less of a problem, as any node could act as failover for any other node. In this scenario, a cluster would only need to reserve free resources to absorb the workload of a failed node, which itself might not be running at 100%. A cluster could therefore be run at much closer to 100% capacity, rather than the recommended 50%.

Obviously, there are challenges in implementing a fully shared-everything design. Metadata that maps storage assignments needs to be shared and kept synchronised across all nodes in real time, resulting in potential network overheads and performance implications. However, with modern fast networking and RDMA, this problem can be mitigated. Similarly, NVMe over Fabrics (NVMe-oF) enables NVMe drives to be shared with minimal overhead and is a technique used elsewhere today (more on that in a moment).

SolidFire

Of course, we should remember that NetApp has experience in scale-out storage designs following the acquisition of SolidFire in December 2015. That platform comprised small compute nodes with integral storage that scaled out in a clustered design. However, the storage itself wasn’t fully shared, so the implementation is only partially what NetApp appears to be envisaging today. SolidFire was discontinued in October 2023. We know of at least one large NetApp client running the SolidFire platform, so the new architecture could be developed partly to fill that gap.

Implementation

So, we think NetApp could implement a design with dedicated controller nodes and separate storage nodes that incorporate an NVMe translation layer using NVMe-oF. Inter-node communication could be through RDMA, creating a design of dedicated networking, compute nodes and storage enclosures.

Some design constraints to avoid would include:

Fixed node types. Ideally, a cluster should be capable of supporting disparate node configurations, enabling cluster hardware to be upgraded over time.
Fixed O/S versions. Again, we would suggest that clusters should support multiple ONTAP versions, to enable rolling upgrades.
Fixed storage devices. An optimum design should allow for any storage device capacity or performance, with only the limitation of keeping similar devices within an aggregate.
Scale-up only. Solutions should be capable of scaling up and down (assuming sufficient capacity exists to remove resources). This capability would provide flexibility for consumption-based storage models and physical infrastructure reorganisation.

Much of the groundwork for a disaggregated ONTAP platform appears to be in place, as NetApp has worked hard over many years to virtualise and abstract physical storage and networking. Moving from the existing ONTAP platform to a truly disaggregated architecture may take a few steps, as there are many internal architectural designs to translate from the current paired architecture to a fully disaggregated solution.

The Architect’s View®

Putting aside the technical aspects for a moment, the concept of a truly scale-out ONTAP platform is very appealing. From an efficiency perspective, full scale-out would provide resource savings for both NetApp and customers, reducing $/GB cost, but also the space, power and cooling requirements of the infrastructure.

A scale-out design could deliver the requirements of storage-on-demand to be delivered more easily, scaling infrastructure up and down (hopefully non-disruptively). This logic also applies to long-term infrastructure deployments, where controllers, networking and storage could each be swapped independently. Rather than perform frequent data migration tasks, a disaggregated architecture would be like “Trigger’s Broom”, or the Ship of Theseus.

However, we should issue a word of caution. Following the acquisition of Spinnaker Networks, NetApp introduced ONTAP GX then attempted to merge the ONTAP 7-mode (previously known as 7G) systems with what became known as ONTAP Cluster Mode (or CDOT). This transition was not a great success for NetApp, as some features that previously existed in 7G release 7.3.2 were not available in 7-mode release 8.0. Initially, 7-mode to C-mode migration didn’t exist and required new hardware and a lot of manual migration work.

Customers will remember the imposition of the 7-mode/C-mode issues, and also the more recent debacle of EMC’s XtremIO disruptive upgrade. So, if there is anything we can advise NetApp, it’s to ensure that data migration from existing AFF and ASA systems to the new disaggregated system is achieved as seamlessly (and painlessly) as possible.

We are excited to see how the new NetApp disaggregated architecture will evolve. It’s rare these days to see new storage designs emerge, as the market is relatively mature.

There are obvious comparisons to be made with VAST Data and its disaggregated architecture, while Pure Storage has the hardware-based FlashBlade, which is also a scale-out design. HPE also recently updated its block offering (snappily called GreenLake for Block Storage with Alletra MP) to support a disaggregated architecture.

We also believe that the scale-out architectural design could provide NetApp with an opportunity to integrate StorageGRID more directly into the platform and develop a truly unstructured solution supporting S3 and file protocols.

Based on the comments made at Insight, we should know the outcome of the work before the next event takes place in Autumn 2025. Interesting times.