VAST Data has Vast Ambitions

At the start of 2019, VAST Data emerged from stealth with a new scale-out storage architecture aimed at high-performance data processing. In that time, the VAST platform has evolved, adding new features and functionality (see the latest announcement of VASTOS 4). We recently met with CEO Renen Hallak to learn more about what the future holds for VAST. If Mr Hallak’s ambitions are met, then the future of data processing could be very different from the way we work today.

Background

VAST Data introduced a new architecture into the enterprise that spans traditional NAS and object storage platforms. NAS provides access to unstructured data through file access protocols like NFS and SMB, while object stores expose data using web-based protocols like HTTPS.

In both instances, the underlying solution implements a scale-out data plane that can grow to tens of petabytes of capacity, all accessible through a single namespace.

New Technology

The VAST Universal Storage platform takes advantage of two types of new media. Intel Optane provides high endurance (although at a relatively high cost), while QLC NAND offers high capacity at low cost (albeit with much lower endurance). The architectural design of VASTOS ensures metadata and new data initially hits Optane, while QLC NAND acts as the capacity layer and delivers fast read performance. You can read more about the architecture in this blog post and two podcasts we recorded with Howard Marks in 2019 (part 1 and part 2). Note that the dependency on Optane is not absolute. VAST has qualified Kioxia FL6 SSDs as an alternative, and other technologies may emerge to meet the endurance requirements.

Market Position

For existing enterprises using scale-out NAS for tasks such as AI, analytics and machine learning, VASTOS offers extremely high-performance throughput with good levels of latency. At multi-petabyte levels of scale, the solution is arguably easier to manage than traditional NAS and higher performing than conventional object stores.

Looking at the type of customers VAST is targeting, the price point of a solid-state platform is unlikely to be an issue; performance, parallelism and scalability are much more critical.

Phase 1

Two and a half years after emerging from stealth (and six years after launch), VAST continues to fill out the feature set required by large enterprises. VASTOS 4 introduces ransomware protection, NFS 4.1 support, and policy-based data isolation – all features demanded by enterprise customers. At the same time, management capabilities have been improved, including the ability to monitor deployments through an online SaaS portal.

The maturity of the VAST platform is good, but at this stage, the company has effectively only introduced a competitor to Isilon, DDN, Scality, Cloudian and a host of other file and object storage providers.

Whilst the architecture represents an elegant solution for adopting new media, history tells us that incumbent vendors have a way of adapting their solutions to be on a par with new entrants. If the likes of Dell, HPE and Pure Storage want to compete, they can transform their products to be “good enough” and make the friction of moving to another supplier just too unappealing. VAST Data has perhaps a two or three-year advantage that represents an opportunity to keep ahead of the competition.

So, what’s in the vision of the future that could keep VAST Data leading the data management landscape? Let’s first examine where the wider industry is heading.

The Cloud Effect

The public cloud has had a transformative effect on business and information technology in general. Two decades ago, IT organisations focused on on-premises infrastructure, with a greater emphasis on managing the physical assets than the data on them.

Cloud introduced a paradigm where infrastructure is de-emphasised in favour of services and data. End users don’t need to care how the technology is implemented and on what hardware (although the degree of indifference is debatable).

Developers and business users want storage to be globally accessible, deliver a performance level optimised against cost and have infinite scalability. The use of object-based protocols provides abstraction and accessibility across the network from anywhere, while NFS/SMB provide granularity, locking and structure.

Idealistic

At the most abstracted level, storage needs to be reliable, globally available, cost-effective and deliver the performance required by applications. Decades of storage development has focused on these attributes with incredible success. Part of this evolution has been to centralise storage then make it accessible to applications in core data centres.

Again, with the public cloud, this model has changed. Compute and application code have become relatively ephemeral, while storage increases in inertia due to the volumes created by both humans and machines. Both compute and data are more distributed than ever before, and if we’re to believe the hype about edge computing, centralising data just to make it more accessible seems like a lot of wasted effort.

A much more practical first step is to look at how data is accessed. Technologies such as NVIDIA GPUDirect provide a more direct path between mass storage and GPUs. In the VAST Data solution, storage is exposed as NFS over RDMA, providing high throughput for applications without bottlenecking the CPU in the server hosting the GPUs.

New Hardware

Another technology gaining momentum is SmartNICs, also called DPUs, IPUs or computational storage. These solutions offload processing to a NIC or accelerator card, either to process data from the network or to accelerate tasks generally done by the CPU.

These products could be used to run the stateless protocol containers used as the access point into VASTOS. In this scenario, the SmartNIC exposes an NFS interface, block device, object front-end, GPUDirect API or any other type of API that can be consumed as a data source by the host. The only requirement is the capability to provide NVMe-oF connectivity from host to storage – a process already in place today.

Another solution on the horizon is CXL or Compute Express Link. The CXL technology will provide a standardised interface that enables memory sharing between server CPUs and PCIe 5.0 devices, with cache coherency.

With these solutions in mind, future designs will build servers with massive internal bandwidth and many cores, making the server a conduit for getting data into a GPU, computational storage device, or hybrid processor like NVIDIA Grace.

Distributed Processing

Ten years ago, application processing was based on virtual machines and traditional SQL databases. The world has since diversified, with containerisation reaching maturity through Kubernetes, and serverless promising “on-demand” processing for data.

Database models have increased dramatically, with new solutions implementing time-series and NoSQL architectures. Businesses now create data lakes with petabytes of structured and unstructured data.

The way in which data is processed has changed. AI deep learning algorithms process data continuously, as new information adds to the accuracy of models. The loop of process, refine and reiterate means data is constantly processed, with high degrees of randomness. Much of this data could be at edge locations and require pre-processing before subsuming into core locations.

Phase 2

What does this mean for the future of VAST Data? The Universal Storage platform is in place, with each release of VASTOS increasing the platform maturity for the enterprise. The next step is to perhaps look at how and where data will be processed in the future.

In the VASTOS architecture, data is stored on enclosures and accessed through stateless servers that are implemented through containers. This enables the VASTOS platform to expose data directly into a containerised environment or run the code on dedicated hardware. We can see some possibilities:

A Kubernetes plugin or operator that directly exposes data to a containerised application, irrespective of where that container runs. The local container is instantiated with the application and lives until the application shuts down.
An implementation of the VASTOS server that runs directly on a DPU, exposing data to the host without the need for host OS connectivity via NFS/SMB or S3 over the traditional network.
Edge-based computational storage devices that run enough code to take edge-created data and add it into a Universal Storage cluster.
Partitioned data pools across geographies that provide universal and consistent access (single namespace) but are biased towards a single location for latency purposes. Data could be both eventually and strictly consistent.
Clusters of clusters – multiple individual universal storage clusters grouped together to form a single large namespace.

These evolutions of the DASE architecture would provide more widespread access to run code wherever data is placed, rather than be forced to centralise content before processing. Obviously, data can be centralised afterwards.

The Architect’s View™

The future of data is undoubtedly the growth in unstructured content. While traditional structured data will remain important, the next decade will see a transformation towards on-demand processing as the content is created, rather than through human interaction.

The challenge for VAST Data is to harness the benefits of new technology to make efficient universal access a cost-effective process. This task is well under way. The opportunity for VAST Data is to change the data access paradigm and deliver on the promise of data ubiquity directly into the hardware platforms that are accelerating data processing. The partnerships with the likes of NVIDIA and other HPC vendors are the areas to watch, including support for CXL.

One last thought; where does VAST Data sit in the public cloud ecosystem? To date, there has been no indication of how the company intends to work with public cloud vendors. We could envisage a co-location model with VAST enclosures delivered through Equinix data centres, high-speed interconnects and the thin server model already outlined above. Imagine the opportunity presented by unlimited high-performance scale-out storage and unlimited access to GPU processing power on demand provided by the public cloud…..