The Rise of Ultra-Fast Storage Appliances

Updated 16 October 2017 with new companies.

The early days of shared storage were focused around fixing some of the problems experienced with having multiple disk drives in every server, namely – availability, resiliency, shared access and efficiency. Shared storage arrays resolved many of these issues by providing centralised shared, optimised and managed storage resources. The popularity of shared storage was made possible by Storage Area Networks, SANs and the connectivity provided by Fibre Channel, iSCSI, NFS/SMB and to a certain degree FCoE.

As these systems have evolved, hard disk drives were seen as the first bottleneck, so solutions were ameliorated with cache, NAND flash and eventually transformed into all-flash solutions. Fairly quickly the network was identified as the next issue, so solutions were developed to cache within the host and remove the storage network latency. Whilst these solutions are good, they increase complexity and can increase cost, unless the external storage can be completely replaced by a cheaper solution. We’ve also seen the evolution of NAND flash for DIMM slots (so called NVDIMMs) that provide a future potential to move persistent data closer to compute.

Andy Bechtolsheim introduces DSSD at EMC World 2014

Looking back at the original aims of shared storage, the benefits of efficiency, shared access and some availability features are lost when moving to caching solutions. Each host has to have cache devices installed; shared access can only be achieved through shared backend storage (and replication algorithms to manage updates) and loss of the cache in a host has a direct impact on availability (the host may have to be taken down to replace the failing device). We’re now starting to see the emergence of storage solutions that offer the benefit of being shared while at the same time remove the performance bottleneck of the current networks & protocols. Currently three platforms are on my radar:

DSSD – Acquired by EMC in May of 2014, with the announcement made at that year’s EMC World (see the link for more information). At the time of writing, there are still no GA products from DSSD, although in typical EMC fashion there are plenty of stories out there to whet your appetite.
Mangstor – An awful name, but potentially an interesting product; Mangstor delivers shared flash storage using either an Ethernet (RDMA) or Infiniband-based fabric. I recently chatted to CTO Paul Prince at Cloud Expo in Frankfurt and was impressed by the simplicity of the solution and the ability to use existing networking hardware.
A3Cube – another start-up that has developed a bespoke PCIe HBA (the RONNIEE) and the ability to link these devices together via PCIe switch.

Each of these vendors are using new data fabrics based on NVMe, RDMA and physical connectivity such as Ethernet and Infiniband. The result is ultra-high performance shared storage. Mangstor for instance claims their NX series appliance is capable of 9GB/s (around 2.3 million IOPS) at under 200 microseconds latency (reads) and under 11o microseconds latency (writes).

These ultra-high performance solutions provide the horsepower to deliver new types of solutions such as those doing intensive Big Data or real-time analysis where the benefits of shared storage still apply.

The Architect’s View™

This brief introduction starts the discussion on the new category of storage appliances which I will dig into during the coming months. Looking at the specifications, I’m very much reminded of the early days of the more traditional all-flash systems – low capacity, limited features – but which ultimately expanded into a lot more as the market matured. I think we’re also at an inflection point in the deployment of applications where containerisation provides the perfect opportunity to combine bare-metal hosts running the likes of Docker with ultra-fast raw storage such as the Mangstor NX6320. Containers strip out all of the processing overhead and should provide the most appropriate solution to drive Ultra-Fast storage appliances to the maximum. I’ve yet to see any examples of this being done, but when we do, I think the future of shared storage will be assured for some time yet.

Update – 16 October 2017

Looking at the three companies on this list, DSSD as a project within Dell EMC has been closed down. Presumably the technology is being deployed elsewhere, and will be integrated into other solutions. Mangstor went very quiet, replaced their CEO and raised further funds in August 2017. The company still hasn’t released a new website, which has been “in development” for months. A3Cube appears to have moved towards an appliance model with a system called GRIFO.

In a relatively short space of time, the Ultra-fast market seems to have evolved to use commodity components as well as the proprietary hardware. We now have Apeiron Data Systems, E8Storage, Excelero and a number of companies developing NVMe solutions (Pure Storage, Kaminario). All of a sudden we’re seeing the same downfall for the initial flash vendors that didn’t have data services in their product and were totally hardware-focused. The industry goes around the same loop again…

The Architect’s View™

Update – 16 October 2017

Further Reading