The Evolution of Solid State Arrays

In the first wave of solid-state storage arrays, we saw commodity style SSDs (solid state drives) being added to traditional storage appliances. This solution provided an incremental benefit in performance over spinning hard drives, however the back-end technology in these arrays was developed up to 20 years ago and was purely focused around driving performance out of the slowest part of the infrastructure – the hard drive. Of course SSDs are an order of magnitude faster than HDDs so you can pretty much guarantee SSDs in traditional arrays results in underused resources, but is premium priced.

Wave 2 of SSD arrays saw the development of custom hardware, mostly still continuing to use commodity SSDs. At this point we saw full exploitation of the solid state capabilities, with architecture designed to provide the full performance capabilities of solid state drives. These arrays removed unnecessary or bottlenecking features (like cache) and provided much more back-end scalability. Within the wave 2 group, Nimbus Data have chosen a hybrid approach and developed their own solid state drives. This gives them more control over the management functionality of the SSDs and subsequently more control over performance and availability.

Notably, some startup vendors have taken a slightly different approach. Violin Memory have chosen from day 1 to use custom NAND memory cards called VIMMs (Violin Intelligent Memory Module). This technology removes the need for NAND to emulate a hard drive and for the interface between the processor/memory & persistent memory (e.g. the NAND) to go across a hard drive interface like SAS using the SCSI protocol. Whilst it could be debated that the savings from removing the disk drive protocol could be marginal, the use of NAND that doesn’t emulate hard drives is about much more than that. SSD controllers have many features to extend the life of the drive itself. This includes wear levelling and garbage collection, features that could have a direct impact on device performance. Custom NAND components can, for instance allow wear levelling to be achieved across the entire array or for individual cell failures to be managed more efficiently.

Building bespoke NAND components isn’t cheap. Violin have chosen to invest in technology that they believe gives them an advantage in their hardware – no dependency on SSD manufacturers. The ability to build advanced functionality into their persistent memory means availability can be increased (components don’t need to be swapped out as frequently – failing components can be partially used).

At this point we should do a call out to Texas Memory Systems, recently acquired by IBM. They have also used custom NAND components; their RamSan-820 uses 500GB flash modules using eMLC memory.

I believe that the third wave will see many more vendors looking to move away from the SSD form factor and building bespoke NAND components as Violin have done. Currently Violin and TMS have the headstart. They’ve done the hard work and built the foundation of their platform. Their future innovations will probably revolve around bigger and faster devices and replacing NAND with whatever is the next generation of persistent memory.

Last week, HDS announced their approach to full flash devices; a new custom-build Flash Module Drive (FMD) that can be added to the VSP platform. This provides 1.6TB or 3.2TB (higher capacity due March 2013) of storage per module, which can then be stacked into an 8U shelf of 48 FMDs in total – a total of 600TB of flash in a single VSP. Each FMD is like a traditional SSD drive in terms of height and width, but is much deeper in size. It appears to the VSP as a traditional SSD.

The FMD chassis is separate to the existing disk chassis that are deployed in the VSP and so FMDs can’t be deployed in conjunction with hard drives. Although this seems like a negative, the flash modules have higher specification back-end directors (to fully utilise the flash performance), which, in addition to their size, explains why they wouldn’t be mixed together.

Creating a discrete flash module provides Hitachi with a number of benefits compared to individual MLC SSDs including:

Higher performance on mixed workloads
Inbuilt compression using the onboard custom chips
Improved ECC error correction using onboard code and hardware
Lower power per TB consumption from higher memory density
> 1,000,000 IOPS in a single array

The new FMDs can also be used with HDT (dynamic tiering) to cater for mixed sub-LUN workloads and of course Hitachi’s upgraded microcode is already optimised to work with flash devices.

The Architect’s View™

Solid state storage continues to evolve. NAND flash is fast and has its foibles but this can be overcome with dedicated NAND modules. Today, only four vendors have moved to dedicated solid-state components while the others continue to use commodity SSDs. At scale, performance and availability, when viewed in terms of consistency become much more important. Many vendors today are producing high performance devices, but how well will they scale going forward and how resilient will they be? As the market matures, these differences will be the dividing line between survival and failure.

Disclaimer: I recently attended the Hitachi Bloggers’ and Influencers’ Days 2012. My flights and accommodation were covered by Hitachi during the trip, however there is no requirement for me to blog about any of the content presented and I am not compensated in any way for my time when attending the event. Some materials presented were discussed under NDA and don’t form part of my blog posts, but could influence future discussions.

The Architect’s View™

Related Links