EMC, ScaleIO and The VMkernel

There’s been a bit of conversation recently on the intention of EMC to put drivers for their ScaleIO platform into the VMware ESXi hypervisor kernel. The discussion was triggered by an article published in LeMagIT on 25 November 2014 by Christophe Bardy and subsequently picked up by The Register. It implied that VMware and EMC were working together to integrate ScaleIO into ESXi. Naturally the discussion turned to ongoing friction and competition between ScaleIO and VMware Virtual SAN.

VMware felt the need to clarify the position in a blog post, positioning ScaleIO and Virtual SAN as different approaches to solving the storage problem. With the release of ScaleIO 1.31 we can see what this difference means and how the misunderstanding could arise.

First of all it’s worth stepping back a bit and looking at some features of the ScaleIO architecture. ScaleIO is designed to run on a distributed node architecture where each node is a physical or virtual server. Each node can consume or provide storage (or both) and therefore the product is designed to sit alongside the application on the same server. Consumers of storage run a component called SDC (ScaleIO Data Client). This presents storage to the host on which it runs in the form of an iSCSI LUN. Providers of storage run a process called SDS (ScaleIO Data Server) that takes storage and makes it available across Ethernet and Infiniband to SDCs.

In the storage stack, the SDC emulates devices that can be provided to the logical volume manager (and therefore to the file system and application). The SDS takes resources from the Logical Volume Manager and makes them available to the SDC across the network. In this way, ScaleIO can integrate with a system that is already configured with storage resources and make only a portion available to each SDC, or if the server is dedicated to ScaleIO, all storage resources.

The problem with implementing ScaleIO in ESXi is that all traffic from guest VMs (either as SDC or SDS) has to traverse the internal IP stack. This is explained in this somewhat clunky diagram from EMC, which is referring to storage presented back to ESXi itself, rather than storage being consumed directly on the VM guest. The change in design sees the SDC move to be part of the VMkernel (the right hand image), removing lots of hypervisor & IP overhead. This specific implementation is for scenarios where ScaleIO is presenting storage to the VM cluster itself and is effectively what hyper-converged vendors such as Nutanix and SimpliVity do, in their case using SVMs (storage virtual machines) running NFS, which is represented back to ESXi in order to create distributed datastores for storing VMs.

EMC claims that moving the SDC into the kernel (presumably distributed as a VIB – partner supported, not VMware supported) will see 2-3x improvement in performance (which is no surprise) – have a look at this EMC video for more details. However I think there is a “bigger picture” story here as to EMC’s move and that involves their entire ViPR ecosystem.

Today, natively, ESXi supports external networked storage through iSCSI, Fibre Channel, FCoE and NFS. Placing the ScaleIO SDC into the kernel means EMC can also provide support for external arrays that run ScaleIO SDS, such as ECS (Elastic Cloud Storage). The upshot of this is that EMC can now sell ECS into customers running VMware clusters with native support for high capacity storage – private cloud storage that integrates directly with their hypervisor. ViPR brings the management of all of these components together.

The Architect’s View

Integrating ScaleIO into the VMkernel isn’t about EMC having privileged access to position their product more attractively with VMware than their competitors. It isn’t about competing with Virtual SAN and it isn’t about bloating the VMkernel. In fact, it’s about providing customers with choice and removing the roadblocks for the adoption of platforms like ECS. As a proprietary protocol, ScaleIO needs a dedicated driver to be written that sits in the most efficient part of the VMware storage stack. With this release EMC are simply making ECS easier to consume and chipping away at customers’ reasons to say “no”.

One final thought; if our friends at Coraid had looked at this integration approach for AoE, then perhaps the company would be in a better position and more widely adopted….

The Architect’s View

Related Links