VMworld 2021 - Bringing it all together with Project Ensemble

As part of VMworld 2021, VMware announced Project Ensemble, a unified management platform for cloud and on-premises infrastructure. Haven’t we been here before with the concept of a single management plane to “rule them all”?

Background

Like all VMware tech previews, Project Ensemble is a plan to test the market with specific technology to see whether the concepts and ideas of a new solution will resonate with the VMware customer base. There are no guarantees that the initial intentions of a tech preview will develop into a product, while any final solution could be widely divergent from the original concept. However, many ideas are clearly based on customer feedback or a perceived new marketplace that VMware wants to exploit.

With Project Ensemble, VMware aims to tackle a problem that has existed for decades, and that’s the management of disparate resources spread across many physical and logical locations.

Resource Management

While the public cloud has existed in some form for the last 15 years, arguably, full-scale multi-cloud is less than a decade old. Both Azure and GCP released generally available commercial services in the 2010s, Oracle Cloud launched in 2016, and IBM started the public cloud journey in 2014 with the acquisition of SoftLayer. AWS was clearly first to market with Amazon S3 in 2006 and then EC2 in 2008. There were also smaller vendors of specific public cloud services, such as Virtustream, founded in 2009.

Look outside of the public cloud, and we can see attempts to create unified management tools in areas like storage. SMI-S was one example, with the noble objective to create a set of standardised APIs for all vendors to adopt. Individual vendors have created platforms – Dell EMC with Control Center and various variants, HPE with AppIQ. Neither SMI-S nor vendor solutions have stood the test of time, mainly for two reasons.

API management didn’t exist until popularised by SolidFire. This made storage management challenging to automate and normalise, as vendors all had their individual designs and configuration schemes.
Vendors wouldn’t unlock the core secrets to managing their products. No vendor is going to provide a competitive advantage to another unless absolutely necessary.
SMI-S definitions effectively dumbed down the set of management processes to everyday provisioning tasks, with the risk of losing some advanced features.
Storage management platforms risked creating additional monoliths with significant investment time and effort. This has the challenge that the “tail wags the dog” with storage management platforms impacting the ability to roll out new storage solutions.

Thankfully, the adoption of APIs in storage management has effectively killed off SMI-S, and other initiatives like Swordfish seem to have gone nowhere. Storage management functionality is becoming increasingly automated and should be totally abstracted and hidden by the end of the decade.

Application Management

There are other examples in the industry of attempts to centralise and standardise management. ZeroStack is a cloud-based SaaS layer to manage multiple OpenStack deployments. Platform9 is a multi-cloud management tool for Kubernetes, which we first saw at Tech Field Day 10 in 2016. More recently, we’ve seen Morpheus Data emerge as a multi-cloud management solution (we saw them at Cloud Field Day 3 and recorded a Hybrid Cloud Podcast with Brad Parks in 2019). NetApp acquired StackPointCloud in 2019 to build out NKS (subsequently shelved) but still has the Fabric Orchestrator solution. Rancher Labs (now part of SUSE) has a cross-cloud Kubernetes management tool (we used it to demonstrate K3S), while Spectro Cloud also has ambitions to be a cross-platform Kubernetes management solution. Even Hitachi Vantara has a Kubernetes Service.

These offerings are only a quick list of the solutions on which we’ve been briefed; there are undoubtedly many more in (for example) the Cloud Native Ecosystem.

Ensemble

Why could the VMware approach be different? To answer that question, we need to look at what problem is being solved. First, we need to recognise that infrastructure management at scale is a challenge. Businesses can’t afford to grow IT staff numbers in line with the increased use of computing resources, so more cost-effective management is required.

Second, with the continuous transition to a cloud-based model, IT organisations must focus less on infrastructure and more on applications. Data and applications are the units of management, not virtual machines or containers. A VM, for example, is just a deployment choice that can be dynamically changed over time.

Third, we need to focus more on the separation of management and consumption in our infrastructure and applications. Some teams will have the responsibility for ongoing infrastructure management and optimisation. Others will simply want automated ways to consume that infrastructure.

Cloud Journey

With so much focus on public and hybrid cloud today, the time is right to review the management infrastructure that will support the use of multiple and hybrid clouds.

If we look back at this post from 2018, we can see the stages of cloud adoption and align them to the features of Project Ensemble. The design of Ensemble addresses two main aspects of cloud management we’ve discussed, namely a focus on applications and the use of persona to expose the features of the platform most appropriate to the user by job function. But the vision has a lot more to offer. VMware wants to build in features like automated SLA management, automated application discovery and resource optimisation. Much of what has been announced reflects our five stages of Hybrid Cloud, albeit with much more of a VMware focus.

The Architect’s View™

What could the announcement of Ensemble say about a future VMware? In this post from 2016, we suggested that we’d seen “peak VMware” without some radical pivot. Arguably, that pivot came in the form of VMware on AWS, Azure, GCP, and Oracle Cloud in their various incarnations (collectively VMC). We can’t see what percentage of revenue is generated from these services, however, VMC, the announcement and release of Tanzu, plus the direction indicated by Project Ensemble all indicate a future for VMware that centres on multi-cloud, multi-platform management.

This direction offers access to a market that is as deep as it is wide. VMware is effectively de-risking its position of being just an on-premises server virtualisation solution, pretty much as we described in our post from 2016. We wonder how much of this strategy can be attributed to Pat Gelsinger and his experience at Intel, especially the mentoring from Andy Grove and his focus on continuous re-invention and inflection points.

If the direction of Project Ensemble is to (eventually) address native cloud management, then we can see VMware moving from a position of private cloud and public cloud “adjacency” to one that directly integrates with the platforms operated by VMware partners (e.g. AWS, Azure and GCP). This strategy will require a delicate balancing act by new CEO Raghu Raghuram, although as he previously focused on the cloud business, this appointment could be another indication of VMware’s increasing transformation to a hybrid cloud model.