Server virtualisation has become a standard method of application deployment, delivering operational and financial benefits compared to the legacy approach of “one server, one application”. Multiple virtualisation solutions now exist for enterprise customers. What are the differences, and how should one be chosen over another?
Background
Infrastructure virtualisation has its origins stretching back to the late 1960s. IBM first introduced hardware virtualisation capabilities into the System/360 model 67 platform, running CP-67 as a hypervisor. This was the forerunner to the Virtual Machine series of operating systems (starting with VM/370) introduced in 1972, running on the System/370 architecture. VM has transitioned through many versions and renames but is still available today as z/VM.
VM was the first practical hypervisor (what we generally refer to as a type-1 hypervisor) from IBM that allowed multiple operating instances (generally MVS and its descendants) to run on a single hardware platform. IBM also introduced virtualisation into mainframes with a feature called PR/SM that divided up the hardware into logical partitions or LPARs. The legacy of VM is still visible today in the names of companies like VMware and the hardware virtualisation extensions added to the x86 instruction set architecture (ISA), which we will discuss in a moment.
x86 Virtualisation
The virtualisation of x86 applications was developed by VMware in the late 1990s. At the time, the x86 ISA had no hardware-based capability to manage the segmentation and abstraction of multiple virtual machines onto a single operating system. VMware developed a system of binary translation which dynamically processed virtual machine code to intercept and manage privileged instructions.
Take, for example, the scenario where an operating system needs to read or write to a network device or persistent storage. This task is performed by the operating system using privileged instructions that run in the kernel. For a virtualised O/S, the privileged instructions must be intercepted and translated into code that talks to the hypervisor, which ultimately has the responsibility of managing device I/O (amongst other things).
VMware developed a virtual machine manager (VMM), which processed the virtual machine instructions, emulating privileged instructions to make the VM believe the I/O or network commands came from a real device. In reality, those devices are just code that acts and behaves as a physical device would.
VMware also developed techniques to implement memory shadowing and page translation to ensure VM memory and physical memory usage were aligned.
Hardware Virtualisation
Software virtualisation, or the binary translation performed by early VMware platforms, is open to security flaws due to inevitable bugs in software. Looking back at the mainframe days again, operating systems such as MVS had built-in hardware protection that restricted the execution of privileged instructions and partitioned system memory for user and privileged processes. This capability was baked into the hardware from the System/360 architecture onwards. Programs operated in either supervisor state (privileged) or problem state (non-privileged), with a specific hardware register (the program status word or PSW) keeping track of the privileged or non-privileged status.
In addition to program states, the System/360 architecture (and successors) also uses hardware-based memory protection through up to sixteen protection keys that determine which processes have access to partitions of system memory.
In the System/360 family, tasks that require privileged instructions are managed through supervisor calls or SVCs, which are similar in concept to an API for tasks such as I/O or other restricted functionality.
(Author’s Note: I’ve referenced mainframe systems here, as this is my experience, however other platforms of the 1960s and 1970s, namely MULTICS and VAX also had memory protection using the “ring” concept that still exists in Intel processors today.)
VT-x and AMD-V
In 2005, Intel released new models of the Pentium 4 processor incorporating technology codenamed Vanderpool, which we now refer to as Intel VT-x extensions (also called virtual machine extensions or VMX). VT-x adds a series of new instructions specifically designed to manage hardware-based virtualisation. AMD introduced similar technology in 2006 under the codename Pacifica and first introduced AMD-V extensions in Athlon 64 processors.
It’s interesting to note that when 64-bit processors were introduced in the early 2000s, Intel processors were deemed more secure than AMD processors due to the additional memory range checks on Intel processors (segment limit checks). AMD removed the checking when moving from 32 to 64 bits but re-instated the checking when it became clear the omission resulted in more insecure systems.
Both Intel and AMD VMX instructions introduce the concept of root and non-root operations. The virtual machine manager (VMM) will run in root mode, passing control over to a virtual machine guest, which will run in non-root mode, a process known as VM Entry. Certain privileged instructions will cause a virtual machine to pass control back to the VMM, a process known as VM Exit. Through this mechanism, the hypervisor can control access to privileged instructions and manage the process by which those instructions are handled.
Intel and AMD later introduced additional memory management capabilities that added extended page table translation. This functionality enabled direct physical to virtual machine page mapping rather than requiring the VMM to manage the process.
Intel and AMD both extended virtualisation to include direct device support (Intel with VT-d, known as Intel Virtualisation Technology for Directed I/O and AMD with AMD-Vi, or IOMMU virtualisation).
Para-virtualisation
All of the above techniques introduce hardware-based abstraction of CPU instructions, system memory and devices that enables an unmodified operating system to run as a virtual machine. The VMM must emulate and support the features of common hardware devices, such as SAS/SATA controllers, graphics cards and NICs.
An alternative approach is taken with para-virtualisation, where pseudo devices provide storage, networking, and other device support. Rather than create software that emulates real hardware, para-virtualisation creates virtual devices that only ever exist in software but provide similar or identical functionality to real hardware.
Para-virtualisation provides two benefits. Firstly, the VMM no longer needs to intercept privileged instructions, as the virtual device driver installed in the guest operating system knows how to talk directly to the VMM. Secondly, both the guest O/S device driver and the VMM code can be optimised to make the virtual device operate as efficiently as possible. This could, for example, mean removing code for specific operating modes that will never be used in a virtual environment.
The most well-recognised virtual I/O devices are the VirtIO framework developed by Paul “Rusty” Russell while creating lguest, a hypervisor that was part of the Linux kernel from 2007 to 2017. The VirtIO drivers are used by KVM (Kernel-based Virtual Machine), a feature of the Linux Kernel that provides hypervisor capabilities. KVM was started in 2006 and added to the Linux kernel in release 2.6.20 in February 2007.
VirtIO provides a standardised approach to developing drivers for virtualisation. The first devices for networking (virtio-net) were added to the Linux kernel in 2007 (release 2.6.23), followed by block device emulation in 2008 (virtio-blk in kernel 2.6.26). Further devices were added in the 2.6.27 kernel release, providing a complete system of abstraction.
VirtIO continues to be extended. Para-virtualisation drivers also exist for other hypervisors, such as Xen, which operate in a similar way to VirtIO.
Emulation
Para-virtualisation, as we’ve just discussed, is a form of emulation. We need to cover one other aspect of emulation, and that’s the entire architecture emulation provided by QEMU and similar solutions.
QEMU (which stands for Quick Emulation) is a free, open-source emulator for a wide range of system architectures, including x86, ARM, PowerPC, and RISC-V. The software is capable of providing hypervisor support as a virtual machine manager, emulating entire systems or program emulation (known as user mode). MacOS from Apple running on ARM-based processors (the M1/2/3 series) includes Rosetta 2, an emulator for older x86-compiled programs.
QEMU is interesting as it provides the capability to emulate across architectures, for example, running ARM-based operating systems on an x86 system. QEMU is used by hypervisors, including Xen and KVM, for device emulation support.
Implementations
The aim of this post is to highlight the importance of the hypervisor when picking a platform for virtualisation. As we’ve discussed, there are two main approaches in use today, either hardware-based support or binary translation. Within those two techniques, device emulation supports either full virtualisation or para-virtualisation of system devices.
VMware ESXi – ESXi is built on a custom O/S kernel called the vmkernel, which manages the downstream support of storage, networking and other devices used by the hypervisor. Each virtual machine on an ESXi deployment runs a separate VMM (virtual machine manager) to coordinate the execution of the VM operating system and handle memory management. ESXi is not Linux, although it used to run a Linux shim layer (called vmklinux) that allowed Linux device drivers to be used for peripherals. It was deprecated in ESXi 5.5. ESXi can support full virtualisation with hardware assist and para-virtualisation devices.
KVM – Kernel-based Virtual Machine, runs as part of the Linux kernel to provide hardware-assisted hypervisor functionality. KVM was merged into the Linux kernel with release 2.6.20 in February 2007. Today, we see KVM (in conjunction with QEMU) forming the basis of virtualisation solutions from Nutanix (AHV), Scale Computing, Proxmox VE, Verge.IO and oVirt.
Xen – a free and open-source hypervisor developed under the Xen Project. Xen supports both full and para-virtualisation and has been ported to Intel, PowerPC, Arm and MIPS architectures. Xen is similar in deployment to VMware ESXi in that the hypervisor boots as a bare-metal instance rather than being deployed as part of another operating system (the KVM model). Early cloud platforms like Linode and AWS used the Xen hypervisor but have since moved over to KVM.
The Architect’s View®
This post is an introduction to the topic of type-1 hypervisors, providing information on the evolution of the technology and the technical challenges of abstracting hardware visibility from a running operating system. While VMware has developed a mature and efficient hypervisor platform with ESXi, there are alternatives. Both KVM and Xen are practical solutions that can easily compete with ESXi.
There are differences in implementation and capability between the platforms we’ve discussed, which is a topic for another blog post (or series of blog posts). With server virtualisation representing a mature technology in the IT industry, why should we care about the differences between solutions, especially as many applications are moving to containerisation or the public cloud?
Despite many assumptions across the industry that server virtualisation is now a legacy technology, there will still be a long tail of virtual server farms in use within on-premises data centres. The transition of Broadcom under VMware has created a degree of future uncertainty that has revived a desire to look at alternative technologies.
As with any technology, the nuances and implementation specifics can create much wider ramifications that impact operational processes and cost profiles. We will dig into some of these differences in future posts, in particular with reference to the implementations in use across the industry.
Copyright (c) 2007-2023 – Post #81a2 – Brookend Ltd, first published on https://www.architecting.it/blog, do not reproduce without permission.