New Microsoft silicon unveiled by Satya Nadella at Ignite 2023

At Microsoft Ignite 2023, CEO Satya Nadella announced a range of new hardware-based infrastructure additions to Azure, which improve the performance and efficiency of the platform. We take an early look at what was announced and offer a view on the impact of these new features.

Background

In the keynote at Ignite, Satya Nadella announced several hardware solutions. We’re looking at the following.

Azure Boost – hardware-based acceleration of storage and networking for virtual machines.
Cobalt – a new Arm-based processor powering Azure infrastructure and available for customers in 2024.
Maia – an “AI accelerator” chip, custom-designed and built by Microsoft for large language model (LLM) training and inferencing.

So far, the details on each of these solutions are scant, so we’ve pieced together our deductions from what’s been published so far.

Azure Boost

Azure Boost appears to be an offload card and associated technology similar to the Nitro system introduced by AWS in 2017. The solution was announced in preview earlier this year (announcement and blog post) but appears to be now in general availability across a broader range of virtual instances following the Ignite announcement.

At the start of 2023, Microsoft acquired Fungible, a company we followed, including a podcast recording with the then-CEO Pradeep Sindhu and Chief Scientist Jai Menon. The Fungible DPU technology was very specifically storage-focused, based on two DPUs. The first was the F1, a back-end storage target, while the S1 was the host-based initiator. However, Azure Boost addresses both networking and storage. Therefore, we think it’s more likely that the new technology was derived or developed from the Fungible solution.

You can read more about SmartNICs, or Intelligent Data Devices as we call them, in our report, linked below.

Intelligent Data Devices 2023 Edition – A Pathfinder Report

This Architecting IT report looks at the developing market of SmartNICs, DPUs and computational storage devices, as data centres disaggregate data management processes, security and networking. Premium download – $295.00 (BRKWP0303-2023)

Download Now

Microsoft claims the Azure Boost technology will deliver up to 12.5GB/s throughput and 650,000 IOPS for storage, while networking will see 17.3GB/s throughput and 3.8 million IOPS using 200Gb technology. We tried to test this claim out using what is supposedly an Azure Boost-enabled virtual instance. However, we can’t find a way to enable the acceleration, even though it should be available on the E2bs_v5 instance we chose for the testing.

Cobalt

It appears that Microsoft has been developing its own Arm-based processor, known as Cobalt. The first device currently in use within the Azure infrastructure is the Cobalt 100, running 128 64-bit Armv9 architecture cores. Cobalt isn’t the first Arm-based processor to be available in Azure. In April 2022, Microsoft announced Ampere Altra virtual instances, based on Arm Neoverse cores, reaching general availability in August 2022.

In the Ignite keynote, Satya Nadella indicated that Cobalt is already in use for Azure infrastructure, with customer-based instances available in 2024. He also claimed that Cobalt is the most powerful Arm processor available for the public cloud – a claim we’re happy to verify when they become generally available.

We’ve been discussing Arm in the data centre for nearly four years, as the posts linked below indicate. From the very beginning, we saw Arm as an opportunity in the public cloud, with this position strengthened by the announcements from NVIDIA that integrate Arm and GPUs.

We believe that Arm will gradually gain significant traction compared to x86 processors for those workloads that are highly parallelised (such as containers). The driving factors behind the adoption of Arm will be cost and efficiency. Hyper-scale cloud providers run millions of instances across dozens of data centres, so even a 5-10% cost saving represents a massive benefit.

Maia

Maia is a new AI accelerator developed by Microsoft specifically for large language models (LLMs) like those supporting ChatGPT. Maia will be used for training and inferencing and is clearly a move to control the cost and supply chain for what is expected to be a huge market for Microsoft.

After discussing hardware, Satya Nadella moved on to software platform features, and specifically Copilot, which is being integrated into everything Microsoft does. Copilot will help with emails, writing content, writing code and with Copilot Studio, enabling anyone in business to bring together all the data sets running on the Azure platform. Copilot will generate billions of dollars of revenue for Microsoft, so securing control over the underlying hardware is essential. The Maia platform is just this – a route to certainty to enable the Copilot demand to scale. Maia comes with a rack-scale design, including cooling, creating bespoke technology for Microsoft to deliver AI.

The Architect’s View®

Microsoft is following the lead of AWS and Google by developing more in-house silicon. Where Ampere was the testing ground, for example, Cobalt will be the long-term strategy that gives Microsoft independence from the chip vendors.

Similarly, Maia represents risk mitigation against the cost of using third-party vendor GPUs. That’s not to say Microsoft won’t offer NVIDIA or AMD GPUs (Jensen Huang appeared onstage in the keynote, while AMD MI300X was announced as part of a new series of virtual instances). Instead, Maia and the technology behind Azure Boost are focused on scaling Azure as efficiently as possible.

Only the hyper-scalers seem capable and willing to make these commitments. This continues to place increased risk on the future prospects of on-premises hardware vendors that can’t follow a similar path. Instead, Dell and HPE will remain dependent on the supply chain from AMD, Intel and NVIDIA and be locked out of more efficient hardware solutions. Is this giving customers just one more reason to move to the public cloud?