AWS Silicon: The Age of Graviton

At Reinvent 2023, Amazon announced Graviton4, the next iteration in the Arm-based processor family. With another step up in performance and efficiency, where is the Graviton series headed, and what’s behind the move to custom silicon?

Background

The Graviton family of processors was first announced at AWS Reinvent in 2018. The Graviton1 featured 16 Cortex-A72 cores, running at 2.3GHz and using a SoC (system-on-chip architecture). Graviton2 was announced at Reinvent in 2019, based on the Arm Neoverse N1 architecture (codenamed Ares), the first Arm design specifically for the data centre infrastructure market (compared to the Cortex use case for client devices).

Neoverse N1 cores are based on the Cortex A76 design, supporting the Armv8.2 instruction set architecture and built on a 7nm process. Graviton2 supports 64-core processors with DDR4-3200 system memory and 64 PCIe 4.0 lanes.

Graviton3 was announced at Reinvent in 2021. This iteration uses 64 Neoverse V1 cores at 2.6Ghz, with DDR5 DRAM and PCIe 5.0, to deliver up to 25% greater performance over Graviton2 and 60% less energy consumption for the same performance as the x86 architecture (AWS numbers).

Graviton4 uses Arm Neoverse V2 cores using the Armv9.0-A instruction set architecture (ISA), with 96 cores per chip, 12 DDR5-5600 DRAM AND PCIe 5.0 I/O controllers. AWS claims a further 40% performance gain for databases, 30% gain for web applications and 45% performance improvement for Java applications over Graviton3.

You can find some more background details in a post we wrote in January 2022.

Evolution

Each step in the Graviton journey represents a significant leap in capabilities while continuing to deliver performance and power cost savings compared to x86-based EC2 instances. At each step change, AWS claims around 25-30% improvement over previous generations, which in total represents about a 2.2x uptick. These figures, of course, should be validated independently and will vary depending on the workload type.

In a purely unscientific test, we compared the performance of a Graviton2 i4g.large EC2 instance to a Graviton3 M7Gd.large instance, both with two vCPUs and running a local NVMe drive. The test created 5000 files with random content and up to 1MB in size, running 20 parallel tasks. The i4g instance completed in 690 seconds, whereas the M7Gd instance finished after 525 seconds.

We then performed a second test that created SHA256 hashes for each file, using a script that automatically parallelises the task across multiple threads. On the i4g instance, the test completed in 15.3 seconds, while the M7Gd instance finished in 10.2 seconds.

Although this test isn’t purely like-for-like, it’s clear that the improvement of Graviton3 over Graviton2 is significant. At the same time, it’s worth noting that the cost difference of Graviton3 compared to Graviton2 in the AWS eu-west-1 data centre is $0.1191/hour compared to $0.1702/hour, a reduction of 30% in cost for an additional 30% or more improvement in performance.

What about x86? We performed the same test, this time using an M6id.large instance (again, two vCPUs) and a local NVMe device ($0.1323/hour). The file creation test took 958.9 seconds, while the SHA256 test took 15.8 seconds. Both tests delivered worse results than the Graviton instances.

Operating System

Now, this test isn’t a direct comparison between platforms for many reasons, not least of which are the other ecosystem components behind the delivery of the two instance types. We also need to consider the operating system and application code, which firstly needs to have a supported version for our architecture of choice and secondly needs to compile efficiently for the architecture. Efficient compilers will be critical for Arm’s success.

We should also note that Graviton3 introduced architectural improvements over Graviton2 (see our post we referenced above), while Graviton4 moves to the Arm9.0-A ISA. The v9 architecture introduces SVE2 (improved scalable vector extensions), which addresses many requirements of AI workloads. There are also memory improvements with Transactional Memory Extension and the introduction of Confidential Computing Architecture. Digging into the details of these features isn’t within the scope of this blog post. However, we can say that both additions are directed at improving the resiliency of public cloud instances.

The Arm architecture is developing for data centre use cases, filling in the gaps compared to x86. Graviton4 represents a serious alternative for AWS EC2 consumers that want to gain a price/performance benefit, subject to application and O/S support being available.

Benefits

Does the Arm architecture only offer benefits for the customer? Clearly not. Just as Apple has moved to custom silicon (and more recently Microsoft, plus others), there are operational and financial benefits to building and implementing bespoke processors.

Feature release control. AWS can break away from the processor release cycle of Intel or AMD and exert more control over the cadence at which products are brought to the market. AWS already uses custom x86 configurations, but custom processor design allows the development of solutions to fit specific use cases, like analytics, database platforms or back-end infrastructure.

Subject to the availability of new core designs, AWS can optimise its fleet of instances based on 15 years of data that shows which instance types are popular and what changes would be welcomed by customers. Imagine making tweaks to add more L2 cache or memory channels mid-way through an architectural release. The process becomes a much more efficient feedback loop.

AWS can also address the direction of infrastructure computing, for example, with more scalable instances to implement serverless or containers. It can also design more “exotic” components, combining CPUs and GPUs with other technologies, such as CXL (as we discuss in this post). Some of this work is already happening, as we learned from the supercomputer announcements at Reinvent.

Supply Chain Management. AWS gets to control the supply chain and remove the dependence on either AMD or Intel. As we’ve seen with GPUs, controlling access to hardware in a timely fashion is hard, unless you’re a big hitter. For AWS, the supply chain is even more important to manage, as customers expect effectively unlimited capacity in the public cloud.

Cost Optimisation. For Amazon (or any at-scale cloud vendor), every percentage point saved on costs represents billions of dollars in reduced capital expenditure. However, savings can also be made in operational overheads by running infrastructure that delivers relatively similar throughput for lower power consumption.

This means data centres can be refitted with denser racks without refurbishing power supplies. Capital costs for new space can be deferred. The list goes on. The question to ask is whether AWS, Azure or other cloud vendors are passing those savings on. With Graviton, it looks like they are.

Differentiation Lock-in. Probably the most interesting outcome of developing custom hardware is the journey towards differentiation that results in long-term lock-in. While the use of Arm is possible on-premises, it’s not widespread. So, any application that runs on an Arm instance in AWS can’t be (easily) ported on-premises. The application and data could be moved (which may require recompilation), but IT organisations won’t be able to lift and shift.

Initially, the difference between on-premises and cloud instances may not be a problem, but as new capabilities are introduced (like the exotics we mentioned) and as the cost profile becomes wider, on-premises infrastructure will be less desirable for large-scale deployments and likely to shrink to only the infrastructure that must be retained privately. This will partly be because on-premises vendors can’t (or won’t) offer similar infrastructure with software to exploit it.

Innovation

AWS is very astute at building a business that customers want to consume. The mindset comes from the retail side of the company, where techniques such as “one-click” ordering and Amazon Prime make purchasing frictionless.

Those skills are being applied to the IT world, with Graviton being an AWS “own brand”. Although we haven’t discussed this in this post, we can see similar approaches with Nitro (hardware-based network and storage virtualisation), Inferentia (AI inferencing) and Trainium (AI model training). Of course, the whole process of “own branding” started with software when AWS forked MongoDB to create DocumentDB and has continued ever since.

Amazon’s approach isn’t necessarily bad, but an alternative approach to gain further and widespread adoption of AWS. From a customer perspective, we recommend the following approach.

Evaluate Graviton and compare to equivalent x86 instances. Do A/B testing to see how performance works out (probably best with test/development environments). Look at the pricing and see how it aligns with similar x86 equivalents.
Review existing instances to evaluate what could migrate to an Arm-based platform. Good examples could be generic infrastructure (self-managed DNS, for example), standardised databases and interpreted code. Build a cost/benefit model for migration to Arm where appropriate, based on normal refresh cycles.
Be aware of disaster recovery and other resiliency measures implemented with virtual instances to ensure there are no x86 dependencies. In addition, validate the locations where Arm-based instances are available (they are not universal).

The first stage of Graviton adoption will be tactical. The next wave will depend on what Arm does next.

Roadmap

In September 2022, Arm presented a roadmap for the future development of the three variations of Neoverse. These are V-Series (high performance), N-Series (general purpose) and E-Series (efficient).

As we highlighted earlier, Graviton2 is Neoverse N1, while Graviton3 is V1 and Graviton4 V2. Neoverse N2 is here, so we can assume AWS is already developing solutions around those cores. Can we expect a “Graviton4L” as a “lite” version of the current version 4 design? Also in the works is an updated E2 core. Past that, future developments of all V, N and E series will be based on PCIe 6.0 and CXL 3.0 technology.

While AWS could keep with the current naming convention, we wonder if there will be a widening of the Graviton nomenclature with new solutions for high performance, balanced and efficient/edge workloads.

The Architect’s View®

Graviton is here to stay and, in typical Amazon fashion, will be used to drive down costs and deliver a unique offering for customers. We recommend that IT organisations review and adopt Graviton instances where it makes sense to do so, based on our outline above.

Once again, though, we are left pondering the future of on-premises vendors like Dell and HPE. Neither company has a mainstream Arm server for the enterprise due to the long-term relationships in place with Intel. Arguably, the issue of adopting Arm is more challenging for Dell, with the additional dependency on Intel for consumer and business laptops and desktops.

We’re at risk of seeing a bifurcation in IT, with Arm gaining an increased foothold in the public cloud. Meanwhile, on-premises vendors remain with relatively inefficient Intel and AMD x86 processors. All that could change, though, as Intel brings in new designs based on efficiency cores. The widespread use of Arm processors across the data centre continues to gain momentum. We asked the question back in January 2020 whether Arm was ready for data centre primetime, and we think the answer is definitely yes.