I’m always intrigued to read the details of new benchmark announcements, especially those that align with vendor press releases claiming the magical top spot in a global list of their peers. WekaIO recently claimed the number one position in IO500, a benchmark for high-performance file systems. Does this result make WekaIO Matrix the fastest file system in the world?
At the start of this year, I posted about being careful when taking benchmark metrics at face value. Every vendor wants to make their results look the best. After all, these tests are expensive, and the spending needs to be justified.
Certain vendors even cry foul at the way in which some performance numbers are obtained. The problem here is that every platform will be designed uniquely and use technology in different ways. WekaIO Matrix and the second-place Intel DAOS were both developed to use NVMe and storage-class memory rather than hard drives. So naturally, these vendors will have an advantage over systems designed to also accommodate hard drives.
But perhaps we should look further into the results to see what the data shows.
The current IO500 results show WekaIO in first place, closely followed by Intel DAOS. WekaIO used 345 nodes and 8625 processes compared to Intel’s 26 nodes and 728 processes. Third place NSC used 480 nodes and 5280 processes. So, does this mean WekaIO simply used more horsepower to achieve a higher score?
- Can the WekaIO Matrix file system be faster than DAS?
- Performance Benchmarks: Reading Between the Lines
Hold on though; WekaIO ran this test on AWS, so didn’t have direct control over the hardware, which could have had a very different specification compared to each node in the Intel DAOS benchmark. In fact, I can’t find any details on the setup of any of the benchmark tests. With that in mind, how are we supposed to make any comparisons?
Of course, the IO500 isn’t like a Storage Performance Council validation for the enterprise. This is high-performance computing and the requirements are different. SPC test results look to show value for money in terms of building the best solution for the budget available and will produce a $/IOPS metric. Most enterprises want generic storage for a mix of workloads and cost efficiency.
IO500 is showcasing a different scenario. Here, the aim is to be as fast as possible and arguably, irrespective of the configuration in place. If a file system isn’t designed to be scalable, then no amount of hardware thrown at a configuration will get the software into first place. This is perhaps what the benchmark shows for WekaIO – that Matrix is scalable, whether running five or five hundred nodes.
Perhaps a more reasonable comparison is the 10-node challenge, which by name restricts tests to a configuration of ten nodes. In this test, DAOS comes out first, but WekaIO Matrix has been in the top position in past ratings. Again, there are no details on the individual configurations, so we’re left to make our own conclusions on the results here too.
The Architect’s View
Vendors will always use hero numbers for marketing purposes. If you’re not even playing in the benchmark game, then you don’t get considered as a player. The top group of vendors will always be vying for first place, as they tweak software, improve the scalability of their products and build bigger systems. Perhaps this is what we should focus on – the desire to improve that these benchmarks undeniably deliver.
Irrespective of this, we also need to remember what any benchmark demonstrates and how it applies to our own requirements. This data is great, but only one aspect of the buying process. You can learn more on WekaIO and Matrix in the following podcast recorded with WekaIO CEO Liran Zvibel.
Copyright (c) 2007-2019 Brookend Ltd, no reproduction without permission, in part or whole. Post #5F53. Disclaimer: WekaIO has been a customer of Brookend Ltd.