Over the last couple of weeks I’ve been doing some perusing of vendor products and benchmarks for both storage arrays and PCIe SSD cards. For the PCIe data, you can see the fruits of this labour at PCIe SSD Vendors and Products. This morning I was looking at the figures for the latest NetApp EF5x0 upgrade, the EF550, their interim all-flash box and I was reminded why I have such an irritation with vendor-quoted performance figures.
My issue is in the unit of measurement and the explanation of what that unit covers. Take for example the EF550 data sheet. This shows “burst” I/O of up to 900,000 IOPS – what exactly does that mean? How long is the burst? What is the block size? How many parallel I/Os were needed achieve it? Essentially this is a meaningless figure based on the marketing department’s need to get as close as possible to the magical one meeelion IOPS, that all marketing teams seems to obsess about. NetApp don’t quote actually latency times with the product, simply quoting “sub-millisecond” as the device capability. This could mean anything but is clearly hiding their poorer capability compared to their peers. Then there’s the throughput – 12GB/s. What configuration was needed to achieve this? Could a customer actually see this level of performance?
NetApp are not alone in this deception. At an EMC presentation recently on their XtremIO product, they chose to show competitor performance comparisons without actually naming the competitor. This isn’t much better than just making figures up. Without some kind of provenance to the source of the data, how can customers believe the testing process was independent? The same applies to the PCIe vendors I have listed; there’s a mix of varying block sizes, presumably designed to show their product in the best light; there are read and write figures, but they don’t always indicate whether this is sequential or random – totally random I/O workload. Then there’s the 100% random read I/O – exactly what use is that statistic other than showing how good your hardware backplane is?
The Architect’s View
I know there are official benchmarks out there, such as SPECsfs2008 (Standard Performance Evaluation Corporation) and the Storage Performance Council, but not all vendors choose to submit to them. Just having consistency on vendor-quoted benchmarks would be a start. A little honesty and transparency goes a long way to gaining customer confidence.
Comments are always welcome; please indicate if you work for a vendor as it’s only fair. If you have any related links of interest, please feel free to add them as a comment for consideration.
Subscribe to the newsletter! – simply follow this link and enter your basic details (email addresses not shared with any other site).
Copyright (c) 2013 – Brookend Ltd, first published on http://architecting.it, do not reproduce without permission.