Understanding Tiered Storage

This week I noticed a conversation taking place on Twitter discussing the subject of tiered storage. The understanding of why tiered storage developed and why it is used today seems to be somewhat misunderstood, so perhaps it’s time for an update on this technology.

Background

Given infinite resources and infinite budget, all data would be stored on the fastest media possible, because we know that storage is the major bottleneck in most IT infrastructure. Unfortunately life isn’t that simple; businesses have budgets and are constrained in environmentals (floor space, power, cooling), plus staffing resources and of course with exponential growth, hardware costs grow with a similar profile.

Tiered storage developed to fix the problems of escalating storage costs, because storage demand grows anywhere from 50-100 per year and even with declining disk prices, a single tier model is unsustainable.

The Long Tail

Now, when we look at data on shared storage, we see the Long Tail effect – a small number of LUNs or volumes consume the majority of I/O. So from a performance perspective, not all data needs to be treated the same way. In fact, in general with large shared storage environments, 80% of the IOPS consumed comes from 20% of the data, although this will vary slightly from environment to environment. The opportunity therefore with tiered storage is to place data on the tier of storage that delivers the performance it requires. To achieve this we need to look at storage I/O density.

Storage I/O Density

I/O Density is the measure of how many IOPS a tier of storage can deliver per unit of capacity. It can be measured, for example in IOPS/GB or IOPS/TB. 15K FC drives therefore provide a higher I/O density than the same capacity 10K drive and flash drives provide significantly higher levels than spinning media. Applications typically have their own I/O density requirements, give or take some burst capacity for busy periods. Designing tiering should therefore be about building tiers that deliver to a specific I/O density profile.

Tiering 1.0

The first implementations of tiering were pretty basic and worked at the granularity of the array. Assuming you could justify multiple purchases, each array would be packed with data that mapped to a certain level of performance, for example, 15K FC drives, 10K FC drives or SATA drives. The performance figures of these tiers could be varied by implementing different RAID types with differing RAID stripe widths (taking into consideration data availability requirements of course). Moving data between tiers in the 1.0 model was cumbersome, requiring host-based migration or array-based replication and in most cases this type of tiering was in place before features such as Storage vMotion.

Tiering 2.0

The second wave of tiering saw multiple tiers of storage deployed in the same array. Each tier presented LUNs/volumes built from that tier of disk. Although this model was more efficient (no need to buy multiple arrays), there was still the risk of wasted resources if the storage was purchased in the wrong ratio, or demand for each tier type wasn’t well understood. This created the dilemma of deciding whether to fully fill an array or to be continually upgrading it with new disk. Data mobility between tiers was slightly better, as tools were available to automate this in the background on some platforms, but all of the data on that volume/LUN would get the same level of performance, which could still be inefficient.

Tiering 3.0

The third wave of tiering saw data being divided up at the sub-LUN/volume level, with arrays determining “hot” or active data and moving it to faster storage, while inactive data moves to lower performing disk. These solutions become more efficient in terms of resource usage but don’t have the flexibility to react to changes in I/O demand as data is moved infrequently to reduce the change of contention or “churn” where more data is moved around than necessary, having an impact on production performance. Of course no wholesale migration of data is necessary, only hot/cold blocks are moved.

Tiering 4.0

The latest wave of technology doesn’t look to place data on specific tiers of storage. Instead, storage architectures have been rewritten to take advantage of high-performance flash devices and use them as read and write cache. This approach is different from previous implementations that placed data permanently on fast storage as data on flash can be more transient. Flash is such a fast I/O technology that in many instances flash performance isn’t fully utilised. These new storage designs allow flash to be fully exploited, creating a more attractive $/GB ratio and allowing more dynamic placement of active and inactive data.

Exceptions to The Rule

Of course there will always be exceptions to these general rules, however they are still forms of tiering. Some applications need low latency I/O and therefore justify dedicated hardware. This may be in the form of a dedicated appliance, or hardware in the server such as NVDIMM or PCIe SSD. Some data such as backup and archive have an I/O profile that creates sequential I/O and can be delivered through high capacity, low (random) performance disk drives. We’re also seeing a move to new application models that distribute the I/O across many servers, allowing relatively inexpensive disks to provide large numbers of aggregated I/Os.

The Architect’s View

The last 10 years have seen an evolution in the placement of data to best exploit hardware resources as efficiently as possible. However the high-end, all encompassing storage array has had its day as new application deployment models emerge. Tiering will still exist and the basic principles of I/O density, latency and performance will all still apply. Ultimately it’s all about optimising cost, regardless of the application architecture being used.

Comments are always welcome; please read our Comments Policy. If you have any related links of interest, please feel free to add them as a comment for consideration.