Rethinking Storage for Microservices

The New Stack had a very interesting article this week that talked about rethinking the way storage is mapped to microservices like containers. The discussion took me back 12+ months to my own attempts to raise money for a startup that addressed the issues raised in the article. I’m still not sure that the issues are well understood and they are certainly not resolved.

Block is Not the Solution

Almost a year ago, I wrote that block storage wasn’t the solution for container storage. The fundamental construct of block storage means there’s no inherent security (logical or physical), no metadata to track LUN/volume usage and formatting issues using data cross-platform (because of specific file system differences, e.g. NTFS and EXT4). One of the biggest problems for block though, is in the dynamic nature of provisioning, as Torsten Volk says in the New Stack article:

“Using drivers for persistent block storage that is dynamically mapped to containerized microservices has proven difficult, as block storage was never designed to be rapidly mapped to and detached from their targets”

Exactly right. Traditional volumes aren’t a good fit. They don’t scale and create management problems.

File Services

Distributed file systems are a better solution for a number of reasons. File systems have metadata, security and data integrity built in. File locking prevents two containers updating data at the same time. Filesystem ACLs can be used to control access at the container level (more on that in a moment). Metadata can be used with file system naming to ensure that the application is associated with a particular file system or directory – a process much harder with block.

The other great benefit to implementing a distributed file system for microservices is that it can be distributed across underlying physical storage that can be abstracted away from the content. This means a file system could span on-premises and public clouds. With some clever thinking around abstracting the logical file view from physical blocks, we even developed a way to have data spanning multiple clouds and significantly reduce the egress charges between clouds. (When we discussed this with Google, they were very interested).

Security

How could container security be managed? The easiest way I found was to dynamically change directory permissions for a locally mounted file system on the container host. Each container runs with a unique UID/GID, which is also dynamically assigned to the container. Making this work means having some software to present what looks like a local file system, rather than a remote NFS mount. However, it does work – I actually have running code that dynamically changes file/directory permissions on a per-container basis. This same code allowed me to spin up a 1TB database running in London to a data centre in Sydney within a few seconds.

Hammers & Nuts

That takes us to the point of discussing solutions. Existing distributed file systems represent solutions that are too complex for container needs. Typically, a container exists in one location or another and may transition between locations. Sometimes data needs to be copied for test/dev purposes, so that requirement has to be considered too. But ask yourself the following about your container data?

Do I really need global locking for every file system or in every location I may run containers? (Typically the answer is no, and with block devices, the answer is definitely no, can’t do it).
Do I need my data replicated to every location all the time, just in case I may move a container workload there? (Again, typically the answer is no, data needs to be logically accessible, but physically data can be moved behind the scenes).
Do I need a consistent data view? (usually yes)
Do I need a security and data model that is application, rather than hardware focused? (Typically yes).

Think of how traditional distributed file systems work and you can see that the first two points are usually over-complicated by existing distributed solutions. They are, by design, made to replicate and keep data consistent across all locations. But that’s not the requirement for containers. The distribution may be limited and in general, with 99% of applications, the data will only be accessed in one place at any one time. Distributed locking can, therefore, be much simplified.

The last two points highlight the need to abstract the physical platform view of data to a more logical application-focused view. Microsoft DFS had this – especially when mapped to an Active Directory structure. The logical/business view doesn’t have to be aligned to the hardware view.

The Architect’s View

So, the solution I’d developed allowed data to be visible and accessible anywhere and moved around dynamically, with minor caveats on performance and data integrity. To me, this was and still is the solution for persistent data and storage. This is the way a true data fabric will develop, with more of a focus on the data/application needs than developing a system for hardware resilience. Hopefully one day someone may deliver it. Block storage and containers will continue to be a sticking plaster solution. In the meantime, I will plug away with my code as an interesting side project!

Comments are always welcome; please read our Comments Policy. If you have any related links of interest, please feel free to add them as a comment for consideration.