Storage Predictions for 2023 and Beyond (Part V – Open Source)

This is the fifth post in a series looking at predictions for the storage industry in 2023. Previous posts:

Open Source has been a significant driver of the modern technology world, including software-defined storage. Will the open-source model be an ongoing threat to commercial software or just another way of consuming storage software solutions? What will we see happen over the remainder of this decade?

Background

The free software movement that has developed into Open Source has existed for decades, probably since the first commercial computers were deployed into businesses. Much of that effort was a labour of love, part of a business model (such as freemium) or part of research work. Modern open-source software is generally made available through one of a range of licensing models that infer rights to the user and protects the author too. These models have evolved in recent years to counter the widespread use of Open Source in the public cloud.

Rationale

Why do developers like the Open Source model? When software takes considerable time, effort (and talent) to develop, why give it away? We know there is significant profit to be made from following a commercial model, so why is Open Source so popular?

Collaboration – software developers can share thoughts, ideas, coding techniques, concepts, and other facets across a broad audience of potential contributors. This process aids in diversity of thinking, as well as assisting in bug tracking and resolution.
Time to Market – with potentially hundreds or thousands of eager coders, projects can develop quicker than a commercial company could achieve in a similar timescale (subject to rigorous acceptance criteria on code development and promotion).
Acceptance – free software has a way of being accepted much quicker than commercial products. We know from the early Shareware phase of free software that free is always popular, even if a product isn’t that good. (I recommend reading a book such as “Predictably Irrational” which discusses the concept of free or zero cost). Early adopters generally promote Open Source.
Kudos or Recognition – contributing to Open Source is a great way to build industry recognition and reduce barriers to entry for those looking to get into IT. Developing open-source software is like creating an online CV.

We could also argue a degree of stubbornness from those new to the industry, wanting to show that they can build things quicker and better than the generation before. Certainly, the IT industry is an area that has few barriers to entry and can enable anyone to develop software.

The business model for Open Source offers greater adoption, as customers can be enticed with free offerings that have commercial support or additional features in commercial versions. This is the modern approach to sales often referred to as “product-led growth”.

Open-Source Storage

Open-source storage solutions have been around for decades. Free mainframe software from the SHARE user group has been distributed since the 1950s. I personally used SHARE (from the printed copy of the newsletter) in the 1980s and 1990s, with some good and bad results!

Arguably, modern Open Source and free storage software have developed from the connectivity of the Internet and freely available Linux and Unix operating systems. Over the last ten years, we’ve seen three or four primary areas of development:

File systems – FUSE, CubeFS, GlusterFS, LizardFS, MooseFS, BTRFS, RozoFS and many more, including ZFS, which is now included in many operating system distributions (although it spans the commercial/non-commercial divide after Oracle acquired Sun Microsystems).
Unstructured data stores – this includes Alluxio, Ceph, MinIO, HDFS (part of Hadoop), Lustre and others. This area has been successful because object storage suits the pure software model.
NAS filers – this category includes TrueNAS, OpenFiler, and OpenMediaVault and generally covers SMB or home user use cases. Some of these products (TrueNAS, for example) are high-quality commercial solutions with free offerings.
Databases – MySQL, RocksDB, Speedb, MariaDB, and many other solutions. These platforms have been developed either as alternatives to commercial enterprise products or to create new categories (such as NoSQL).

Probably the only storage category without widespread development is block-based storage systems. DRBD seems to be the only product that is available to build block storage, although some of the other unstructured solutions do offer iSCSI volumes. There are also container-attached storage solutions which we discussed in the previous post.

The Cloud Factor

One interesting aspect of the development of Open Source has seen more restrictive licensing being put in place for solutions from MongoDB and Elastic. The driver for this change has come from the use of these platforms in the public cloud. Open Source is effectively a free treasure trove of resources on which to build a public cloud infrastructure. There is no requirement for cloud providers to pay back by helping in the development of these solutions.

If AWS and other cloud service providers choose to fork popular open-source solutions, then the market will become fragmented with diverging products. Developers will be disincentivised from creating new solutions if there is little expectation of future rewards for their efforts. This conundrum raises challenges for the future of Open Source. As a model, Open Source provides ready access to developers and a consumer market, albeit at the cost of exposing that intellectual property for others to use. One area where a recent infraction occurred was the use of MinIO software by Nutanix. These problems aren’t purely public cloud related.

Predictions

What should we look out for in the open-source storage world during the rest of this decade?

Increased adoption for structured data. Linux and the public cloud have driven the use of open-source solutions and made them easily consumable. Many NoSQL platforms have developed exclusively from the open-source model. We can expect to see a long-term decline occurring in the use of commercial database software in favour of their equivalent open-source solutions. This change will occur through the gradual use of containers and the public cloud, where end users choose to either run solutions natively or consume supported versions of common platforms.
Consolidation & Rationalisation. There are arguably too many open-source file systems available on the market today. This part of the market needs to rationalise file system solutions that offer real value to end-users. Building file systems is complex, and these offerings are least likely to be adopted across commercial and enterprise customers. We will look at file systems in a future post.
Object storage moves 100% Open Source. Object storage may become entirely delivered through the open-source model. The rationale for this can be explained by the cost/value benefit of paying for storage software to retain large volumes of static data or content with no current perceived (but future) value. Existing commercial object storage solutions can then choose to bifurcate and offer commercial versions for high-performance object and file stores. Vendors such as MinIO are hoping this is precisely where things head.
Analytics becomes the next battleground. Open-source solutions move “up the stack” and offer solutions for analytics and AI. This area is currently lucrative for commercial solutions and so could generate interest for open-source development. There are some natural barriers to making this approach work; however, as an example, Ceph has introduced a telemetry option. With a focus on sustainability and efficiency, optimisation will become critical.
Greater Forking and Fragmentation. Although we’re suggesting rationalisation is needed in some areas, some solutions are likely to see increased diversity (as we discuss in a moment). Partly this will be driven by the success of initiatives like the CNCF.
Increased focus on data, not storage. As the public cloud gains more followers, the underlying infrastructure has less exposure and is abstracted further from the user. At this point, the value for businesses is the data, not the physical infrastructure (or to a degree) the storage software supporting it.

The Architect’s View®

Despite the challenges of the public cloud, we expect to see open-source storage continue to thrive. Open Source is an excellent source of new ideas and directly or indirectly drives other parts of the industry. Many graduate research projects have turned into open-source storage products, providing a wealth of new opportunities. Hopefully this trend will continue and we will see more innovation.