Storage Predictions for 2024 and Beyond (Part V - Open Source)

Open source has been a remarkably successful business model for the creation and adoption of software across the IT industry. This has included a wide range of storage solutions, generally classed as software-defined storage. As the public cloud continues to be prominent, is there a threat from vendors taking free software and repurposing it for commercial gain, or can we expect to see open source as part of the SDS landscape for years to come?

Background

Free software has existed for decades, pretty much as soon as the first computers were commercially available from the 1950s onwards. In many cases, creating software was simply a sideline or hobby, whereas the Freemium and Shareware approaches have been used as part of a business model for attracting customers. We’ve also seen software come out of both business need and research projects, for example Ceph, and many file system formats.

Drivers

Why do software developers choose an open-source model, when the creation of software requires so much investment of time, skill, and effort? We can assume at least some of the following drivers influence the decision-making process.

Time to Market – Open-source projects can tap into an eager market of coders, with potentially hundreds or thousands prepared to work on popular projects. This enables a much quicker time to market or even to develop that first MVP at a much lower cost compared to a commercial venture. This approach mitigates significant financial risk.
Collaboration – Extending on the previous point, with many developers involved in creating software, there are more ideas, concepts, and techniques available to create software. Many of the tools available today aid that diversity of thought and enable efficient bug tracking and resolution.
Acceptance – Everyone loves the concept of free, even if a product isn’t fully formed. Early Shareware was a excellent example of this, while freely licensed software can spawn an entire series of related products and solutions (although we’re now seeing licensing issues, which we will discuss in a moment). A “product-led growth” approach can help both gain adoption and plot product direction.
Kudos and Recognition – for many developers, open source is a great way to demonstrate skills and much quicker and more efficient than trying to get a company to review a CV or resume. Project maintainers and creators of popular platforms can gain “rockstar” status within the industry.

It’s worth highlighting that there is a downside to the low barriers to entry in developing open-source software. That is the desire to constantly “re-invent the wheel” when, in many cases, the solutions in the market are already good enough (Torus was a good example of this). Some of this approach is a case of inflated egos, assuming that they can do better than the previous generation of developers. However, we should also accept that as hardware advances, software solutions need to be revisited.

Open-Source Storage

As we mentioned already, open source has been around for decades. For example, I have experience from the 1980s with the SHARE user group and mainframe software, which has been published since the 1950s. The early years of Linux saw a huge rise in Shareware and free software, and the emergence of the open-source movement, which has now evolved into four primary areas.

Unstructured Data Stores – solutions such as Alluxio, Ceph, MinIO, HDFS and Lustre. This segment of the market has been successful due to initial development in research and academic environments, while solutions such as MinIO are targeted at developers.
NAS filers – distinct from the scale-out nature of unstructured data stores, NAS filers were one of the earliest software-defined storage solutions available. This category now includes TrueNAS, OpenFiler, and OpenMediaVault, with most offering capabilities equivalent to commercial offerings.
Databases – MySQL, MongoDB, RocksDB, Speedb (recently acquired by Redis) and MariaDB are all examples in this category. Most solutions have developed as alternatives to commercial offerings or to address new techniques and requirements, such as key/value stores and NoSQL databases.
File Systems – there are many free and open-source file systems that include FUSE (part of Linux), CubeFS, GlusterFS, LizardFS, MooseFS, BTRFS, RozoFS and ZFS (which is included in many O/S distributions and spans the commercial/free divide). File system diversity addresses the need of I/O profiles and use cases. However, many of these solutions are still niche platforms.

The least well served storage category for open-source software appears to be block-based storage, of which we only see DRBD as a solution. We should also remember that development kits such as SPDK are also open source, providing a raft of additional capabilities that can be integrated into both commercial and other open-source offerings.

Cloudy Effects

Probably the greatest impact on the open-source software model has come from the widespread adoption of public cloud solutions. MongoDB and Elastic were early companies changing licensing terms to prevent public cloud vendors profiting from open source without giving anything back. More recently, we’ve seen licensing changes from Redis and HashiCorp (although this isn’t storage related).

Forking software and building competing cloud-based platforms doesn’t incentivise the development of open-source software, so we can expect to see a rise in restrictive licensing (more on this in a moment).

Predictions

What should we expect and look out for in Open-source storage for the remainder of the decade?

Increased adoption for structured data. Linux and the public cloud have driven the ease of adoption for open-source structured database solutions. Solutions like Redis, MongoDB, MySQL and MariaDB are widely used in production environments. We believe there will be a long-term trend towards using open-source solutions and moving away from the tightly controlled licensing of commercial products. In part, this change will be driven by new deployment models, such as databases in containers and database-as-a-service vendors such as Fauna, ScyllaDB, Tessell, MongoDB (Atlas) and Portworx. We can see structured databases becoming a fourth storage protocol.
Object Storage becomes primarily open source. We previously suggested that adoption might reach 100%, with solutions such as MinIO on the market. If companies such as Qumulo (with its cold storage solution) can make their offerings commercially successful, then perhaps 100% is a stretch goal too far. However, we expect large volumes of data to move towards much cheaper solutions as businesses grapple with retaining data for future use and the costs associated with that storage.
Consolidation/Rationalisation. Many of the file system solutions, for example, could drop away, as features are consolidated or introduced into competing products. The same may happen for structured databases where forked products don’t gain traction.
Greater Forking/Fragmentation. In direct contrast to the last prediction, there is also the risk of greater fragmentation in the industry as licence terms change and product forking occurs. The recent Redis changes have already created new forks, including KeyDB, Redict and PlaceholderKV. We expect to see these projects birth and die with increased frequency.
Increasing focus on data in place of storage. Although storage has been the bedrock for storing data, increasingly the features of physical storage are being abstracted towards data-focused metrics. We see this trend increasing, as open-source solutions focus on storing data and obfuscate the physical aspects of storage.
Restrictive Licensing. Open source may well be affected by increased use of restrictive licensing, which looks set to become more of a trend in the industry. License structures and terms will continue to evolve.

The Architect’s View®

Probably the most obvious example of the evolution of open-source storage is to look at the CNCF landscape map. This shows 78 solutions for databases and 72 for storage. Not all of these vendors and products are open source, and arguably, some no longer exist, but the diversity of solutions shows the breadth and depth of modern open source in the storage and data industries.

It’s clear that even with the success of the public cloud, open-source storage will continue to grow and thrive. The ability to create free software with little or no barriers to entry will provide the industry with new ideas and directly challenge the incumbent commercial platforms. We hope to see this trend continue.