Commentary: Businesses need to establish Cyber SLAs and SLOs

This week I attended Commvault Shift in London, during which we were invited to take part in a “Minutes to Meltdown” ransomware simulation. One aspect that resonated with me from the session was the need for cyber-RPO and cyber-RTO service levels, which don’t seem to exist in the industry. Shouldn’t we be treating cyber-recovery with similar SLAs and SLOs as a typical disaster event?

Background

Minutes to Meltdown is a workshop event offered to Commvault customers (and prospects), which takes the participants through a multi-stage simulated ransomware attack. A large organisation gets hacked, and it’s up to the CEO, CISO, CTO and legal counsel (role-played by participants) to agree a strategy and action plan at each phase of the attack.

The simulation is interspersed with video and commentary from the workshop facilitator, as well as a cyber-security expert from Commvault who explains the impacts of the teams’ decisions after each deliberation. Ultimately, it is down to each team to decide whether a ransomware should or should not be paid.

The two-hour session is engaging and thought-provoking. It will definitely make any IT professional sit up and think hard about the impact of a cyber-attack on their environment. In addition, within the teams, the beliefs and opinions of colleagues may surprise people. This is all good for creating an engaging discussion. The Commvault team that designed and now runs the simulation should be applauded for their hard work and expertise in delivering these sessions.

Service Levels

During the session this week (and previously in a video I recorded with Thomas Bryant from Commvault, embedded here), I reflected on the apparent lack of service level objectives (and possibly agreements) in place when recovering from ransomware attacks. Most of the news stories that discuss breaches and downed systems will generally highlight recovery times in days or weeks (sometimes even months). Additionally, some businesses never recover and disappear as a result of a cyber-attack.

Yet, in the case of data protection, business continuity, and disaster recovery, businesses are used to the concepts of RTO and RPO. Just as a reminder, these terms are generally defined as follows:

Recovery Time Objective (RTO) – the target time expected to recover an application or system back to operation, typically measured in minutes or hours (hopefully not days). RTO values will vary and depend on the requirements of each business application and its criticality to the ongoing operations of a company.
Recovery Point Objective (RPO) – the volume of acceptable data loss experienced after a recovery process. A zero value, for example, represents no data loss at all, while many businesses accept an RPO of 15 minutes or perhaps an hour. RPO may be much higher for systems where the data can be easily recreated from another source.

Both RTO and RPO collectively form service-level objectives (SLOs), which may be contractually encompassed in a service-level agreement (SLA). As an example, cloud service providers such as AWS offer service level agreements for many of their services (see this link for examples), with penalties for digressions. Unfortunately, cloud service providers typically only offer service credits rather than hard cash, so businesses need to build resiliency into application design to mitigate against cloud service outages.

In the last few years, data protection vendors have started to introduce ransomware guarantees (a form of SLA), which provide financial compensation if the company fails to adhere to SLAs for data protection and recovery. It is debateable whether these guarantees have any merit (there is a lot of small print), as we discussed in this article a few years ago. However, where IT within a business has chosen to use a backup service rather than deploy backup software directly, then some level of service agreement is necessary.

Recovery

Within businesses, recovery generally adheres to SLOs. There isn’t merit in having financial penalties for internal business units unless the corporate structure places the IT function into a separate legal entity. Generally, IT and lines of business will agree on RTO and RPO metrics per application based on the relative importance of the application plus the cost of delivering that recovery capability. Mission-critical systems, for example, might implement synchronous replication or clustering solutions, while other applications are simply recovered from archived snapshots or backups.

One other aspect to mention is recovery order. Some systems will be critical to application recovery, such as Active Directory services, DNS, email and file services. There will be a recovery priority order for those systems in the event of a disaster. Equally, in a ransomware attack, those systems will be recovered first and will need higher levels of protection and resilience.

Siblings

So, many businesses will be comfortable with disaster recovery planning and execution. Is cyber-recovery that different? The most obvious difference between recovering from a disaster and a cyber-attack is control. As we saw from the Minutes to Meltdown simulation, the hacker appears to be in control while the business is lurching from one crisis to another. The fear of the unknown (how much is compromised, how much data has been exfiltrated, how long will recovery take) can drive an apparent panic scenario if cyber-recovery hasn’t been planned in advance.

In a “typical” disaster situation, the recovery process takes place from a known good backup. Recovery has probably been tested by the business (hopefully for real, rather than just as a paper exercise) in some limited form, giving confidence that recovery is possible.

Contrast this to a cyber-attack. Hackers may have been within a company network for weeks or months (the dwell time), meaning backups could be compromised with malicious code that has sat dormant in production systems since the first breach. Any recovery of encrypted data risks simply bringing back the malicious code, for the hacker to strike again.

It is perhaps this factor of the unknown that makes a ransomware attack feared much more than a traditional disaster scenario.

CRTO and CRPO

Outside of the uncertainty around exactly which backup should be restored, are the processes of cyber-recovery and disaster recovery any different? Potentially, there may be a difference in scope. Excluding the type of disaster introduced by the CrowdStrike update in July 2024, most IT disasters are likely to be physical in nature – the loss of a server, rack or data centre. In contrast, the scope of a ransomware attack could include data centre servers, the public cloud, edge devices and mobile (laptops/desktops) infrastructure.

But in reality, these disparate computer systems just need to have specific service level objectives assigned to them. For example, the Cyber Recovery Time Objective (CRTO) for mission-critical production systems in the data centre should be aligned to that for a DR outage. For non-critical infrastructure, that recovery time could be much more elongated, but still prioritised by critical workers.

Similarly, Cyber Recovery Point Objective (CRPO) should align with traditional RPO but may be extended for non-critical devices and applications, where appropriate. Why establish these differences? Some infrastructure, for example, laptops or mobiles, may be quicker to reinstall than recover, especially if data is kept centrally (on file servers). This also applies to edge devices that may be quicker to re-image and configure than recover.

Process

What we’re advocating here is a mix of recovery techniques, depending on the issue being experienced. Technologies such as Commvault’s Cleanroom Recovery, for example, can provide the capability for businesses to validate backups and ensure the most recent recovery copy is clear of malware. Threat-hunting technology can assist in reducing the dwell time of a breach by more quickly identifying malicious users and anomalous behaviour. Some storage appliance vendors, such as Infinidat, are building cyber-resiliency into their core products, enabling very fast recovery. Appranix, recently acquired by Commvault, provides disaster recovery for the cloud, which could also be used for ransomware recovery.

Gap

There is perhaps one gap that we should discuss. That is the lost data if systems are recovered from backups. With an RPO/CRPO of zero, there is no data loss. All other values risk there being some degree of missing data. In high-activity systems, a minute could represent thousands of transactions, whereas a 15-minute RPO/CRPO might be acceptable on a rarely used platform.

I suspect many businesses struggle to hit the “big red recovery button” because they have no understanding of the impact of data loss. In a DR scenario, the recovery must be done. In a ransomware scenario, there’s a small possibility that decryption keys provided by the hacker may unlock data without needing to recover. So, some businesses defer recovery in preference of paying a ransom and receiving the keys. Unfortunately, as highlighted in the Commvault workshop, those keys rarely work or are an administrative nightmare to validate (there may be a unique decryption key per device).

This gap is an opportunity for data protection vendors to offer solutions to quarantine compromised data, allowing it to be processed forensically, including attempting decryption with known keys used by ransomware software. Having at least some capability to recover data and reduce RPO closer to zero will be welcome, especially across large estates with thousands of impacted devices.

The Architect’s View®

With the plethora of data protection solutions available today, there is no reason not to apply CRTO and CRPO definitions to all business applications. The recovery process from a cyber-attack is no different from a traditional disaster – if steps have been taken to ensure that backups are available and clean from infection.

IT organisations should be working to make cyber-recovery a BAU (business-as-usual) task, not an exceptional event. With the right processes in place that drive an efficient recovery, the chances of being hacked are reduced, hopefully making well-organised businesses less attractive to hackers who will simply move on to another, easier target.