Understanding RTO (Recovery time objectives)

By: Danny Poull, V.P. Network Operations Center

Running business continuity and disaster recovery scenarios and exercises are nothing new to the CIO. It can be easy to get overwhelmed with these exercises, and one should break it down and take it one step at a time. Following this rough guideline below, you will be able to run the exercise for your own company, and also get you the data to support your conversation with management.  At the end of the day, you need to be to realistic with management, so they understand the risk and recovery before an emergency and not after.

The first place you want to start is with RTO (Recovery Time Objective). The best way to figure this out, is to break down IT by application or system. For example, one organization might have Office365 for email, a Fileshare with Quickbooks, a CRM for sales, and a SQL application for production/operations. You need to look at the items individually.

Application	RTO
Office365/Email	1 hour
Voice /Phones/ Call Center	1 Day
File shares	5 Days
QuickBooks	5 Days
CRM	15 Business Days
SQL production	30 minutes

Other items to think about to add to the above list:

• Internet
• All Computers
• All Phones
• Network Infrastructure in general
• Server Infrastructure in general

For this made up company, I simulated a conversation with management. We determined that email is our primary form of communication and need email to be running within 1 hour or our business is crippled because of communication loss (specifically email). 

Our phone system is also an important line of communication, but not as important as email. We determined we can always forward phone lines to a cell phone, but only as a temporary solution knowing that going on with this longer than 1 day would cripple communication with customers and tarnish our brand image.

File Shares have forms and templates that we need for day to day business, but we could operate for a week without it. Eventually new work coming in would require forms and would start to hinder business operations.

Quickbooks is how we send invoicing. This simulated conversation started off as management stating “we need to be billing every day or we are not making money. We cannot be down longer than 8 hours”. After we took a step back, we realized it is important and will impact business, but we wanted to determine what is the maximum tolerable downtime. Realistically we could go 5 days without invoicing before business is impacted in a non-recoverable way.

We determined that without the CRM portal, the sales team could still reach out to customers, send out quotes and receive orders, but we couldn’t track commissions, funnels or forecasts. We could not update notes, roadmaps, or projects. We determined that our sales team would be in the dark, and it would be embarrassing to customers, but we could tolerate 5 business days without a CRM before really damaging our company and loss of sales that impacts the company.

Our SQL database application is the most important. In this scenario, we pretended to have 30 assembly operations workers who count on the database to run. Every 30 minutes we are down costs the company $450 in payroll, $5000 in lost/cancelled revenue, and $3000 in lost future orders/miscellaneous costs.

We determined that we cannot be down longer than 30 minutes.

Now that we know the maximum time, we can try to determine what solutions we can provide to make or exceed those RTOs.

Going through the list one more time, we provide surface level examples of how we can obtain those objectives.

You need to document your procedure thoroughly in your BC/DR plan, be realistic, and remember to include items such as:

• Time to diagnose an issue and determine which recovery method is needed
• Shipping cut off times
• Will call or turnaround time on hardware
• Power/Internet/Connectivity
• Communication with end users and training if users do not practice DR plan

Application	RTO	Recovery Method or Process	Meeting, Exceeding, or missing objectives
Office365/Email	1 hour	Office365 has SLAs and 99.99% uptime, RTO should always be met	Exceeding
Fileshares	5 Days	Backups are done daily, vaulted off site, remote access to offsite files would take IT about 1.5 days to get users back into the system	Exceeding
Quickbooks	3 Days	Backups are done daily and vaulted off site, if a disaster situation arose, we would be able to copy Quickbooks files to one users machine in less than 8 hours	Exceeding
CRM	5 Business Days	Assuming total database and server loss, we would need to acquire hardware and provision it, before downloading and restoring from offsite backups. An eta would be about 5 days	Meeting
SQL production	30 minutes	Backups are done hourly to a standby backup server. Offsite backups are done daily. Recovery would be difficult as we could spin up our server in the cloud as part of our offsite vaulting service, but that is accessed over VPN. Our company does not have a terminal server, so recovery in the cloud helps our BC/DR plan, but not our RTO	Missing

As you can see by the notes in the red above, we are doing everything we can in the budget with what we have today, but our RTO is about 5 days. We now know that we need a solution to get our RTO from 5 days to 30 minutes. Which moves us into the next phase – Presenting.

After we analyze the costs of the solutions, you can now present the solution to management.

Items you need to prepare:

• How RTOs are being met and not met today?
• What does that delta in time cost the company in hard and soft dollars when RTO is not met?
• What reputation/customer relationship damage is done and what is that cost?
• What is the cost of a solution that meets the RTO?
• If management thinks that the cost of a solution is too high, the conversation turns to what other solutions are available or what is the new (more realistic) RTO that we can meet?

You can then adjust your plan and start the process over.

When it comes time to find that solution or engage with management on realistic expectations, CCC can assist with it. CCC has multiple vendors, solutions, and out of the box thinking to bring to the table. We are here to help develop a customer’s BC/DR and RTO plans as well as to provide a solution to meet the needs based on the businesses lost revenue and RTO presented by the customer.