A Region Half Down

Disaster recovery tabletop exercise: a hybrid company's dispatch engine times out while its cloud dashboard and status page stay green.

By Synergetic Solutions

Disaster recovery Cloud region Hybrid, a server room and a cloud account

Summary

The dispatch engine in a company’s server room starts timing out against its cloud database. The cloud dashboard and the provider’s status page are green. This scenario is written for a company running a hybrid environment. The heads of engineering, operations and customer success are in the room with the CEO. The failover plan assumes a region is either up or down. What does the team do when that assumption breaks?

Ground rules

  • This is a no-fault discussion. Disagreement is expected, and nobody is being graded as a person.
  • There are no trick questions. Some reports in this story will turn out to be wrong, as they do in real incidents, but the narrator never misleads you.
  • The facts are as the facilitator reads them. Work with what arrives.
  • Answer with the people, tools and plans you have today.
  • Have your own disaster recovery plan open. Some questions ask what yours says.
  • Decisions are not precedent setting and may not reflect your organization's final position on a given issue. This exercise is an opportunity to discuss and present multiple options.

Setting

Millbrook Operations, a fifty person organization, schedules and dispatches field crews for regional water utilities and property managers. The dispatch engine, the file server and the build servers run in a server room on the second floor of the office. The customer portal, mobile app backend, database and photo storage run in a cloud account in one primary region with a standby in a second region. The dispatch engine reads and writes to the cloud database on every job it assigns. The server room reaches the cloud over a dedicated private connection installed by an outside network contractor.

The disaster recovery plan is fourteen pages, written two years ago for a utility customer’s vendor questionnaire by an infrastructure manager who has since left. It has a section per cloud region and a separate section for the server room. It names Dale, the head of engineering, as the recovery owner.

Roles at the table: Dale (head of engineering), Chiara (on-call engineer), Joaquin (backend lead), Ruth (dispatch operations manager), the customer success lead, and Helen, the CEO. Ray, the network contractor, is reachable by phone.

The plan that exists

Read this to the room. These are the relevant parts of the disaster recovery plan:

  • Production has two halves: the server room and the cloud account. Each has its own recovery section.
  • A regional outage is handled by failing over to the second cloud region. Promoting the database replica and switching the application takes about thirty minutes.
  • The provider’s status page and the cloud dashboard tell the team which region has failed.
  • The server room reaches the cloud over a private connection. A backup tunnel over the office internet line can carry production traffic and can be switched on in ten minutes.
  • Batch jobs are non-critical and can be left alone during an incident.
  • Monitoring covers every production system.
  • The recovery time objective is four hours. The recovery point objective is fifteen minutes.

The backup tunnel was switched on once, the day it was installed. Nobody at the table has read the server room section since the questionnaire was returned.

The assumption that fails

T+0, Tuesday 09:10. The dispatch engine starts timing out on writes to the cloud database. Technicians’ phones show new jobs stuck on “sending.” Chiara checks the cloud dashboard: every service green. She checks the provider’s status page: green. The server room’s own alerts are red, all pointing at the cloud.

Ask the room: The server room’s alerts blame the cloud, and the cloud looks healthy. How would you diagnose the fault?

T+12 minutes. Customer success reports that four property managers cannot see this morning’s work orders in the portal, and one writes, “your cloud is down again.” Dale reads it as a regional outage the provider has not admitted yet. He opens the regional failover checklist and tells Chiara to start.

Ask the room: What evidence does the room have that the cloud region is the problem? Who can authorize a failover, and does a customer’s guess count as a reason?

T+40 minutes. Failover is complete in twenty-eight minutes, close to the plan’s thirty minute standard. The portal and mobile backend run cleanly from the second region. The dispatch engine is still failing because it cannot reach the second region either. Chiara suggests the plan’s regional failover worked as written, and the outage was never regional.

Ask the room: Does the team stay in the second region, return to the first, or change nothing until the cause is known? Who decides, and what does each choice cost?

T+50 minutes. Chiara notices the server room’s network traffic is at its maximum, and traces most of it to the job that copies technician photos from the file server to cloud storage. A former contractor wrote that job. It retries every failed upload immediately and without limit.

Ask the room: Who owns this job, and who can stop it? What is lost if it stops? Would you consider this a batch job?

T+65 minutes. Joaquin, on his own initiative, starts building a copy of the dispatch engine on a cloud server in the second region, to cut the server room out entirely. Ten minutes in, he needs the dispatch license file and routing configuration. Both live on the file server in the server room.

Ask the room: Does Joaquin keep going? What happens if the original engine recovers while his copy is also running?

T+80 minutes. Dale contacts Ray. Ray logs into the cloud console and opens the private connection’s own health page, which none of the team has visited. It has shown heavy packet loss since two minutes before the first timeout. The provider offers an alarm on that measurement. It was never turned on, because the plan’s monitoring section lists services inside regions and nothing between them. Ray says the fault is in the carrier’s path and recommends the backup tunnel.

Ask the room: Why wasn’t checking that connection’s health page in the plan? Do you agree with the decision to leave monitoring off for that connection? What does your plan say about when to contact a third-party vendor?

T+1 hour 55 minutes. The backup tunnel is up. The switch took thirty-five minutes, not the plan’s ten, because Ray had to change the routes by hand and got two wrong on the first try. The office internet line carries only a fraction of the private connection’s traffic. The photo job is still running, and it cloggs the tunnel. Dispatch crawls until Dale stops the job.

Ask the room: When was the last time your company rehearsed switching to its backup systems to get honest numbers?

T+2 hours 10 minutes. Ruth reports that sixty utility and property-manager technicians have spent the morning working from routes cached on their phones ten minutes before the first timeout.

Ask the room: What does Ruth tell those crews?

T+2 hours 40 minutes. Dispatch is working again, and the portal shows today’s work orders. The team beat the plan’s four-hour recovery time objective. Then Ray calls back with the carrier’s findings. The fault was two miles away, at a junction only the private connection passes through. The office internet line and the private connection share a carrier and leave the building through the same conduit. Ray points out that a break in that conduit would have taken down both, with no tunnel to switch to.

Ask the room: Is the company recovered? Should Ray’s feedback about the single point of failure be worked into the disaster recovery plan? Does your disaster recovery plan have a hidden single point of failure?

Standing questions

  • Who owns recovery for the connection between the server room and the cloud?
  • Does it change what customers are told if the team calls this a cloud outage or a network failure?
  • Which customers were affected differently, and how would the team know which crews worked from stale routes?

Generic rubric

Did the team:

  • Name an incident lead early, and did everyone at the table agree who that was?
  • Name someone to keep a timed record of every decision and action as it happened?
  • Assign someone to communications who was not also working on the incident itself?
  • Reach the plan within the first ten minutes without relying on the system that failed?
  • Reach the credentials and consoles they needed without relying on the system that failed?
  • Establish a team communication channel that did not depend on the failed system?
  • Have a named person declare a disaster, or decide explicitly not to, against criteria they could state?
  • Establish which restore point they would actually use, and compare it with what the plan claims?
  • Estimate how long the restore would really take, and compare it with what the plan claims?
  • Set a point at which they would abandon or switch any parallel or untested recovery attempt, and make that call explicitly?
  • Decide who acts when the person who would normally approve a recovery decision cannot be reached?
  • Preserve the damaged state before restoring over it, so the cause could be established and any newer data recovered?
  • Verify that restored data was complete and consistent before declaring recovery?
  • Determine whether this is an outage, a data loss, a breach, or some combination, and on what evidence?
  • Determine whether this triggers anything in their customer agreements, and check the agreements rather than assume?
  • Check what their vendor and provider agreements say about outages, in both directions?
  • Decide whether legal should be involved, or decide explicitly that it is not needed?
  • Acknowledge the problem to customers early?
  • Tell customers what was known and what was not?
  • Give customers an expected restore time, with how confident they were in it?
  • Tell affected customers what was lost or delayed, in terms they can act on?
  • Use a customer channel that did not depend on the failed system?
  • Leave with a named owner for each correction to the plan?
  • Test whether the fault was inside a region before declaring a regional failover?
  • Decide who could stop the photo job, and stop it before it filled the backup path?
  • Check the health of the connection itself, not only the systems at each end?

Postmortem prompts

  • The backup tunnel had been switched on once, at installation. What is the smallest recurring exercise that would test your own backup paths?
  • The connection alarm was available and off. What do your providers already measure that nobody at your company is watching?
  • Failing over to the second region took twenty-eight minutes and did nothing for dispatch. What would your team check before starting a failover?
  • A job written by someone who has left nearly took down the only working path. How many jobs like it run at your company, and who would know?
  • The recovery time objective was met while crews worked from stale routes. Does your recovery objective measure what your customers actually lose?

Sources and inspiration

The organization and people in this scenario are fictional. It draws on these published accounts.

How can we work together

Schedule a discovery session

A complimentary, exploratory meeting to discuss your business goals and challenges, identify potential solutions, and determine if our services align with your needs.

Hire us for a project

Augmenting a specific initiative or project, leveraging our expertise and resources to deliver a defined outcome.

Retain us long term

Ongoing collaboration and support through regular meetings, check-ins, and access to our team's collective knowledge and experience, empowering you to make informed decisions and drive sustained growth.

Send us a message

Tick the box if you'd like our newsletter. You can unsubscribe anytime.

We care about your privacy. Learn how we handle your data in our Privacy Policy.

To deliver your service, we need your permission to store and process your personal data.