Who Deleted the Backups?

Disaster recovery tabletop scenario: stolen admin credentials encrypt production after deleting every snapshot the recovery plan depended on.

By Synergetic Solutions

Disaster recovery Backups Identity Small SaaS

Summary

Someone with the cloud administrator’s credentials encrypts the production database and empties the file storage behind it on a Sunday night. The disaster recovery plan says restore from snapshots. The snapshots were deleted eleven minutes before monitoring paged the on-call engineer, by the same credentials. This scenario exercises what a team does when the recovery plan’s only input is gone, and how it discovers that the plan assumed the backups were out of reach of whatever caused the disaster.

Ground rules

  • This is a no-fault discussion. Disagreement is expected, and nobody is being graded as a person.
  • There are no trick questions. Some reports in this story will turn out to be wrong, as they do in real incidents, but the narrator never misleads you.
  • The facts are as the facilitator reads them. Work with what arrives.
  • Answer with the people, tools and plans you have today.
  • Have your own disaster recovery plan open. Some questions ask what yours says.
  • Decisions are not precedent setting and may not reflect your organization's final position on a given issue. This exercise is an opportunity to discuss and present multiple options.

Setting

Brightmoor Payroll runs payroll for small businesses. Thirty remote workers, one public cloud account for production and a second account nobody remembers creating for staging. Production is a web application, a managed relational database with automated daily snapshots retained for fourteen days, and object storage holding uploaded documents, with versioning turned off because it doubled the bill.

The cloud account has a single administrator role. Four people can assume it: the two founders, the head of engineering, Dana, and the platform engineer, Theo, who does most of the actual work. The administrator role can do anything in the account, including delete snapshots and empty buckets. Multi-factor authentication is required to assume it.

The disaster recovery plan was written two years ago. Its ransomware page says: isolate, restore from the most recent clean snapshot, rotate credentials, and communicate.

Roles at the table: Dana (head of engineering), Theo (platform engineer), one founder, the backend lead and the customer success lead.

The plan that exists

Read this to the room. These are the relevant parts of the disaster recovery plan:

  • Daily database snapshots run at 03:00 and are retained for fourteen days.
  • Uploaded documents live in object storage, which “is durable” and therefore does not need a separate backup.
  • A ransomware event is handled by restoring the most recent clean snapshot to a fresh database instance. Estimated time: two hours.
  • The administrator role requires MFA, so a stolen password alone cannot use it.
  • Credentials are rotated after any incident. The plan does not say which ones or in what order.
  • The recovery time objective is eight hours. The recovery point objective is twenty-four hours.

Nobody at the table has checked whether the snapshot retention policy can be changed by the administrator role. Nobody has asked what “durable” protects against.

The assumption that fails

T+0, Sunday 23:40. Monitoring pages Theo: the application is returning errors on every request that touches the database. Theo opens the cloud console. The production database instance is running. Every table he queries returns rows whose contents are unreadable. A text file has appeared in the documents bucket with a contact address and a price.

Ask the room: Theo is alone with the errors and the text file. Who does he call first, and what does he leave untouched until they answer?

T+12 minutes. Theo wakes Dana. Dana opens the plan and reads the ransomware page aloud: isolate, restore, rotate, communicate. Theo starts isolating: he changes the database’s network rules so nothing can reach it, and revokes the application’s database password.

Ask the room: Isolation is starting before anyone knows the scope. What does isolating protect right now, and what evidence or options does it destroy?

T+25 minutes. Theo goes to the snapshot list to pick the most recent clean one. The list is empty. The retention policy on the database now reads zero days. He checks the account’s activity log: at 23:29, the administrator role modified the retention policy, which caused every existing snapshot to expire, then emptied the documents bucket, then issued the commands that rewrote the database. Every step came from the administrator role, with MFA satisfied.

Ask the room: Who declares that the plan no longer applies, and what replaces it?

T+40 minutes. The founder reads the activity log and points out that Theo was the last person to assume the administrator role, on Friday afternoon, and that the Sunday session came from the same cloud region Theo works from. Dana and the founder start weighing whether Theo did this.

Ask the room: Does Theo keep working on the recovery while this is checked, and who decides?

T+50 minutes. Theo’s laptop logs show it asleep from 22:10 until the page woke it at 23:40. The session’s source address belongs to a hosting provider, not a home connection. Dana and the founder let it drop. Theo has stopped volunteering ideas.

Ask the room: Neither fact rules Theo out. Who tells him where he stands, and does his access change while the investigation runs?

T+55 minutes. Dana asks the founder whether the company has cyber insurance and who the contact is. The founder thinks so. The policy document is in a shared drive that authenticates through the cloud account’s identity setup. Nobody wants to log in to anything until they know what is compromised.

Ask the room: What can be safely accessed, from where, and who decides?

T+70 minutes. The staging account, the one nobody remembers creating, sends a billing alert: compute spend has increased forty-fold in the past hour. Someone is running workloads there. Theo checks. The staging account trusts the production administrator role for cross-account access. Theo says the attacker has been in both.

Ask the room: If Theo is right, does that change whether the team can trust anything in the production account?

T+90 minutes. The backend lead remembers that a contractor set up a nightly logical export of the database to a personal object-storage bucket in a different cloud provider eighteen months ago, as a stopgap during a migration. Nobody turned it off. Nobody knows if it still runs, whether the credential still works, or whether that bucket has been touched.

Ask the room: Who goes to check the export, with which credential, and from which machine?

T+2 hours. The export exists. The last file is from Sunday 03:00. It is a plain export with no encryption. It has been sitting in a contractor’s personal account, reachable with a key that is still in a repository the attacker may have read. It is the only copy of the business the team has.

Ask the room: Does the team restore from it, and if so, what do they assume about its integrity? Is it inside the twenty-four-hour recovery point the plan promised, and does that number mean anything now?

T+2 hours 30 minutes. Dana asks the question the plan never did: how does the team restore into an account the attacker still controls? The administrator role’s credential has not been rotated, because rotating it requires the administrator role, and no one is sure whether the attacker has added a second path in. The only person who has ever built an account from scratch is Theo.

Ask the room: Can recovery proceed in the existing account, or does it need a new one? If it’s a new one, does Theo build it, and who watches what he does?

Standing questions

  • The price in the text file has not been discussed. Does anyone at the table have the authority to discuss it, and who would they need to involve first?
  • Who owns recovery tonight, and did anyone confirm that with Dana before Theo started isolating?
  • What has been preserved from the rewritten database and the account’s activity log before anything is restored over or rotated away?

Generic rubric

Did the team:

  • Name an incident lead early, and did everyone at the table agree who that was?
  • Name someone to keep a timed record of every decision and action as it happened?
  • Assign someone to communications who was not also working on the incident itself?
  • Reach the plan within the first ten minutes without relying on the system that failed?
  • Reach the credentials and consoles they needed without relying on the system that failed?
  • Establish a team communication channel that did not depend on the failed system?
  • Have a named person declare a disaster, or decide explicitly not to, against criteria they could state?
  • Establish which restore point they would actually use, and compare it with what the plan claims?
  • Estimate how long the restore would really take, and compare it with what the plan claims?
  • Set a point at which they would abandon or switch any parallel or untested recovery attempt, and make that call explicitly?
  • Decide who acts when the person who would normally approve a recovery decision cannot be reached?
  • Preserve the damaged state before restoring over it, so the cause could be established and any newer data recovered?
  • Verify that restored data was complete and consistent before declaring recovery?
  • Determine whether this is an outage, a data loss, a breach, or some combination, and on what evidence?
  • Determine whether this triggers anything in their customer agreements, and check the agreements rather than assume?
  • Check what their vendor and provider agreements say about outages, in both directions?
  • Decide whether legal should be involved, or decide explicitly that it is not needed?
  • Acknowledge the problem to customers early?
  • Tell customers what was known and what was not?
  • Give customers an expected restore time, with how confident they were in it?
  • Tell affected customers what was lost or delayed, in terms they can act on?
  • Use a customer channel that did not depend on the failed system?
  • Leave with a named owner for each correction to the plan?
  • Notice within the first thirty minutes that the disaster and the backups shared a blast radius, and say so out loud?
  • Decide who is authorized to speak with the party behind the text file, and whether to, before anyone replied?
  • Locate the cyber insurance policy and its notification contact without logging into a compromised system to do it?
  • Treat the contractor’s export as evidence and as a risk as well as a way back?
  • Establish a clean environment to recover into, and define what “clean” means for this incident?
  • Rotate credentials in an order that did not hand the attacker another window?
  • Protect the person who was briefly suspected, and record how that suspicion was resolved?

Postmortem prompts

  • The administrator role could delete the backups the recovery plan depended on. What would it take for that to be impossible at your company rather than merely unlikely?
  • “Object storage is durable” was true and irrelevant. What true statements in your plan protect against the wrong thing?
  • The staging account trusted production’s administrator. How many trust relationships like it exist at your company that nobody drew?
  • The best backup was an accident. What would it say about your plan if your most useful backup were an accident too?
  • Rotating the administrator credential required the administrator credential. What is your team’s plan for when the credential that fixes things is the one that is compromised?
  • Ten minutes were spent suspecting a colleague. What would shorten that to two on your team, and what would make it never happen?

Sources and inspiration

The organization and people in this scenario are fictional. It draws on these published accounts.

How can we work together

Schedule a discovery session

A complimentary, exploratory meeting to discuss your business goals and challenges, identify potential solutions, and determine if our services align with your needs.

Hire us for a project

Augmenting a specific initiative or project, leveraging our expertise and resources to deliver a defined outcome.

Retain us long term

Ongoing collaboration and support through regular meetings, check-ins, and access to our team's collective knowledge and experience, empowering you to make informed decisions and drive sustained growth.

Send us a message

Tick the box if you'd like our newsletter. You can unsubscribe anytime.

We care about your privacy. Learn how we handle your data in our Privacy Policy.

To deliver your service, we need your permission to store and process your personal data.