Summary

A small SaaS company still runs a VPN appliance from the days before single sign-on, and nobody has used its admin console this year. On a Sunday night someone logs in, and by Monday morning production hosts are encrypting themselves. The scenario covers the first three hours. The team has to decide whether to cut the network before it understands what is on it, find out where the backups actually are, and discuss a ransom note, which is a conversation nobody in the room has ever had.

Ground rules

• This is a no-fault discussion. Disagreement is expected, and nobody is being graded as a person.
• There are no trick questions. Some reports in this story will turn out to be wrong, as they do in real incidents, but the narrator never misleads you.
• The facts are as the facilitator reads them. Work with what arrives.
• Answer with the people, tools and plans you have today.
• Have your own incident response plan open. Some questions ask what yours says.
• Decisions are not precedent setting and may not reflect your organization's final position on a given issue. This exercise is an opportunity to discuss and present multiple options.

Setting

Corvid Logistics is a thirty-five-person company that sells route-planning software to regional delivery fleets. Some fleets also install a small Corvid program, the connector, on their own networks. Production runs on a public cloud, but the company began life in a colocated rack, and some of that era survives: a VPN appliance that engineers still use to reach a jump host, a file server with customer onboarding documents, and a backup server that takes nightly copies of the production database and pushes them to cloud object storage.

The VPN appliance was set up by an engineer who left two years ago. It authenticates against the company’s directory with a service account that also has administrative rights on the jump host and the backup server, because that was the simplest way to make the nightly job work. Nobody has logged into the appliance’s admin console this year.

Everything else uses the identity provider with single sign-on and MFA. There is an incident response plan. Its ransomware section, copied from a template, merely states “isolate affected systems.”

Roles at the table: CTO, the platform engineer on call, a second engineer, the head of customer success, the finance lead, and the CEO.

The first sign

Monday, 06:50. The on-call platform engineer, Dani, gets paged because the production API is returning errors. The database is up. The application servers are sluggish. When Dani opens a shell on one of them, the home directory contains a text file named in capital letters that was not there on Friday. It says the company’s files have been encrypted and gives an address to contact. Dani’s first reaction is that it looks like a template someone pasted in as a joke.

Dani checks a second application server. The same file appears and the log files end in a new extension on both hosts.

What happens next

T+15 minutes. The CTO joins the call, reads the note and asks Dani to shut down every production host immediately. Dani points out that shutting them down destroys whatever is in memory, which is the only place the encryption process’s keys might still be. The CTO asks how long it takes to encrypt a database server. Nobody knows.

Ask the room: Shut everything down, isolate the network, or leave hosts running while capturing evidence? What does each choice cost, and who has the authority to choose?

T+30 minutes. The second engineer finds that the database host itself is not encrypted yet, but its disk activity is unusual, and a process is running under the backup service account that nobody recognizes. The service account has administrative rights on that host. The engineer asks whether to kill the process or capture it first.

Ask the room: Kill the unknown process on the database host or capture memory first? How long is the room willing to wait for a capture while encryption might be progressing?

T+45 minutes. Customer success reports that three fleet customers called because their morning dispatch failed to load. One of them says drivers are sitting in a parking lot waiting for routes. Customer success wants to send a status message. The CTO says nothing goes out until they know what happened.

Ask the room: What can be said to customers before the cause is known? Does “we are investigating an outage” become misleading once the room knows it is ransomware?

T+60 minutes. The finance lead mentions that a payment processor sent a security bulletin on Friday about a compromised integration partner, and suggests the intrusion came through the payment integration.

Ask the room: If the intrusion came through the payment integration, what does the team do next, and who follows that thread?

T+75 minutes. Dani starts a restore from the backup server and finds it is unreachable. The backup server accepted a login from the same service account at 22:10 on Sunday. The cloud object storage bucket that receives the nightly copies still exists, but all seven objects in it, one for each of the last seven nights, were overwritten at 23:30 with files the same size as the originals and a new extension. The bucket has no versioning turned on.

Ask the room: Does this change the containment strategy or the communication strategy?

T+80 minutes. The second engineer pulls the VPN appliance logs. They show an administrative login on Sunday at 21:40 from an area where the company has no customers, using the same service account, and a firmware update the company never applied.

Ask the room: Does this rule out the payment-processor theory, or does someone stay on it?

T+90 minutes. With the backups gone, the CEO asks what the ransom is. The note names no amount. Nobody has contacted the address in it, and nobody knows whether contacting it is a decision the company can make on its own. The finance lead asks whether the cyber insurance policy has an opinion. Nobody has read the policy.

Ask the room: Who decides whether to make contact with the attacker? Who decides whether to pay? Is that a board decision, a CEO decision, or an insurer’s decision?

T+2 hours. The second engineer finds a cloud snapshot of the database volume from eleven days ago, taken manually before a schema migration and never deleted. The second engineer says it is the only clean copy anyone has found. The head of customer success does the arithmetic on eleven days of lost dispatch history and asks whether that is better or worse than paying.

Ask the room: Restore from an eleven-day-old snapshot or wait for a better option that may not exist? Who owns the decision to accept eleven days of data loss on customers’ behalf?

T+3 hours. A customer’s IT lead emails to say their own security tooling flagged an outbound connection from the Corvid connector installed on their network, and asks whether Corvid has been compromised. The connector shares a code-signing certificate with the rest of the product.

Ask the room: A customer’s own tooling has flagged the Corvid connector on their network. Who tells them what, and does the company’s obligation to that customer differ from its obligation to the others?

Standing questions

• Is this a breach, an outage, an extortion event, or all three, and who makes that call on what evidence?

Generic rubric

Did the team:

• Name an incident lead early, and did everyone at the table agree who that was?
• Name someone to keep a timed record of every decision and action as it happened?
• Assign someone to communications who was not also working on the incident itself?
• Decide, against criteria they could state, whether this was an incident and how severe it was?
• Decide when leadership would hear about it?
• Establish who could authorize high-impact actions, such as taking a system offline?
• Decide who acts when that person cannot be reached?
• Determine whether this is a security incident, a data breach, a contractual matter, or some combination, and on what evidence?
• Check how far it reached beyond the first system they found?
• Keep a current list of what is known, what is suspected and what is unknown?
• Weigh containment against investigation before acting?
• Preserve the logs and other evidence before taking any action that would alter them?
• Check what their customer agreements say about incidents and notification before drafting anything for a customer?
• Check what their vendor agreements say about incidents, in both directions?
• Call legal, or decide explicitly not to, at the point they first suspected unauthorized access?
• Make an explicit decision about contacting law enforcement?
• Check their cyber insurance policy's notice and consent terms before engaging outside responders or discussing payment?
• Agree one approved message for staff, customers and leadership?
• Send it over a channel they had reason to trust?
• Isolate the VPN appliance and disable the shared service account once the entry point was found, and check what else that account could reach?
• Inventory every copy of production data, including the manual snapshot, before deciding what recovery was possible?
• Decide who may communicate with the attacker and who may not, and enforce it?
• Treat the customer whose network flagged the connector differently, and decide what to say to customers who have not yet asked?

Postmortem prompts

• The VPN appliance had no owner, an admin console nobody had used this year, and a service account with rights it did not need. Which of your systems fit that description today?
• The service account that opened the VPN also opened the backup server. What would make your backup copies unreachable from a compromised production account?
• Object storage versioning was off. Which safeguards in your cloud accounts are one setting away and have never been evaluated?
• A bulletin pointed the team at the payment integration while the VPN logs sat unread for twenty minutes. What would send your team to the logs first?
• The processor’s security bulletin reached the team through the finance lead. Who owns third-party risk at your company, and where do vendor security notices go?
• Nobody had read the insurance policy before they needed it. What would it take to make that document the first thing someone opens?
• The plan’s ransomware section said only “isolate affected systems.” What would your plan need to say to settle the argument the CTO and Dani had at T+15?
• The only clean copy existed because nobody deleted it. What would make a restorable copy something your company chooses to keep?