The Ransomware Files, Episode 3: Critical Infrastructure

(A toilet flushes in the background)

Jeremy Kirk: That's a sound that everyone will recognize. It's also a sound to be thankful for. More than half of the world's population does not have access to sanitation services that adequately treat waste. It’s one of the world's biggest sources of pollution.

We take it for granted that everything that goes down the drain flows to a place where it's properly treated, doesn't damage the environment, and of course, doesn't make us sick.

But there is increasing worry about the plants and facilities that enable safe water and wastewater treatment. They're the kind of facilities that no one really notices but are part of the infrastructure that is critical for society. Like many organizations, information technology in the last two decades has transformed their operations and also increasingly put them at risk of cyberattacks.

The Hampton Roads Sanitation District serves 1.7 million people in Eastern Virginia. That includes the gigantic Norfolk Naval Base and Camp Peary, where the CIA trains its agents. It runs 100 pumping stations, nine major plants and eight smaller plants. It can treat 249 million gallons of effluent a day. It’s also one of their caretakers of the Potomac aquifer, an ancient source of trillions of gallons of pressurized water, which is the region's main drinking source.

Roger Caslow had only been on the job as HRSD's first Chief Information Security Officer for six months in November 2020. On November 17, 2020, he stayed late at the office writing an information security governance document, covering aspects such as audit trail policies and backup standards.

Roger Caslow: Quite literally, I had my document up on my screen the night it happened, and everything's connected in, and everything's good. I'm like, "I don't need to save this. I'll come back to it tomorrow morning." It was 8 p.m. when I was leaving the office. I get a call an hour and a half later: "Roger, something bad has happened to the Windows systems." I drive back into the office. And that's really the realization that the document that I've been working on for weeks is now gone because it's encrypted.

HRSD had become the latest casualty in a growing wave of ransomware. There are increasing concerns about the cybersecurity weaknesses in water and wastewater plants. They're often underfunded, they may run outdated systems, and often use automation to save money. But that automation may have also increased their vulnerability to ransomware or nation-state actors. And there are a lot of these types of plants: there are some 52,000 drinking water plants and 16,000 wastewater systems in the United States. There have been at least four ransomware attacks in the last two years against water and wastewater facilities, and experts fear that tally could increase unless action is taken.

This is The Ransomware Files. I'm Jeremy Kirk.

In this podcast mini-series, I'm going to talk with those who have navigated their way through a ransomware incident and learn how they fought back, what they learned, and what tips they can pass on to others. No ransomware infection is ever welcomed, but there's invaluable knowledge gained. There should be no shame in getting infected, but it's important to share the lessons.

This episode of The Ransomware Files is sponsored by Cofense. Cofense is a leading provider of phishing protection, detection and response solutions. It’s the only company to combine a global network of 30 million people reporting phishing attempts with advanced AI-based automation to find and stop phishing attacks. Stay on until the end of this episode to hear me speak with Tonia Dudley, Strategic Security Advisor at Cofense, and we’ll discuss ransomware, phishing, and how to protect yourself from a breach.

Roger had clear instructions for his team: don't turn the machines off, but disconnect them from the network.

Caslow: Knowing what our infrastructure lacked, or had or didn’t have, I immediately told my engineers, "Go downstairs and disconnect everything." They said, "Disconnect everything?" I said, "Everything. If it has an external point, disconnect it. If it's internally connected, disconnect it. I want the servers disconnected from each other. And I want this as a hard disconnect. Everything is hard: pull plugs, pull plugs, pull plugs. Should we turn it off? No, don't turn it off." Because typically, users want to turn it off. And from the forensic standpoint, I want that thing up and going in case I need to do a forensic assessment on it. It's still working, and the bad guy still thinks he's in there doing something.

Ted Henifin is a civil engineer by trade. He has been General Manager of HRSD for 15 years. He'd just spent a relaxing weekend with his wife in Kiawah Island in South Carolina when he came back to work.

Ted Henifin: I'm an early riser and early arriver at work. And so I'd gotten in the car and headed to work and got there before six in the morning. And I'm not fully engaged. So I get to the door and there's a handwritten sign that says, "If your computer is off, don't turn it on. If it's on, don't turn it off." That was really strange.

Kirk:When HRSD was infected, it was already on the path to improving its information security practices. It was in the second year of a five-year plan designed by Gartner, which included the hiring of Roger as its CISO. But of course, as we all know, none of these types of changes happen overnight.

HRSD was infected by Ryuk, which if you remember from episode one, is the name of a character in a Japanese manga series called Death Note. Ryuk is a ransomware-as-a-service group: other cybercriminals use the Ryuk malware and pay a portion of the ransom to those who develop it. Ryuk has hit corporate targets but also has shown no mercy: hospitals and medical facilities, even in the midst of the COVID-19 pandemic, have been hit.

Ryuk has proved to be one of the most profitable ransomware groups. In January 2021, security researchers estimated that it had taken in as much as $150 million since it came on the scene three years prior. But that prominence and profit have been noted. In mid-November 2021, the U.S. Department of Justice announced the arrest of a Russian national named Denis Dubnikov. Dubnikov ran two cryptocurrency trading platforms called Cayote Crypto and Eggchange. He was indicted in August by a grand jury in Oregon on charges of conspiracy to commit money laundering. He stands accused of receiving $400,000 worth of Bitcoins that came from Ryuk victims. He is now awaiting extradition to the United States from the Netherlands. Dubnikov's arrest marks some of the first fruits of a push by the U.S. and other countries to try to hold those involved in ransomware accountable.

The attackers had been inside HRSD's systems for three weeks before Ryuk was deployed. An employee had opened an Excel spreadsheet that contained embedded malicious code. That led to the installation of Zloader, which is a descendant of the infamous Zeus banking malware. Zloader is all-purpose malware: it can steal information or install other malware on a system. The employee had administrative access, which allowed the attackers to spread deeper in HRSD's systems. That included nesting in one of HRSD's domain controllers, which was known as SSADS3.

The attackers also tampered with the group policy objects, which are the set of virtual policies in Active Directory. They modified the policy in a way that told HRSD's antivirus software - which was McAfee at the time - to ignore certain suspicious activity.

Eventually, the attackers used Zloader to plant a Cobalt Strike beacon on the system. Cobalt Strike is a penetration testing toolkit, and its beacon is an agent that’s deployed on a remote machine. The beacon is kind of a like a Swiss Army knife, with a bunch of tools that allow for an attacker to explore and persistently stay on target. Eventually, Ryuk was pushed out to 45 unique systems across HRSD's network. However, its operational technology - which are the systems that control actual wastewater treated - were unaffected. Long before Ryuk entered the scene, Ted says HRSD ensured that there was a strong division between IT and OT.

Henifin: We take a pretty strong stand with our operations. They just weren't going to be able to access the network and control critical processes remotely, in any old fashion. And so that was that battle between OT and IT that went on for a long time. And we basically just locked it down and said, the only way we're ever going to allow control is on a HRSD box. You're not going to be able to go through the internet on your home computer, and go change something. It's going to be really well controlled to make this work. And that frustrated the heck out of our operators for a long time. I think they appreciate it now.

Kirk:When an organization gets infected with anything, there's a good chance that someone has clicked on a malicious attachment. And forensic investigations can get down to the person who did the click, down to almost the minute that it happened. Ted thought it might have been him.

Henifin: At the end of the day, someone opened an attachment. Socialization stepped up tremendously. You put yourself back into that time frame that I started with: six to eight months of pandemic response, remote work, and people doing things differently. It’s no surprise that the end of 2020 saw a huge uptick in ransomware attacks. Everybody was on a bit of a COVID fog. It set the stage for an environment that was rich for people opening things they shouldn’t have. When it first happened in the time, it was CrowdStrike doing their digging through to find out the keystrokes and what happened, and I kept waiting for them to say it was me. My nightmare was going to be: "general manager opened something which he was not supposed to on an email".

Kirk: It turned out that Ted wasn't the source. But CrowdStrike, which did the forensics investigation, figured who did. Do you tell the person that they were the one who opened the door for a ransomware attack?

Henifin: There was curiosity on the incident response team. Everybody was working on it, especially the CrowdStrike folks and others, to understand what she was looking at, at the time, because while they could get it down to keystrokes, they couldn't exactly tell what exactly she had opened. So I had that uncomfortable phone call. And she was heartbroken and crying. And I said, "It's not your fault, this could happen to any of us." I kept telling her that I was waiting for it to be me. But we got a description of the files she opened and why she opened them. And in that COVID fog and remote work environment, it is not surprising. I'm surprised that we didn't have more problems. It's a challenging conversation to have with somebody. But we managed it the best we could. And she's a great employee - one of our superstars.

Kirk:Ryuk took out a whole lot of HRSD's Windows systems. There was no email. Customers couldn't pay their bills, and HRSD bills hundreds of thousands of accounts. The customer call center was down, as well as the customer self-service facility. HRSD also relies on water meter readings from localities in order to bill for their sewerage, and those connections had been severed as well. Communication inside HRSD was hampered because its internal SharePoint site was down. All of this posed immediate problems for Leila Rice, who is the Director of Communications for HRSD, and a former broadcaster and journalist.

How did you find out that the district's computers had been infected with ransomware?

Leila Rice: I got a very early morning phone call, shortly after or before my morning alarm - I can't recall which. My general manager Ted Henifin called me. I thought he was joking - I was like, "you're kidding." And he said, "I wish I was."

Of course, the call I got was that we'd been struck by ransomware. I had no access to any resources. I couldn't get on our website to try and put any kind of messaging out, I had no email address that was recognizable. So we had to start thinking how we were going to communicate this information.

Kirk: So, she turned to the telephone. She contacted organizations such as the Hampton Roads Planning District Commission to get the word out. She also relied on external channels, and a social media app called Next Door. She and others in HRSD set up new work email addresses, and media releases following the incident bear Leila's Gmail address. One primary concern from the public was whether any customer information had been compromised and if its actual operational systems were affected.

Rice: It was critical to let people know. Any billing information - we have that secured on a third party, so we don't have any of that. It was key information to let them know that. Also, we had to make sure that they were aware of billing delays and not make them feel like they were going to be penalized for this.

Most importantly, the systems that operate our services were separate. So there was never any interruption to our wastewater treatment, there was never any danger of public health or environmental damage or anything like that, because we could continue providing the service that we provide.

Kirk: For internal communication, she drafted a daily newsletter that was distributed as best it could be. People set up new email accounts for work while HRSD's main system was being restored. The website still worked, and Leila was able to send news posts to their website provider who would then post those updates on the site. And there were other channels like Twitter. Roger says it all worked.

Caslow: Our Communications Director Leila Rice was a rockstar. She managed all that communications out. She took care of all that stuff. Because again - lack of a formal incident response communication plan in place. We were shooting from the hip. Now, we shot from the hip with a 12 gauge gun on the side of a barn. I think we hit it pretty well.

Kirk: Leila says a takeaway from those trying circumstances is that organizations should give thought to how they would communicate if none of the normal systems work.

Rice: It's really making sure that you have a backup plan. And communication, when everything is down, is still necessary to ensure that updates are provided both to your external customers and your internal customers.

Kirk: Other departments also had their challenges. Anisea Burl is HRSD's Accounts Payable Supervisor. HRSD has lots of suppliers who, of course, still wanted to be paid even though there was a ransomware incident and some were concerned. Anisea says HRSD had help from its banking partner in making automated clearing house payments and also using ePayables, which are essentially card payments. She says vendors were understanding.

Anisea Burl: It was really, one day at a time. Just making sure we kept the line of communication open with our vendors and suppliers, giving them as much information as we could. And we had some suppliers that needed to get paid. Also some suppliers who were like, "when you're up and running, let us know". So they were very understanding. We have really good working relationships with them.

Kirk:HRSD's recovery kicked off with a call to Beasley, its insurer. HRSD had taken out cyber insurance several years prior, and Ted, at the time, thought it might be a bit of a waste. He now says that after Ryuk, it was one of the best decisions HRSD had made. The insurer organized other providers that immediately went to work. CrowdStrike worked on incident response. A firm in Virginia, Moxfive, started to rebuild systems. Ted says a robust insurance policy was critical to getting HRSD running again in just three weeks. And it helped take pressure off HRSD's own employees.

Henifin: I kind of knew this from previous disasters and other areas, but it becomes a matter of huge pride for your employees. First, it's been damaged, you were attacked, you've been taken down. And so your whole IT department is reeling because they take it very personally and they want to do the recovery all by themselves. And so it took a while to get an agreement that we're going to need staff augmentation remotely. They've got to give up some of this because they kept wanting to, "I'll take all this, and I'll take all that," and it's like, "Well, no, we got to get up fast, got to get back, let's get some more resources in here". And it was having to help them get past that personal pride that's been dinged-up because they wanted to rebuild this all on their own. And I know they had the ability; we just didn't have enough resources to make that happen.

Kirk: Let's talk about cyber insurance. The first cyber insurance policy was written back in the days of Netscape and Ask Jeeves. It was in 1997 and was called Internet Security Liability Policy, according to a March 2018 story in the Insurance Journal. The policy was launched at a party in Honolulu appropriately and prophetically called Breach on the Beach. Many organizations have since considered cyber insurance essential to mitigate the costs of a breach and the popularity of such plans has grown with the rise in cybercriminal activity. But it's now grown so bad and reached a tipping point, and insurers are in the midst of change.

The number one reason for a cyber insurance claim is due to a ransomware incident. That means that cyber insurance is getting a lot more expensive. In August 2021, insurer AIG said it had increased its cyber insurance premiums up to 40% while lowering coverage limits, and it looks as if other companies may follow suit. New policies are getting harder to secure. Renewals are sometimes being rejected. Lloyd's of London was recently discouraging its syndicate members from taking on new cyber insurance clients. Ransomware operators quickly caught on that cyber insurance policies were covering ransoms and have openly said that made those kinds of organizations preferred targets. Insurers are increasingly requiring that organizations meet baseline cybersecurity standards, but of course, cyber defense is ever-challenging and changing.

Roger says he saw some of this coming.

Roger Caslow: For $15,000. We got like almost $200,000 back.

Insurance companies are out for profit. I mean, who isn't? They're in a business. Business is profit-motivated. If they don't understand the business they are going into, they will get fleeced. You will get fleeced every time. And the insurance providers, again, as I mentioned before, didn't understand the environment they're going into. And then ransomware is en masse.

Kirk: What may not be good for the insurance companies has been quite helpful to policy holders. Insurance has pulled many organizations out of the muck of a ransomware incident. It helped HRSD get back and running in around three weeks. That's a pretty quick recovery, but it was aided by some good fortune such as the fact that not all of HRSD's systems were vulnerable to Ryuk.

Less than half of HRSD's systems were Windows. Many of its business systems were Linux and Unix, which stored its employee data and ran its ERP system. Also, its backups, which ran on Dell’s EMC Avamar platform, were not in terrible shape. Some were corrupted or encrypted, but Roger says they were able to go back a few weeks to get clean copies. And then sometime while HRSD was getting back up on its feet, Ted says that FBI got wind of its troubles.

Henifin: We are not part of a city or a county. We don't have any association with a law enforcement agency. And so unlike a lot of city organizations that get hit, their police departments immediately get involved. I don't even know how the FBI found out that we were battling this. This was in the early days, and I get a call saying, "Hey, the FBI wants to talk to us." And I called the attorneys who are into insurance that hooked us up - we were running everything through the insurance attorney’s office. And the FBI wants to come in. What's the right answer here? The advice I got was, "In Virginia, if you've lost information, you've got to bring law enforcement in. If you haven't lost any information that you know of, you don't have to bring law enforcement in."

Kirk:The FBI is now very, very interested in ransomware. And this year, fighting ransomware has become a top law enforcement and national security priority. The White House estimates that ransomware payments amounted to more than $400 million in 2020 and at least $81 million for the first quarter of 2021. This year, President Joe Biden's administration launched an interagency task force to tackle ransomware, including tracing virtual currency payments, disrupting ransomware actors through offensive operations and pressuring countries such as Russia that are believed to be knowingly harboring ransomware actors.

To move quickly against ransomware actors, however, the FBI has said it needs victims to quickly step forward. As a case in point, it cites the assistance it gave to software developer Kaseya during one of 2021's most notable ransomware incidents.

Kaseya developed remote monitoring and management software called the Virtual System Administrator (VSA). The software is used by managed service providers to manage the systems of their clients. An affiliate of the REvil ransomware gang exploited vulnerabilities in the on-premises version of VSA. Then they used VSA to infect the clients of those managed service providers with ransomware. Up to 60 managed service providers were affected as well as 1,500 of their clients.

Shortly after the attack in July 2021, the FBI obtained the universal decryption key that could unlock data from all Kaseya victims. It was an unprecedented revelation and one sign that the U.S. government, in cooperation with other foreign partners, was moving to offensively strike back.

In November 2021, there was more Kaseya-related news. The Justice Department announced it had indicted a Ukrainian national, Yaroslav Vasinskyi, for the attack. Vasinskyi was arrested in October 2021 in Poland. At the press conference announcing Vasinsky's arrest, Deputy Attorney General Lisa Monaco said that Kaseya's early cooperation in the incident was key.

Ted says that he understands the FBI's perspective, but it brings with it a layer of complication. The FBI investigates crime and doesn't do incident response or help victims rebuild systems. There also appeared to be no legal obligation on HRSD's part to get the agency involved: it didn't appear the attackers had exfiltrated any data. HRSD's lawyers recommended to touch base with the FBI later - after the remediation work had been done.

Henifin: We were maybe four or five days into the recovery at this point. The lawyers said, "They're going to want to stop and interview everybody." And yeah, we were hitting our stride at this point. They added: "You're really going to get slowed down, and they're not going to bring value, they aren't going to be able to help you recover. They're just trying to understand who did it. And they’d really like to trace the money." Their recommendation was for me to call back and say, "Hey, we'll be happy to share the results once we know what happened. But we don't want you to help us right now."

That was a pretty uncomfortable phone call. I called the head of the Norfolk office of the FBI to tell him, "Hey, we'll get back to you. Thanks." And he [the FBI Norfolk officer] continued to follow up. Then we finally did connect with him once we had a full report on the incident, and we were able to share with them exactly what happened. But they weren't that interested. I think they purely wanted to follow the money if you're paying the ransom.

Kirk: You may have guessed by now that HRSD did not pay the ransom. But it wasn't clear cut at first. Ransomware events are, of course, stressful. Ted says that HRSD bills for some 480,000 accounts and the billing systems were down. He feared a repeat of something that occurred in 2007. In that year, HRSD changed its billing system. The conversion didn't go well, and it turned into a press and public relations disaster. Ted knew the ransomware situation could start to go south fast as well, so he was keen to get systems up and running. In this case, HRSD knew the decryption key would work. It had sent a couple of encrypted files to the threat actors, who did return the decrypted files. But Ted says recovering a file one at a time was going to take too much time.

Henifin: It was a serious consideration. I was putting a lot of pressure on the team to give me time. Every passing hour, I was pushing hard to say, "What is the faster path here? Is it paying the ransom faster?" And over work, I'm all in. I needed the fastest solution. And I'd sit in all the meetings, listen to all the experts, and then give them another six hours or 12 hours to do more work to figure out which would be the fastest. And they finally convinced me that even if we paid the ransom, it would take a significant amount of time using the decryption key to get all this back running. And even then they weren't sure how long that would take. So they convinced me at that point that we could recover from backups, they could rebuild the system faster than we could do it from the decryption key. So the decision was not a financial decision. It was a timing decision.

Kirk: Roger jokes that he wished they would have paid just to get the document back he was working on the night the ransomware struck.

Caslow: I just want my PowerPoint presentation back. Can I have that back? We pay them a little money to get that, and get the key for that specific file.

Kirk: Over the last two decades or so, water and wastewater plants have moved to automation ¬ webs of remote sensors, programmable logic controllers and SCADA systems that can be controlled and monitored by computers from afar. Those technologies have allowed for costs savings, fewer employees and conveniences such as remote management. Valves and pumps operate automatically, and chemical management systems can make their own autonomous adjustments based on readings. The savings from those automation investments though were never put towards cybersecurity because in those days, there wasn't an obvious cybersecurity threat.

That's changed. In October 2021, the Cybersecurity and Infrastructure Security Agency issued an advisory about ongoing cyber threats to water and wastewater systems. In a two-year period, there were ransomware attacks against facilities in California, Maine, Nevada and New Jersey. There were other incidents as well. In March 2019, a former employee of a water and wastewater facility in Kansas used access credentials that had not been revoked to remote access one of the facility's computers. More recently, in February 2021, a small town in Florida called Oldsmar went public with a disturbing tale. An intruder gained access to a TeamViewer remote access account connected to a human-machine interface. That HMI was used by the facility to remotely control parts of the plant. The intruder increased the level of sodium hydroxide to be added to the water from 100 parts per million to 11,100 parts per million. Sodium hydroxide, or lye, is used to make the water more alkaline. But too much lye can be caustic, and it's the main ingredient in drain cleaner. Fortunately, city official said a plant operator actually noticed that a mouse pointer was mysteriously moving on its own, indicating someone was in the system, and tampering with the plant controls.

Aside from operational technology systems, plants also just have normal IT systems. They run Microsoft Office and Active Directory, and have email applications and billing systems and phone systems. Like any other organization, the systems are potentially vulnerable to exploitation. And it's the intersection of those IT systems and OT systems that many experts says poses a national security threat. If your adversaries can meddle with water - jumping from IT systems to OT systems - it can have a lot of downstream effects, so to speak. It's not just ransomware, either, it's also nation states.

Mark Montgomery is a retired rear admiral. He is Senior Director of the Center on Cyber and Technology Innovation with the Foundation for Defense of Democracies. The FDD is a nonpartisan think tank. He's also a former Executive Director of the Cyberspace Solarium Commission, which was formed in 2019 to develop ideas to help defend the United States against cyberattacks. Mark says that water may represent one of the most insecure sectors as far as critical infrastructure is concerned. Here he describes a worst case scenario involving the U.S. water supply and a cyberattack.

Mark Montgomery: My ability to either turn your system off remotely or probably even more dangerously, change the chemical balance of the water, change the alarm set points, introduce unhealthy water into the potable water system. It is a possibility. And I know we like to not say that in public, but it is a possibility. And that possibility introduces significant risks and likelihood time’s consequences. Because here's my worst case, my worst fear in terms of the drinking water system in America: An adversary attacks four or five drinking water systems simultaneously around the country and you start to get reports from four or five different areas that the water is bad for elderly or for young or just in general unhealthy, and don't drink it. Most people don't believe that they can tell from the taste of their tap water, whether they're drinking good tap on or bad tap water. So what you introduce is a loss of credibility into the drinking water system. That's the worst case.

Kirk: There are many issues in play here: knowledge gaps, technical debt, the fact that most critical infrastructure is run by private companies or state and local governments, which adds layers of complication for coordination. And of course, funding challenges. Mark says that people don’t want to see their water and wastewater bills go up.

Montgomery: We're talking about the real witch's brew of trying to fix things, a mix of water utilities with limited excess capital. Most citizens are like, "Hey, I like my water and I have no intention of paying higher rates.’ And you figured out internally how you're going to pay for this new cybersecurity problem. And that's just not reasonable."

Kirk: In November 2021, Mark and the FDD released a set of recommendations to strengthen the cybersecurity of the water and wastewater sector. The recommendations are set to be passed onto lawmakers and hopefully translated into legislation.

One of the recommendations is to give a boost to the Environmental Protection Agency. The EPA is the agency that is responsible for the cybersecurity of the water and wastewater infrastructure. According to the Cyberspace Solarium Commission, there have been concerns that there has been insufficient coordination between the EPA and other stakeholders in water utilities’ security.

Mark says the FDD would like to see more EPA grants given to the water sector to go towards cybersecurity. The EPA should also be better resourced in order to fulfill its role as Water Sector’s Risk Management Agency.

Montgomery: We have to resource and empower the part of EPA that's responsible for the cybersecurity to succeed as the Water Sector Risk Management Agency. As the government lead in this area, first resource the EPA properly. This means that EPA has got to put its hand up and ask for the right amount of resources. I think EPA routinely asked for about a third of the amount of money they need to do this right. Because they want to spend their money elsewhere. And I think at some point, I acknowledge the important work EPA does in climate change, drought issues, rising sea level and natural disaster issues. And this is the problem with cybersecurity. We're competing with the four signs of the apocalypse. But we need to find cybersecurity or you have a problem worse than those other four in your backyard right now.

One of the FDD's recommendations is to direct the Cybersecurity Infrastructure and Security Agency to increase support for the water sector. In fact, CISA has already been helping. James Cratty is the acting Regional Director for the CISA Region 3, which covers the mid-Atlantic region ranging from Pennsylvania down to the District of Columbia.

James Cratty: The importance of cybersecurity is just as important as physical security around sensitive and critical functions such as water and wastewater. We have to go out and establish buy-in at the top. There has to be a culture of security and importance that comes from the board of directors or the owners and operators of these companies to say, "Hey, this is important to us. Move that forward."

With a small staff, it can be even more challenging, especially if you're looking at using a third-party vendor or third-party contractor to manage a lot of your security or your infrastructure and technology that you run your plant on.

Kirk: James says that CISA does assessments of such facilities, which often result in a candid view of risks and vulnerabilities.

Cratty: Investment in water infrastructure and wastewater infrastructure is all over the map in this country. Some places have very high-tech facilities, others have done just a little and have not reinvested. So they've made an initial investment but may not keep up with those investments going forward. So there's a lot of dated systems out there that may not be supported anymore.

Kirk: The money issue is a problem, of course. But James says there's lots that organizations can do to improve their baseline cybersecurity that does not cost much.

Cratty: I would say the first thing that our cybersecurity advisors recommend when they go out and they meet with folks, is what can you do initially that's low cost, no cost, a lot of times low tech solutions on your system. And are you doing those because a lot of times, it's that initial employee sitting somewhere that does something that makes it easier for a threat actor and nefarious actor to get into your system.

So do the top five things first. Don't click on any suspicious links. If you are using a remote desk protocol, it needs to be secure and you need to monitor it, you got to watch your systems. Make sure your operating systems and your software are kept up to date. And that you're patching on those systems as well. Additionally, use strong passwords and use multi-factor authentication whenever you possibly can. Multi-factor authentication alone can reduce the risk of a cyberattack by 99.9%. And all the things that I just described, don't cost a lot of money for organizations to do. And a lot of organizations aren't doing all of those.

Kirk: A year on from the ransomware attack, HRSD has much stronger defenses in place. Its backup regime has been revamped, its IT and OT is even more rigidly separated, its access controls have been closely scrutinized and bolstered, and it’s using new security software and provider to help detect attacks. It has improved its communication strategy in a time of crisis. In reflection, Roger has an interesting view of the incident: it was painful at the time, but perhaps needed.

Caslow: I think every organization needs at least one ransomware incident in their life. I know people won't say that they want it. Nobody really wants it. But I think you kind of need it. We all need to work out a little bit more and eat leafy greens.

Kirk: Roger says the commitment of everyone in very trying circumstances pulled HRSD through.

Caslow: My leadership and my management support was amazing. My general manager, Ted Henifin - amazing man, super supportive. My CIO, super supportive. Our staff, they are although not security experts, they came in and they did what they needed to do when they needed to do it. And that's the important part because security protection is not all about just security professionals. It's about the whole team.

Kirk: If you enjoyed this episode of The Ransomware Files, please share it on your social media platform of choice. If you would like to participate in this project, please get in touch with me - my DMs are open on Twitter and I'm easy to find on LinkedIn. I'm looking for other people, organizations and companies that can share their unique experiences for the benefit of all until ransomware, hopefully, becomes a thing of the past.

Next up is a chat with our sponsor for this episode, Cofense.