Lessons to Learn from the Industrial Grade Hack At Oldsmar, Florida

Summary: A hacker’s attempt to inject lye into the city of Oldsmar’s water supply, raises troubling questions about whether cities in the United States have appropriate security controls over their critical infrastructure. This article explores those issues and provides security guidance for other cities worried about possibly similar attacks.

Nothing new at Oldsmar

The hacker took control of the water treatment system through TeamViewer, a remote access software that allows you to access your laptop while working from home. The attacker tried to increase the level of sodium hydroxide (also known as “lye”) to more than 100 times the normal levels. Fortunately, the operator noticed his mouse moving by itself and immediately intervened to bring the levels back to normal. The City of Oldsmar has disabled the TeamViewer software now. They were using it to work remotely during the ongoing COVID-19 pandemic.

Ukraine faced a similar attack in 2016, reminding the world of the deep issues in industrial control system security. Such attacks have hit closer home as well with the Illinois water pump system hacked¹ by Russian hackers in 2011. So, industrial control (IC) and supervisory control and data acquisition (SCADA) systems have been ripe hacker targets for a while now. And they’ve been equally ignored for a while.

Attack Raises Thorny Questions

• Oldsmar is about 15 miles northwest of Tampa, Florida. The attack took place a couple of days before a widely anticipated Super Bowl game in Tampa. This raises the question: Is there more than meets the eye here? Hackers are often known to “test the waters.” The timing and location of the attack seem a bit too coincidental.
• Industrial control plants like the one at Oldsmar are part of what is known as “operational technology.” These should be ideally segmented and segregated out from typical IT networks. The twain just met thanks to the TeamViewer software that was used. Whether or not this was rushed through due to COVID-19, were the risks of using this software considered? Were alternative solutions were identified before going with a solution that has clearly failed in the past?
• Hackers use automated bots to scan for openings such as these. This is a well-known fact. These bots pick up such openings within hours, if not minutes. When opening up ports for TeamViewer, were these risks considered? What were the mitigating controls put in place to counteract those risks? Was the remote access setup implemented in a secure manner?
• The Assistant City Manager at Oldsmar confirmed that the system did require a password. That would suggest that the password was compromised either because it was too weak or that it was given away in a phishing attack, or worse, a much larger city-wide IT infrastructure-based attack. Have these considerations been closely examined?

Massachusetts Message

While Oldsmar officials haven’t provided further information yet, a State of Massachusetts advisory² goes into quite a significant amount of detail regarding the state of cybersecurity at the Oldsmar plant. Of note, that state advisory states (in italics):

• All computers used by water plant personnel were connected to the SCADA system and used the 32-bit version of the Windows 7 operating system. It is important to note that, as of mid-January 2020, Microsoft ended support for the Windows 7 operating system. So, computers at the Oldsmar plant were apparently running an operating system that was no longer receiving security updates, patches, or even technical support from Microsoft.
• Further, all computers shared the same password for remote access and appeared to be connected directly to the Internet without any type of firewall protection installed. The TeamViewer password being used for remote access was essentially one password that was shared between employees.

The Elephant In The Plant

Cities, counties, and local government organizations across the U.S. are standing and taking notice of the events at Oldsmar. Props to Oldsmar officials for openly providing a detailed account and also warning other government organizations in the process. But the smaller local government organizations have a much larger systemic problem at hand.

These organizations are often running on thin budgets. Cybersecurity is one of many competing priorities, creating a problem that is only compounded by understaffing. As a result, there are serious cybersecurity vulnerabilities, oversights, and issues that are unaddressed at the root of a fast-growing weed. Now add to this the fact that these organizations need to run some of the most critical industrial plants. Plants that control some of the most life-critical aspects of cities, towns, and villages such as water, power, gas, and oil, among others. There is an elephant in the plant.

Protection Plea

Organizations across the globe running IC/SCADA systems need to observe and act. These are systems that serve a significant purpose and even the smallest of compromise in them can have catastrophic consequences that, thankfully, the City of Oldsmar did not face.

The steps below are recommended to all organizations operating IC and SCADA systems. Would these steps require a budget? Well, some of these actually don’t and you would do good to have those in place already. Some of them do, but to that we say this – whether in-house or outsourced, there will have to be a resource that performs these steps and that resource will cost either time or money or both.

Immediate Steps

• Separate and segment out the IC/SCADA system network from your typical IT network and ensure that you have monitoring measures in place to ensure any violations of such segregation.
• Secure your remote access setup. Introduce two-factor authentication. At a minimum, if you’re using TeamViewer or an equivalent software, these typically offer the option to “confirm” a remote session. This would require an on-site operator to confirm such a session before handing over the remote control ability. It might seem like an extra step and possibly inconvenient, but after what has just happened should that really be a consideration?
• While remote access software is built to bypass firewall protections, not having a firewall at all in place is a really bad idea. You need the ability to control traffic flows and especially so when we’re talking about such critical infrastructure support systems. Make sure that you have a firewall in place with specific rules on what ports, protocols, and related traffic is permissible.
• Make sure that all software (including the operating systems themselves) are fully patched, up-to-date, and supported by the respective manufacturer. Security updates are critical due to the fact that new vulnerabilities and exploits surface on a daily basis. You should be, at least, protecting yourself against those that are known and “out there."

Medium-Term Steps

• Create a comprehensive IC/SCADA network system map. This map should depict and detail all the systems/devices in your IC/SCADA network as well as every connection that they make. Such documentation ensures that you fully know what you’re trying to protect, before you think about protecting it.
• Leverage the system map to perform an IC/SCADA system-wide risk assessment. The risk assessment should start by identifying the risks and threats posed to the system, prioritize them on the basis of what impact they would have on the confidentiality, integrity, and availability of your information, and then devise means and methods to counteract those risks and threats. Such a risk assessment should be performed at least once a year or each time there is a significant change in the technical infrastructure.
• Perform IC/SCADA specific penetration tests on a periodic basis. These penetration tests are very different from the way typical network penetration tests are conducted. Don’t let network penetration tests provide you a false sense of security.
• Perform comprehensive security reviews of the IC/SCADA network. All technical infrastructure that the IC/SCADA systems are connected to must be brought into the scope of such a review. The infrastructure as a whole and as well as each individual system/device within it should then be reviewed at a configuration level to ensure that each configuration, down to the very micro-level, is configured in adherence with cybersecurity best practices. An annual comprehensive security review is recommended.
• Provide comprehensive cybersecurity awareness training to all your employees. Such training should typically begin with social engineering assessments that aim to test the existing cybersecurity awareness levels of your employees. Based on these test results, you should look into providing ongoing cybersecurity awareness training to everyone, right from top management all the way to the employees.

References

1. https://www.infosecurity-magazine.com/news/russian-hackers-behind-first-successful-us-scada/
2. https://www.mass.gov/service-details/cybersecurity-advisory-for-public-water-suppliers