Posted by Michael Applebaum in Critical Infrastructure, Cybersecurity, In the Industry, Threat Management

While some have claimed the warnings about SCADA system vulnerabilities are merely exaggerations and vendor FUD, this talk should be put to rest with the news that a US utility has suffered real physical damage from a cyber attack.

Image attribution: http://bit.ly/rqYb85 under http://bit.ly/uJqvTC

As widely reported, a water pump at a utility in Springfield, Illinois was burned out by a remote attacker repeatedly turning it on and off over a period of months.  Certainly not as dramatic as Stuxnet, but effective nonetheless.

How did it happen?  The attacker allegedly infiltrated the network of the vendor whose software controlled the SCADA systems, including the water pump.  Through this access, the attacker is believed to have gained customer user names and passwords, including those for the Springfield utility, which enabled remote access to the systems.

Reactions to the news range from indifference (it’s just a water pump; there was no disruption of service due to redundant systems; wake me when I should care) to alarm sounding (the vulnerabilities are real; the potential impact significant; the urgency high).  At Q1 Labs, our view (and that of our customers!) is that critical infrastructure providers, their vendors and government authorities need to take these risks seriously.

What can we learn from this attack?  Here are five lessons:

1. Information security is just as important as physical security. It is obvious now that cyber vulnerabilities exist, can be exploited, and can cause physical damage.  But too often information security best practices are ignored.  For example, why are SCADA systems even connected to the public Internet in the first place?  ICS-CERT has reportedly “received a number of reports from multiple independent security researchers who have employed the SHODAN search engine to discover internet-facing SCADA systems ‘using potentially insecure mechanisms for authentication and authorization.’”  This should never occur, but it happens through ignorance of security best practices, limited budgets and good old-fashioned manual error.

Defense in depth approaches should be adopted, and best practices understood and applied.  For example, many organizations assume they’re secure because they’ve deployed traditional defenses such as firewalls, antivirus, and identity and access management solutions.  This attack shows that these traditional approaches are no longer sufficient; you also need continuous monitoring in order to quickly spot unusual or suspicious activities, because cyber criminals might be using legitimate credentials to access your critical systems.

Utilities and other critical infrastructure providers have no excuses, and there is no “A for effort.”  This is not only a national security issue, but also a business continuity and viability issue.  If you fail your customers catastrophically, you will find yourself out of business.

2. Rapid detection matters. The breach is suspected to have occurred in September, but was not discovered until November 8.  During that time, security researcher Joe Weiss reports, “minor glitches were observed in remote access to the SCADA system for 2-3 months before it was identified as a cyber attack.”

The reason operators typically let “glitches” go by for months is they don’t have an easy way to mine network data. If the utility had centralized logging, data normalization, and simplified searching and data pivoting, its operators would have been able to analyze the data faster, and identify and stop the attack.  Instead of wondering how to find the root cause, they could have used a Security Intelligence solution to troubleshoot and explore the forensic data with a single, easy-to-use console.

There were also obvious clues that should have tipped off operators to a potential breach, such as the systems being accessed by Russian IP addresses.  A modern SIEM solution would have automatically alerted on anomalous network activity, such as access from outside the US.

3. Assume you are already breached. Although rapid detection is vital for responding to new attacks, you should also assume you have already been breached and are now under covert surveillance or attack.  Operation Shady Rat showed that US federal agencies, energy providers and other large sophisticated organizations – let alone smaller businesses – can remain unaware of attacks over a period of years.

Would you know if you were already compromised?  Stop wondering, and get to work finding the breaches that likely already took place.

4. Aggressive information sharing must become the norm. Besides highlighting weaknesses in security defenses and monitoring practices, this story also demonstrates the industry’s opportunity for improvement in how it responds to a cyber attack.  Although the Illinois Statewide Terrorism and Intelligence Center identified the incident, Weiss points out that “the incident has not been disclosed by the Water Information Sharing and Analysis Center, the Department of Homeland Security’s Daily unclassified report, by the DHS Industrial Control System-Cyber Emergency Response Team or other government and industry security groups.”  Thus other water utilities remained unaware of the attack, according to Weiss.

5. The full impact of this breach is unknown. Without falling into hyperbole, one has to consider that the known damage may be just the tip of the iceberg with this exploit.  Since “many industrial control systems rely on passwords that are hard-coded, making it difficult to change stolen passcodes without causing serious problems,” are other water utilities – or even nuclear power utilities – exposed to this compromise?  Weiss notes that “If this is a [big software vendor], this could be so ugly, because a biggie would have not only systems in water utilities but a biggie could even be [used] in nukes.”

Regardless of whether this incident proves to be a minor blip or the start of a series of attacks on the SCADA vendor’s customers, the lessons it presents are clear.  Aggressively protect your critical infrastructure.  Focus on both parts of the Security Intelligence timeline:  pre-exploit (vulnerability and configuration management) and post-exploit (threat detection, investigation and remediation).  Learn from the best practices of California ISO and other critical infrastructure providers that have adopted Security Intelligence.

Update 11/23/11:

According to reports here, here, here and elsewhere, the Department of Homeland Security and FBI have announced that the destruction of the Springfield, Illinois water pump was not due to cyber hacking.  The DHS announcement reads in part, “After detailed analysis, DHS and the FBI have found no evidence of a cyber intrusion into the SCADA system of the Curran-Gardner Public Water District in Springfield, Illinois.  There is no evidence to support claims made in the initial Fusion Center report — which was based on raw, unconfirmed data and subsequently leaked to the media — that any credentials were stolen, or that the vendor was involved in any malicious activity that led to a pump failure at the water plant. In addition, DHS and FBI have concluded that there was no malicious traffic from Russia or any foreign entities, as previously reported.”

Questions remain about why the Illinois terrorism center reported this as an attack.  But either way, the lessons shared here hold true.  SCADA system vulnerabilities do exist, can be exploited, and can cause physical damage.  The time to strengthen your pre-exploit and post-exploit security capabilities is now.

Update 12/16/11:

This story continues to play out in the headlines, with the FBI’s Cyber Division acknowledging that hackers recently accessed the infrastructure of three cities through SCADA systems.  As this post notes, the good news is that the FBI’s budget for cyber defense will likely rise over the coming year.  The bad news is that although the Cyber Division’s deputy assistant director “expects” the division to double in size within 12 to 18 months, the FBI’s budget request for cyber defense was only 12% higher for the 2012 fiscal year.  How much impact will a 12% increase have?  Your guess is as good as mine.

What is clear is that the vulnerabilities surrounding SCADA systems are real, and this issue will only become more significant over time.  Consider that my first security prediction for 2012.