Introduction
On February 8, 2021, there was a press conference stating that a municipal water system in Florida was the target of a malicious and unlawful cyber-intrusion into their water treatment plant control system. Many of our clients are concerned about the safety and security of their systems. As a national Water/Wastewater Systems Integrator, we are not surprised that an event like this occurred. System operators are increasingly seeking the benefits of having remote access for support of their plant control system, mobile access to their SCADA (Supervisor Control and Data Acquisition) or HMI (Human Machine Interface) system, and the data gathering flexibility from IIoT (industrial internet of things) devices which can put an organization at risk if certain best practices are not followed.
Background
What we have learned through the media is that unknown person(s) infiltrated the plant control system via the TeamViewer remote control software. TeamViewer uses cloud-based technology for online remote support of computers and servers. The plant operator noticed that the intruder was using the computer mouse to adjust the setpoints of the sodium hydroxide (lye) chemical feed, which the plant operator immediately reversed as well as preventing the remote access session. Other water quality safeguards were in place to prevent any potential danger to the public. More details of the announcement can be found in a YouTube Video or local news article.
Concerns
Water system operators are professionals who take water quality and water safety very seriously. Technology and cyber-security is ever changing, making it challenging to stay on top of the possible security risks that could be introduced. There has been an increasing number in attacks on our critical infrastructure. Some operators are confident that their systems are secure and up-to-date; however, they can quickly fall behind. These concerns are valid and should be addressed.
Benefits of Remote Access
“Why even allow remote access to a water treatment plant control system?” “Why not have an air-gapped system, where there is no connection to the Internet? Would this keep the bad guys out?”
This is a good question that we encounter from time to time. Remote access is an efficient and cost-effective way to perform maintenance and provide support. Air-gapped systems could potentially keep out “bad guys” but does require regular maintenance to keep secure. Security threats can still attack your system through USB Drives, local networking, and malicious onsite personnel. Instead of removing remote access functionality completely, a recommended option is to implement and maintain a remote access solution that follows the recommended security best practices and procedures.
Who can help?
Although there may be some reactionary steps that can be taken to address the immediate concerns about avoiding a similar security breach, the solution isn’t as simple as pulling the plug to the Internet, disabling your remote access software, or enabling two-factor authentication. A water system operator should begin working with a qualified systems integrator to assist with the implementation of recommended security best practices. Be sure that the systems integrator employs certified professionals, especially those with designated cyber security certifications such as a Global Industrial Cyber Security Professional (GICSP) or a Certified Information System Security Professional (CISSP). A system integrator with this expertise will help you to address your security concerns and keep your plant control automation running as designed.
Recommended Best Practices
To help protect from similar activity as witnessed in Florida, the following best practices should be considered, if not made mandatory:
- Use non-trivial authentication methods, such as implementing two-factor based authentication
- Prevent the use of weak passwords and require strong, more complex passwords
- Prevent the auto-acceptance of remote support sessions without any notification
- Implement SCADA/HMI application security with designated individual usernames and passwords
- Review systems for potentially stolen or lost credentials
- Update the software and implement/enforce two-factor authentication for any cloud-based remote access solution
- Consider implementing a secure Virtual Private Network (VPN) for all remote access
- Implement security awareness training program for plant operators
- Implement geo-blocking and other network access filtering to help prevent access from unauthorized people or places
- Encrypt network traffic
- Implement network segmentation
- Review internet accessible devices and evaluate if warranted
Each of these recommendations deserve their own space for more details and discussions, and they are not listed in order of importance or any other priority.
Additional Layers of Resilience
Additional steps can be taken to alert the staff in the event of a successful network comprise where the control system is now at risk.
- The chemical feed equipment in the water treatment process can be configured with independent feed rate monitoring and alarming, along with continuous water quality analyzers.
- Routine operator rounds can provide another layer of system monitoring.
- Operators should have the ability to switch a process or a system to local control and disable remote SCADA control, particularly in the event of a compromised SCADA/HMI system.
- Water system operators can implement a water distribution monitoring system that continuously checks water quality in real-time before it reaches the community, alerting the operators to a contamination of their water distribution system.
Long-Term Solutions
While certain best practices such as the ones listed above offer some short-term gains, there are long-term solutions that a water system operator can pursue to help keep their control system secure.
- Partner with a qualified systems integrator (Concentric Integration) and implement a managed support agreement where customized proactive and preventative maintenance can be routinely performed, which can help to keep software and hardware updated with the latest performance and security patches.
- Work with a water industry consulting engineer (Baxter & Woodman) to perform a Risk and Resilience Assessment (RRA), which is now required by the U.S. Environmental Protection Agency (USEPA) through the America’s Water Infrastructure Act (AWIA). The RRA evaluates and documents the risks of manmade and natural disasters of every component of your water system including supply, treatment, distribution, storage, as well as cybersecurity. Communities with a population served between 3,301 and 49,999 people are required to complete their RRA by June 30, 2021.
Follow-Up and Next Steps
We’re ready to assist! Contact Mike Klein at mklein@goconcentric.com or Carolyn Grieves at cgrieves@baxterwoodman.com to discuss how we can help keep your water system secure.