Anyone who has spent a significant amount of time in any U.S. state where college football is popular, has likely seen a “house divided” bumper sticker or license plate cover, with contrasting university logos. Many of us (and our friends and families) enjoy spirited rivalries (Roll Tide vs. War Eagle, The Egg Bowl, Bedlam, The Backyard Brawl, “The Game”). But we’re fundamentally not all that different and we have the same goals – we want our team or program to win and to be successful.
Spending time in the field or on the manufacturing shop floor coordinating cybersecurity priorities between Information Technology (IT) and Operational Technology (OT) is often reminiscent of a college rivalry. However, in this instance, it’s important to keep in perspective that in an organization with OT systems and technology, everyone plays for the same team. It’s also critical to recognize that implementation and ongoing management of cybersecurity measures in IT vs OT often look quite different. The list below is far from comprehensive but, based on our experience, represents some of the more common and impactful differences or challenges that organizations may encounter when attempting to advance the current state of cybersecurity in the OT environment.
Patching/outdated systems
Applying security patches is much less straightforward in an OT environment. Operating system patches (e.g., for Windows devices) often require a reboot, which may be difficult and disruptive to schedule in a production environment. Additionally, Original Equipment Manufacturers (OEMs) such as Emerson, Honeywell or Wonderware/AVEVA test Windows or Linux patches in their lab or development environments, and then on a periodic (monthly or quarterly) basis will release a list of patches that are approved for installation on their system. Given these two factors alone, it is common to see patch levels on key OT systems that are months or even years behind the current release, potentially impacted by critical or exploitable vulnerabilities.
In addition to the operating system layer, it’s important to consider applications and software (e.g., the Industrial Control System, or “ICS”) running in the OT environment. Countless times we’ve witnessed legacy and unsupported operating systems such as Windows XP or Windows Server 2008 that have not been replaced due to application dependencies. For example, there may be a program developed by a vendor 20+ years ago that isn’t even in business anymore, so the software was never updated to run on modern and supported operating systems. Running systems that have reached end-of-life (EOL) or end-of-support (EOS) isn’t ideal, but have all of the potential impacts of replacing those OT devices been considered? Has a cost-benefit analysis been conducted to determine the cost of updating or replacing EOL devices, which if the OEM is no longer in business might require the replacement of additional hardware, equipment or machinery that could carry a seven-figure price tag? Our recent Global Technology Executive Survey indicates these issues are widespread and not isolated.
Network monitoring and visibility
Professionals working in or around cybersecurity likely have heard an expression to the effect of, “you can’t protect what you don’t know about.” An incomplete and unreliable asset inventory (of network-connected devices) is a common pain point in OT. Traditional network scanning or discovery tools are minimally informative at best, and in many networks (where aged, legacy infrastructure is present and/or bandwidth is limited), could render systems unavailable and disrupt operations. Organizations that grow and expand through mergers and acquisitions may particularly struggle with maintaining a full picture of network-connected devices in the field, plant or manufacturing site.
Enumerating security vulnerabilities and potential security events in OT environments also poses unique challenges. Network vulnerability scanners have the potential to negatively impact system availability and may not be able to identify insecurities on OT devices (such as Programmable Logic Controllers (PLCs)). Security tools such as anti-malware or endpoint protection are not always able to be widely distributed in OT, as OEMs may only support select endpoint detection and response (EDR) tools, which could differ from what the enterprise has selected for IT networks/assets. If a Security Operations Center (SOC) is in place and logs from OT devices/networks are being ingested, do we have confidence that we’re getting data from all critical systems or devices? Have our security analysts been trained on how to recognize indicators of compromise in OT and are clear on who from the operations side of the business (such as a plant manager or shift supervisor) should be engaged to investigate a potential security incident?
Identity and access management
Many controls that are considered basic cybersecurity hygiene in IT networks may also face roadblocks in OT environments. Related to identity and access management, some systems and applications cannot support lengthy or complex passwords that align with the organizational standard. Certain devices don’t support passwords at all! Knowledge of data such as IP address, open port, and software required to communicate may allow a user to connect to and even manage or make changes on critical automation devices without authentication. The ICS or other management programs may not have the ability to create accounts that can be assigned to individual users, and the default account (e.g., “admin”) must be leveraged by all users of the system. Lastly, it is common to see processes, programs and applications on OT systems running with local administration privileges, compounding the risks outlined above.
Takeaways
What can be done to bridge this divide and improve the security of our OT networks? I realize I’m not revealing a state secret here, but it starts with empathy and transparent communication. I’ve lost count of the number of instances I’ve encountered where the IT or security team discovers a device that they didn’t procure or manage and the OT team is told “you can’t have that server.” No solution or alternative is offered, nor is effort dedicated to understanding the business or operational use case for why that server needed to be deployed, the response is just “no.” On the other hand, the OT personnel should be more proactive in engaging IT/security to develop solutions to challenges or problems jointly. The path of least resistance may not be the most secure one and deploying it without evaluation from in-house or external/third-party security experts could introduce risks that not only lead to operational disruption and lost revenue but may also impact the health and safety of employees and the environment. There are alternative paths available to architect a solution that meets the needs of the business that also factors in cybersecurity. Reaching the ideal destination often requires multiple navigators from both IT and OT.
To address challenges in patch management, ensure there is a defined cadence with OEMs to receive and apply security updates (prioritizing critical updates such as those with exploit code publicly available) on a periodic basis. For EOL systems that cannot be upgraded or replaced, explore options to isolate and restrict traffic to/from those systems (for example, using a firewall or access control lists on a network switch).
Consider implementation of an asset discovery and threat detection solution designed for OT networks (device discovery and enumeration is conducted passively). Leverage the results from the discovery solution to identify and document the systems that are critical to site operations and prioritize those devices in your security monitoring strategy. Additionally, organizations may consider standing up a test network reflective of the production environment, to enable safe methods to evaluate patches and identify potential vulnerabilities.
For Identity and Access Management, enforce complex passwords and create unique IDs/logons where possible. Limit the provisioning of administrative or privileged access as feasible. Enforce multifactor authentication if supported on all remote access connections to the OT environment.
When it’s possible to get IT and OT personnel playing for the same team, the organization increases its odds of scoring a secure operational environment.
Read the results of our new Global IT Executive Survey: The Innovation vs. Technical Debt Tug-of-War.
To learn more about our security consulting services, contact us.
