What I Have Learned from Assessing OT Networks

1. Identifying security risks

Looking at their OT network, customers always ask us: “Now, what do you think are our security risks?”

With many years of power grid cybersecurity knowledge under our belt and 20 years of experience in power utility automation and SCADA networks, we still had difficulties answering this question.

To find satisfying answers, we spent three years (2017-2020) to evaluate power plants and substations across all continents, from Switzerland over India and the United States to Namibia. The experience in cybersecurity and functional monitoring we gathered aided us in the development of a new approach to security in the OT environment – because protecting power utility automation networks is not just about cybersecurity alone.

Functional security and a continuous operability play a vital part in the overall security concept as well. Only through the combined coverage of all the factors which are involved in a correctly operating SCADA and Substation Automation System (SAS) can a general security concept be created and, more importantly, sustained.

Today, we proudly call StationGuard our product. It is an Intrusion Detection System (IDS) which not only empowers us to uphold the cybersecurity of SASs and control centers, but also to assess the security of the OT network with an unprecedented level of accuracy and usability.

In the following, I will explain

why we provide security assessments for control centers, power plants, and substation networks,
why we offer it for free, and
what some of my most interesting findings are.

2. Security assessment for the OT network

Before adding StationGuard to our repertoire, OMICRON engineers have tested and assessed many SASs, power plant, and control center networks: We performed penetration tests, helped develop secure network architectures, supported the writing of secure OT engineering procedures for substations, and performed risk assessments in power plants, to name just a few. The knowledge we gathered during this time, we used to develop our StationGuard IDS.

With its help, security assessments are simple, easily understandable, and uncomplicated – the most important problems for the cybersecurity of an OT environment, which regularly converges with the IT network of the utility, can be thoroughly revealed.

At the end of 2020, I was contacted by a colleague of mine who asked me to create a security assessment report for one of our customers using StationGuard. I received engineering files from the customer and was familiarized with the utility’s network architecture. These documents always constitute the conventional basis for a security assessment, before we continue our assessment on-site.

At the end of our assessment period, we presented our findings to our customers. In our security assessments, we usually deliver:

an exhaustive asset inventory of all devices communicating in the networks, >
a list of “unusual” services seen on the network, and
functional issues that we have found in the automation or SCADA system.

The data of my first security assessment proved to be highly unexpected to substation engineers and IT specialists alike. Among the risks our assessment brought to light were multiple unnoticed external connections, unexpected devices in the network, outdated firmware, unsuccessful RTU operations, configuration errors, and issues with the network and its redundancy protocol (RSTP).

In the years since, my colleagues and I conducted (and improved on) many security assessments around the world, not only for substations but also for power plants and control centers, including utilities with IEC 61850, IEC 60870-5-104, DNP3, Modbus TCP/IP, and many other IT protocols.

The findings of our security assessments were always highly interesting and, sometimes, alarming.

3. Findings of our security assessments

IEDs (Intelligent Electronic Devices), which my colleagues and I jokingly refer to as “emotional assets,” can react erroneously to unexpected queries from RTUs, configuration errors, and communication issues.

In our security assessments, we usually find several security risks in the plant network. Here are some of the most frequent ones:

undocumented external connections accessing IEDs and switches directly,
outdated firmware with known vulnerabilities,
unused services, and
unauthorized access.

Usually, these security problems are fostered by functional problems, such as:

configuration issues in IEDs, RTUs, and network switches,
time synchronization failures, and
network redundancy issues.

In my experience, most control centers and plant networks run smoothly after commission, but face problems after a few years or even months. There can be underlying problems, even in an apparently well-functioning automation or SCADA system.

For example, we have seen networks which reconfigure themselves every 10 seconds because of issues in the Rapid Spanning Tree redundancy configuration. This can cause major problems during a fault in the grid, when more bandwidth is needed. Down the line, these problems can cause errors, and they represent a risk of failures and cyber attacks.

In the following, I will name some frequently occurring risks that I have encountered in recent years.

3.1. Unused Services

Just by looking at the user interface of a PC, IED, or RTU you can never know what communication is occurring in the network – until you monitor the network, that is. Open/Unused services offer a disproportionate increase of opportunities for hackers to attack your automation or SCADA system. Thankfully, we can easily detect these unused services communicating in the network:

Here are some common unused services we found:

IPv6: Mostly activated on PCs, but sometimes also on IEDs. IPv6 was never actually used, but it provides several attack vectors in the network.
Windows file sharing: File sharing service was always activated on PCs and Windows-based RTUs and Gateways, but not used.
Software licensing services: These run with administrator permissions on PCs and Windows-based gateways, but don’t make sense in an isolated plant network. Yet, they are preinstalled with engineering tools and some HMI software. There have been several critical vulnerabilities known for this service.
CoDeSys Gateway Server, a service for the PLC program development environment active 24/7 on critical IEDs in power plants and substations. There have been several critical vulnerabilities known for this service, too.
PTPv2: It is enabled by default on some industrial switches, even if it is never used.

Simply turning off these services will decrease the number of cyber risks of your assets.

3.2. External connections

Power plants and control centers with remote connections from the corporate IT network always have the highest risk of a cyberattack. Let’s consider the example of a substation network in South America, which I assessed in 2021.

In this security assessment, we captured the activity of multiple clients with external IP addresses who were using the web services of the IEDs and switches in the substation. This utility allowed their engineers to connect and configure IEDs from home using a remote connection (VPN tunnels).

One finding that concerned the IT security officers was an IP and a MAC address which weren’t recognized or documented by anyone in their team. Eventually, we were able to track the IP address and find where this connection originated from and blocked its access to the system.

Even with all modern security measures that the substation offered, such as VPN Tunnels, Firewalls, RBAC etc., a considerable security risk remained: Besides architectural concerns, there were simply too many connections and even undocumented ones. We made them visible through our security assessment and the utility overhauled their connections.

3.3. Outdated firmware with known vulnerabilities

In EVERY security assessment, I come across outdated firmware versions in the utilities.

As part of our assessment, we provide a passively discovered asset inventory to the utility engineers and IT experts. By importing engineering files, such as IEC 61850 SCL project files and CSV inventory lists, we augment the asset information with more details, like the software version, HW configuration and serial numbers. This asset inventory is then a good base for performing vulnerability and risk analysis.

3.4. SCADA communication configuration errors

Misconfiguration of RTU and SCADA devices can result in critical events on-site which aren’t transmitted to the control center and slow down communication.

In a European substation we visited, for example, there were configuration errors in the MMS Report configuration. Reports were configured to be sent to the wrong client IP address.

After resolving these configuration errors, the communication speed of the IEDs improved verifiably. The issue was not a security threat by itself, but the impeded communication speed posed an operational risk and would have obstructed response processes.

3.5. Detecting configuration changes

There is more than one way to detect configuration changes on important assets in the OT environment.

In an US substation, we detected a misconfiguration of GOOSE messages. This problem occurred because the devices were configured by two different people. In turn, this lack of communication between engineers caused communication problems between these OT devices.

We discovered that certain remote command activities in the substation didn’t function properly due to interlocking conditions that were not valid. This means, they wouldn’t have been able to operate their switchgears remotely in urgent cases. What frightened them was the fact that they didn’t even know the problem existed, until we found it during a security assessment.

This example was truly eye-opening for me as well. It showed me that such small issues in GOOSE communication could cause bigger troubles for the plant.

Therefore, to further our knowledge, and get in contact with other employees from power plants, substations, and control centers, we still offer these basic security assessments for free.

4. Benefits of our free security assessment

Whether you are a cybersecurity officer, a control and protection engineer, a network engineer, a manager, a system integrator, or someone else working at a control center, a power plant, or a substation, I would like to present to you further reasons why such a security assessment is definitely worth your while.

4.1. Third-party users in your OT network

It is no secret that many third-party companies are heavily involved in the commissioning (and maintenance) of SASs and other OT networks. As mentioned above, having different engineering groups on-site can create challenges when it comes to keeping the OT system sustainable.

I have heard many complaints about third-parties which misconfigure systems and leave connections open. A well-operating utility depends on managing this chaos and, at the same time, cares for a continuous operability of the system. Often, the management of these risk factors is time consuming, costly, and a waste of resources.

Our security assessment effectively identifies assets and risks, and aids utility operations dealing with third party companies – to avoid long term costs.

4.2. A simple way for IT officers to understand the OT world

Classic IT IDS solutions unfortunately cannot evaluate OT protocols and events. Learning-based IDS approaches require a lot of OT knowledge to initially assess which activity is allowed and which should be forbidden.

In addition, IT security officers usually don’t know the OT environment and its working principles to make these decisions. This proves to be challenging when it comes to managing OT incidents and introducing security processes in OT.

Our product StationGuard offers the solution: it provides understandable data for IT and OT engineers. Our security assessment with StationGuard can merge these two different worlds and creates a foundation for common understanding.

4.3. Have a look at the patterns of your network

As mentioned before, OT networks are full of surprises and complexities.

It is always better to take counter measures today for potential issues that could arise tomorrow: maybe we find the reason for a chronicle issue at your network? Maybe there is an undiscovered problem waiting to be found?

Many things are yet to be discovered, but one thing is for sure: Any risk that could violate your operation in a time critical environment will be costly.

5. Request a security assessment for your substation

We provide Free Security Assessment worldwide for our potential customers!

Are you an IT officer/manager who wants to know more about the OT part of your organization? Do you wish to learn about the security risks in your OT network?

Are you a SAS Engineer/Manager and want to find (or solve) your functional problems, or verify your network communication?

In any case, request a security assessment from us!

To request a security assessment, please send me a message with the subject “SG Free Assessment” under the following email address: info@omicronenergy.com (Due to our privacy policies, I may not write my personal email address here. However, please be assured that all emails will be forwarded to me directly.)

Thank you for reading my report!

Best wishes,
Ozan Dayanc (OT Security Engineer)