This episode features Jos Zenner, managing director and CTO at Welotec, a provider of rugged computing and communication platforms for DSOs and TSOs in the electrical sector. Jos and Klaus talk about the challenge to think and implement “security by design” on infrastructure level and from the very first step of planning. And we dive into the discussion on how OT security in distributed infrastructures could work as a platform solution to balance limited budgets with the need to secure thousands of secondary substations.
Listen on:
Transcript
Klaus Mochalski
Hello and welcome to a new episode of the Rhebo podcast OT Security Made Simple. With me here today is Jos Zenner. He is CTO of Welotec. We are meeting at E-world in Essen, one of the most important trade shows in Germany for the utility and energy sector. Before we start, Jos, maybe you introduce yourself. What are you doing and what is Welotec doing?
Jos Zenner
Perfect. Thank you very much, Klaus. My name is Jos. I'm Managing Director and CTO at Welotec. I'm coming from the product management and have been working in the energy sector for more than one decade. What we are doing as Welotec is delivering computing and communication platforms to DSOs (distribution network operators) and TSOs (transmission network operators) that are made for harsh environments, where applications can run within substations and secondary substations.
Klaus Mochalski
Maybe to give our listeners a bit of a background, Rhebo builds anomaly detection systems, intrusion detection systems, so mostly software that we deploy at electrical utilities from DSOs and TSOs to secure their critical networks, their OT infrastructure. While we today quite often deploy our own sensor hardware we are a software company. Our long term vision really is that this type of monitoring that we're offering today will become a function of the communications infrastructure.
That's really for us one of the key strategic facts for the corporation of Welotec that we see devices like the ones that Welotec is providing in harsh environments, in electrical substations, for instance, as a key delivery vehicle to deploy our software and the services around it. Which are the most critical environments where you deploy your solution today, where you see OT security as the biggest immediate challenge over the next months and few years?
Jos Zenner
There are plenty because we are in the most critical infrastructure. We have many customers in the high voltage direct current lines. We have applications in transmission grid substations. We have applications in distribution grid substations and of course, also in secondary substations. This is the typical playground where we install our appliances where you or we can run your software on.
Klaus Mochalski
High voltage and secondary substations?
Jos Zenner
Yes, and the highest voltage, which is the transmission grid lines which are transporting the energy from the the wind parks in the North Sea, for example, to Bavaria.
Klaus Mochalski
How do you see security awareness developing among your customers? Are they typically asking for security solutions? Or is it something that they still keep forgetting in the years past?
Jos Zenner
It's a little bit mixed. We work quite a lot with the large vendors and they are looking more and more into security, but still there are quite a lot of things they need to do. Then we work with transmission grid and distribution grid operators.
The transmission grid operators are quite good in IT and OT security. They are in distribution with sites, especially if you're looking at smaller sites. There's still a lot of things that need to be improved in the future.
Klaus Mochalski
If, for instance, transmission grid operators build new substations, do they build them with security in mind? Is it something that you observe? Because what we've seen is that security is still something that is put on top of existing infrastructure, and this is of course not ideal. Our hope would be that for every new project, every new substation that is being built, security is an integral part, a feature of the system itself. Is it something that you observe or is it still too early for this?
Jos Zenner
Well, at the moment, they still put it on top. So, for security they installed a firewall.
Klaus Mochalski
So, no security by design for substations?
Jos Zenner
Well, partly, I guess yes, but most of the designs that I've seen are not security by design.
Klaus Mochalski
From your perspective, what keeps them from doing this? There are lots of headlines. There have been utilities being hacked. We see simple attacks like ransomware attacks being successful, targets being hit. Why is it still that this is not an important part of the day-to-day business?
Jos Zenner
Well, I guess first of all, lack of understanding. So if you're looking at a substation, you are looking at OT guys. You are looking at people that are coming from conventional designs where there was not much networking infrastructure in the substations, minor connectivity. And now everything is changing. We see more and more IEC 61850 deployments within substations. We see more and more networking infrastructure. We see remote access. We also see ideas to have cloud connectivity.
Klaus Mochalski
You mean in the substation?
Jos Zenner
In the substation also, yes.
Klaus Mochalski
Okay.
Jos Zenner
Partly separated from the network, but still that data is going into the cloud. If you have voltage and current data, that's where you need to have big data analysis or asset monitoring, for example in transformers. This is something that can be done in a cloud environment much better than in some traditional data system. This is changing quite fast.
At the same time, you have, well, let's call it a bit of competition between OT and IT and misunderstanding between both parties. It's still not solved yet.
Klaus Mochalski
Well, it shouldn't be a competition, right? It should be cooperation.
Jos Zenner
It should be, yes. But the IT department and the OT department are not working that well together. I was talking to a TSO today, and to get some IT stuff from the IT department seems to be a very long process. This is why other ways are gone and done that are probably not the perfect solution.
Klaus Mochalski
I think a lot has been said about this typical problem of the challenges in cooperation between IT and OT departments. The IT security departments, they have the skilled people, but they are lacking the understanding of the special requirements in the OT infrastructure. They don't feel themselves understood, and this goes both ways. I don't want to spend too much time discussing that. But from what you see, like in the previous episodes, we also talked about this challenge of IT/OT security consolidation.
From a theoretical perspective, the ideal would be having a centralized security dashboard, a system monitoring all of your security incidents in the entire organization, no matter if it's IT or OT, because the boundaries will become more ambiguous and less clear. And so the consolidation is something that ultimately will happen. But today, we don't really see this working well in practice. What would you recommend companies looking to solve OT and IT security challenges?
Jos Zenner
Security is a topic that is mainly driven by IT. It's understood by OT, they understand everything. They know that they need it, but it's mainly driven by IT.
At the same time, if you're looking at security solutions, we need to look at what value does it bring for OT guys? If you put them on the same system as IT guys, they probably don't understand, so they lose control. You need to create value for them, not only in security incidents, but also understanding the network traffic, understanding what they build there. So monitoring the network and give them a tool at hand which not only solves security issues, but helps them to understand the network.
Klaus Mochalski
Okay, so you're saying a key to get the OT people on board – and I'm not saying OT security people because mostly there's no real classic IT security staff working in OT, not yet -– is to give them something beyond security because security is not a value for them in itself. They just take it for granted that they get not hit by a cyber incident. When it happens, it's really bad but if it doesn't happen, then there's no value for the OT security solution.
The benefit, if I understand correctly, is that we should also try to give them a tool to monitor their day to day operations of their infrastructure? Simple things like the health of their communications infrastructure that they need for their day-to-day operations. Is this correct?
Jos Zenner
This is correct. The reason behind this is that if you're looking at security incidents, they are probably not always from external attackers, but also based on the configuration of systems, network issues, and these things. Network analysis tools and next generation firewalls can help you solve those challenges.
Klaus Mochalski
Exactly. Because they are watching data that is also relevant not only from a secure perspective, but also from an incident perspective.
Let's come back to the beginning of our discussion when we looked at the different, let's call it, units that a typical grid operator operates. These are the high voltage substations where we understand they have significant infrastructure. They are modernizing that with protocols like IEC 61850, which also provide more exposure to network attacks.
But let's look at the secondary substations. They are usually much simpler in their setup. You have many more of them, so monitoring them could be a challenge. First of all, how do you see the security challenge or the security or the attack surface of these secondary substations today, the ones deployed in the field?
Jos Zenner
Well, most of the substations that are deployed in the field today are very conventional, mainly copper. But most of the DSOs, especially the larger ones in Europe, are installing more and more intelligent devices inside of the substations. So the conventional secondary substations become smart substations or intelligent secondary substations.
This means, like RTUs are installed, measurement devices for the low voltage side or for the medium voltage side are installed. Additional equipment for power quality is installed. And also connectivity to, for example, private households or power lines for smart metering and smart perspectives is built up. So you are creating some new tech surface within the secondary systems to have more value from the automation of monitoring to get sensor data. But at the same time, you have the cyber risk.
Klaus Mochalski
Right. So you can really differentiate from a security perspective. The legacy secondary substations are pretty secure. They are hard to detect from outside because they simply don't have any external channels. But if they are becoming smart substations with the things you mentioned, like monitoring services, RTUs, power quality monitoring, then they become more susceptible to cyber attacks.
Jos Zenner
Definitely, yes. One thing is also very important to look at: The secondary substations are very distributed.
Klaus Mochalski
How many are we talking about? Let's say a typical DSO serving 1 million households, how many substations would they operate, approximately?
Jos Zenner
Well, around one or two hundred households are supplied with one secondary substation.
Klaus Mochalski
Doing the math, this is a couple of thousands for 1 million.
Jos Zenner
A couple of thousands, yes.
Klaus Mochalski
Quite a lot actually.
Jos Zenner
They actually need a more. Because we need more to support the heat pumps as well as mobility.
Klaus Mochalski
So the substations become more distributed and a higher quality of substations. Looking at the numbers, thousands or even 10,000s, it's pretty clear that we can't go to all of them and install a piece of security monitoring hardware. This has been our business case for the past years. Certainly at larger systems, but as these smaller, more distributed assets need to be secured as well. From your perspective, what's your recommendation? What's the best practice for securing these substations that you would give to the operators looking to secure their upcoming smart grids?
Jos Zenner
Well, the best would be not to connect them.
Klaus Mochalski
Really? This is coming from you?
Jos Zenner
But you need to connect them. That's really the demand.
Klaus Mochalski
I think there's no doubt about this. We can't turn back time.
Jos Zenner
You need to make them smart. To secure them, you need to think about security from the first step on. You need to think about security by design.
Klaus Mochalski
Security by design means, you shouldn’t put security in place after the substation has been deployed, but do it as part of the planning stage when you start designing your substations?
Jos Zenner
Doing it afterwards would be too expensive, and would probably not be the best solution. They need to consider how to deal with security – and not only with the security requirements that you have today, but also with the security requirements that you might have tomorrow. Think about the design of a substation, about the networking. But you also still have serial protocols, so how do you wire everything and where to install firewalls, where to install monitoring capabilities.
This would also be, in our discussion, quite interesting, to understand where do we have to monitor RTUs, where do we have to monitor the traffic to get most out of the solution.
Klaus Mochalski
I mentioned at the beginning of this episode that our vision is that in the not too far future, the specific security function that we provide – monitoring of communications, incident and anomaly detection – will be a service, a feature that can be enabled running in the existing infrastructure. There are additional security controls like firewalls you mentioned and there may be others.
Do you share this idea of having this as a function that you can enable by a click of a button and not on a single device, but on 10,000 of devices, or is it different or more complicated?
Jos Zenner
It should be that simple. If you're talking about secondary substations, and when I'm talking to DSOs, many of them are thinking about putting edge computing into secondary substations. They're not doing that for cybersecurity, they do it for automation purposes.
So for example, to load a container with software with RTU capabilities, or another container with protocol conversion capabilities, or another container to monitor the low voltage side of the grid or medium voltage side of the grid, and probably also a container to control something. In addition to that, they are looking into cybersecurity, of course. This could deploy as a container as well. So it's an additional function.
So if you have a platform inside of a secondary substation where you can run a container for automation monitoring purposes, you can just install a firewall container, you can install an IDS container and monitor and add additional security.
Klaus Mochalski
This really sounds simple. Basically, we would piggyback on the existing infrastructure, and edge computing devices. There may be an app store concept where we just reload certain functionalities that we need, including security functionality.
Why is this not yet happening on a large scale? Or is it? Or when is this due to come and be part of the regular planning exercise of new secondary substations?
Jos Zenner
Well, it's not yet happening, but I think we are very close to it. We see tenders and ideas about this type of edge computing in secondary substations ramping up and there will be a road map starting probably next year. First approaches are done already. Of course, at first this is looking at the automation side because they need to automate.
But I believe we need to think about the cyber security side now to understand, if we can deploy, for example, one of the Rhebo containers for monitoring purposes? Or did we do something wrong in the design at the beginning, or did the DSO did something wrong in the design, that doesn't make it possible to deploy a container like this. Or we need to add additional hardware, for example, a switch with the monitoring part. We would like to avoid that. So I'm concentrating all the traffic on the central gateway and monitoring the single interfaces there.
Klaus Mochalski
Okay, so it all sounds very simple. If you start early enough in your planning process and design process to include these kinds of security functions. So what could be the concerns of customers? What do you observe?
Jos Zenner
Well, first of all, there is the issue of budget. They have budget for automation and cyber security is just part of it. They are going to different vendors and they want to have a secure solution. Security means that you regularly give me security updates. But if you are looking at security, we cannot only look at single devices, we need to look at the system itself.
Klaus Mochalski
That's an interesting idea you bring up here. We also quite often have that discussion. I have the feeling that many operators are mixing up product security with infrastructure security. They think we have purchased secure products so this makes their infrastructure secure, too. Sometimes they don't understand that it's not the same thing. If you have 10 products and they are all certified secure by the vendor and you connect them all together, you don't automatically get a secure infrastructure, but they assume so.
What needs to happen to educate operators about things that need to be done if they connect secure products from different vendors?
Jos Zenner
Well, let's start at a different point. As I mentioned before, we have a distributed infrastructure, which means it's very difficult to protect them physically. You can just access the things and this is not too complex. If you have access to multiple different devices with networking capabilities, they can access the whole system. If you have that possibility, you are not looking at a single device, but something that you might install in between two devices or so, which is monitoring traffic and so on.
You need to understand the security of the whole system. You need to understand the communication between the single devices. You also need to understand the communication from the secondary substation to the data cloud, so that you also monitor that traffic to understand what happens in your network and also to detect security incidents there.
Klaus Mochalski
Maybe it's important for the device manufacturers not just to deliver secure devices, but also providing their customers who integrate these devices into a complex infrastructure to give them security controls which enable a simple and affordable – I don't want to say cheap but affordable – security monitoring of these infrastructures, no matter how secure the product is, as claimed by the vendor. Would this be something?
Do we need some vendor alliance that we have a stand up for a security control that is provided by all critical infrastructure component manufacturers? By the way, I don't believe that is ever going to happen, but is this a theoretically good idea?
Jos Zenner
Well, maybe it doesn't need to be a vendor alliance, maybe a reference design could help already. So if you have a reference design on how to monitor, where to monitor. And this is done. If you're looking into cyber security regulations and certifications, for example, like IEC 62443, you see that there are some recommendations for cyber security and monitoring is also required.
Klaus Mochalski
Okay. So the operators seeking to build new infrastructures put these into their tenders as minimum requirements? They say: Well, we have certain security blueprints, certain standards and we need devices and systems that we can add here to these standards and that can be monitored in a multi-vendor environment?
Jos Zenner
Well, yes, but usually the tender is for multiple parts of the system, not the full system. This is something where you need to have an additional approach where you are looking at the full system and not only on the different parts that are tendered. Because if you think about cybersecurity, if you have one part of the system that is not fully secured, the whole system becomes insecure.
Klaus Mochalski
That's an interesting point. It seems like today's tendering process for critical infrastructures may limit the scope of security for each given tender because a tender may only look at a particular type of substation or maybe only a few substations. There may be security requirements for, say, five new substations, but it doesn't incorporate the central control system or the secondary substations.
There needs to be a more holistic approach to security, but this is probably not compatible with the tendering guidelines as we see them today. What could be a solution here?
Jos Zenner
Well, you cannot solve it from a tendering perspective. What you need to create is some platform and this will be the new standard. We can run applications from third parties and just install containers so that the DSO can add additional applications over the lifetime of the system, which might be 10 years or even longer, to increase the cybersecurity.
At the moment, we probably just simply need a standard firewall. In the next few years, we will need some next generation firewalls. Then we would need IDS systems and probably something totally new in cybersecurity. And this is evolving fast.
The threat landscape includes new threats from the geopolitical system, but also from artificial intelligence and those things so that we don't know what we need in the next years on cyber security, but that we need to react very fast.
So the only possibility to do that is to deploy the required measurements as software agents, because we cannot always go to every secondary substation – which are hundreds of thousands in Germany – and install new hardware if there is a new requirement on cyber security. It's not possible.
Klaus Mochalski
The key takeaway for this episode, I would say, is that we as vendors, device manufacturers and security solution vendors, need to remain flexible, make deployment simple and easy for our customers, even in large and evolving and highly distributed infrastructures. And we need to be ready for whatever is there to come in terms of regulation and other requirements.
Jos Zenner
I fully agree on that. Well, it's exciting times, so let's work on this.
Klaus Mochalski
Absolutely. Thank you for the good discussion and it was great to have you on this show.
Jos Zenner
Thank you, Klaus.
