OT Security Made Simple host Klaus Mochalski talks to Stefan Grützmacher, who has managed several energy suppliers over the last few decades. For Stefan, OT security in critical infrastructures is still treated far too neglected. As an industry expert, he also has clear requirements for solutions for the critical infrastructure sector.
Listen to us:
Transcript
Klaus Mochalski
Hello and welcome to a new episode of the OT Security Made Simple Podcast. I'm Klaus Mochalski, founder of Rhebo. I'm particularly pleased to have Stefan Grützmacher as my guest today. I actually met Stefan quite a few years ago, I think in 2016. We had just successfully completed our financing round [with Rhebo] and wanted to establish an advisory board. All of the shareholders wanted an external advisory board with industry-specific expertise. And that's when we actually came across Stefan Grützmacher. And now I would like to hand over to you, Stefan, so that you can briefly introduce yourself. Because your background is actually very exciting for our topic of OT security and I think you can contribute a lot of experience from the last few years, but also in a future-oriented way.
Stefan Grützmacher
Yes, thank you, Klaus. Indeed. It's been many years since we met. I was and am a child of the German energy supplier scene, of municipal utilities, I started out in 1999 in the energy trading cooperation between municipal utilities, then managed various municipal utilities, the one in Solingen, the one in Kiel, then from 2012 to 2014 I was also head of GASAG in Berlin. Since 2015, I have been an independent consultant and interim manager. I have managed three municipal utilities on an interim basis, usually in times of crisis, have various supervisory board mandates and was also on the advisory board for a while until the sale of Rhebo to Landis+Gyr and then chairman as a representative of both you founders and the investors.
Klaus Mochalski
At the very beginning, you actually supported us operationally in business development, i.e. in sales, in early-stage sales. We hoped that the contacts you brought with you from the energy sector would give us what I would call privileged access to potential customers. And the two of us traveled around the country a bit, I can still remember it well, and visited potential customers. That didn't always work out so well, or at least not immediately.
Looking back over the years that have passed, what do you think is the reason for this, especially if we look at the years 2016 to 2018? The market was already there, at least in theory. We're talking specifically about the security of critical infrastructures and the energy sector. The energy sector is a large and very, very important part of the critical infrastructure. And the problems we are talking about today already existed back then in many, many areas. What are you doing today with the experiences we actually had there? Why didn't the customers welcome us with open arms?
Stefan Grützmacher
Yes, Klaus, that's a very good question. I asked myself that question back then, and to some extent I still ask myself it today. Yes, I think it's a classic problem that everyone knows, that IT security and OT security are really essential, especially in critical infrastructures. But it's like insurance. I get the insurance in anticipation of possible damage, which can also be existential, and pay the insurance premium for 20 years and nothing happens. I had the impression that it was a bit like that when we were traveling across the country from 2016 to 2018. Everyone was aware: "Yes, there is a diffuse risk. But why buy and implement a system for it now?"
We also spoke a lot with those responsible in the control rooms, who were of the opinion that everything was secure, everything was air-gapped, and it couldn't happen. That's probably why there was such a theoretical, diffuse understanding of the problem, but not really the will to tackle anything. In addition, of course, there is always the problem that there are more than enough projects and activities in organizations, and [cyber security] was not necessarily the most pressing issue, but that was my analysis. Not really intelligible arguments. But that's how it was and is.
Klaus Mochalski
In this context, let us briefly explain once again what Rhebo specifically offers its customers. The solution that Rhebo offers is a prevention solution. In other words, we try to detect incidents, especially cyber attacks, at an early stage in order to prevent the negative effects that such an attack can have by taking appropriate countermeasures. In other words, prevention.
At the same time, however, we can see that the entire cyber security industry has grown in recent years, especially in this area [of OT security], but that a lot of money has been spent in response to attacks. There have indeed been repeated successful attacks, some of which have been publicized. You have to assume that the number of unreported cases is much higher. There are also many published figures, for example from [German ICT industry association] Bitkom. How high the damage is, i.e. the billions in damage that are caused every year. In other words, real incidents actually happen. My feeling is that much more money is often invested in the response, i.e. in dealing with an incident, than in prevention. Is this a specific issue in OT Security or is it just the way it is? Is it in our human nature?
Stefan Grützmacher
That's a very good question, Klaus. I can't answer it definitively either. The fact is that there have been various security incidents in our industry in recent years, as far as I have been able to observe. Not yet in OT, but in classic IT in the ERP system. Enercity has been hacked, Integer has been hacked. Enovos in Luxembourg was hacked. As far as I know, these were significant incidents that caused considerable problems. To my knowledge, not yet in the OT, but nevertheless it is there, and it has certainly increased massively for a year and a half now, since the war in Ukraine. And we can see that we are being attacked as a critical industry, as a critical infrastructure. But surprisingly, I don't think we are doing enough about it.
Klaus Mochalski
I'm just thinking about what we observed in the years after we went our separate ways, from 2019 until today. And the challenge on the customer side is that even if an energy supplier or a municipal utility in a medium-sized city with, let's say, 300,000 citizens, decides to procure and operate such a prevention system, there is a major hurdle that needs to be overcome, the operating costs, but also the expertise to operate such a system.
So we have the typical problem of a shortage of skilled workers, which exists to a greater or lesser extent in all sectors. We then responded to this at Rhebo by moving away from being a pure product company, as had been the case until then, and increasingly offering service packages. To the point where we offered our customers the option of completely managing and operating the solution. This also led to significant success. And that was a trend that could clearly be observed on the market.
My question to you now is about the mandates that you have taken on in recent years, where you have specifically accompanied energy supply companies on their way: For some time, you were also able to observe this skills gap topic there. Did you have an insight into the security division in particular, and is it a widespread problem or are there simply companies that are in a much better position? And can we learn anything from this?
Stefan Grützmacher
Well, that is definitely the issue. We have already discussed this intensively during our time together. In the Advisory Board, it made sense to switch from being a pure software provider to a service provider. And my experience is that the people, the men and women in the control rooms who operate the grids are true experts when it comes to handling disturbances, malfunctions, and classic switching operations. Really, nobody can fool them in the classic physical operational business!
But people are not trained in OT security and the possible detection of attacks or even anomalies. And to be honest, they don't have the extra capacity to do it. So running a network is really not trivial. As a rule, there are network control rooms that operate electricity, gas, heat and water together. I actually know at least two media. So electricity and gas, and sometimes also heat and water, so there really is a lot going on in the control rooms, and not just in the event of a disturbance. And the ability to recognize this and react to it with a service is, I think, very important. And that's why there are also these SOCs, the Security Operations Centers, which can offer this service in order to be able to react immediately at the outset and then, of course, to initiate suitable response measures. I think this is a very important step.
Klaus Mochalski
This is actually the observation we have made and continue to make on the market. And that is why we are continuing to invest in the service sector. You have now spoken about specific incidents. Have you ever had to observe a cyber security incident live during your various mandates in recent years or when you were still a full-time managing director at various energy suppliers, and support them from the perspective of a managing director or consultant?
Stefan Grützmacher
Fortunately not. Best of luck. Last year, I had a service provider attacked by a DDoS attack. But that was only a temporary problem, lasting one day. But I can say that when I started my last interim mandate in May last year, shortly after the start of the war in Ukraine [note: the podcast was recorded in 2023], that was the top topic for me, the third or fourth conversation I had was with our IT security people, with our IT people, but also with our control room people. Because in my opinion, this is still one of the biggest dangers that can happen to a company, but especially to a critical company. We then initiated penetration, tests, etc.
Of course, the human factor is crucial, i.e. the person sitting in front of the IT and also in front of the OT. We have provided ongoing training. We even got cyber security insurance, which is now almost non-existent. So for me, this is one of the top risks for a company, but also for a critical company. And if it's all about money so to speak, which is a common business model, you simply have to say: on the other side is an industry that has simply recognized [cyber attacks] as an economic sector, as a very profitable economic sector. And there are professionals at work. And we have to counter this professionally with both the "human factor" and the "systems factor".
Klaus Mochalski
There are probably successful attacks against companies on a daily basis. Nevertheless, as you have just confirmed, there are few successful attacks against operators of critical infrastructure in the energy sector. Although this should be a very interesting target, especially in the current geopolitical situation. What do you make of the fact that there are attacks, so to speak? We have these examples time and again where administrative areas, where the state, areas of the state parliament are attacked, where systems are then encrypted using encryption Trojans, where in some cases it is necessary to switch to paper files and manual operation for many weeks or months. Why did we get off so lightly in the area of critical infrastructure, i.e. really in the area where the energy supply could be acutely affected, where there could be a threat of a power outage? Is it luck or are we really so well prepared in Germany?
Stefan Grützmacher
I think it's a combination of everything. On the one hand, you have to say that it comes from history. The control systems were also physically air-gapped, they are self-contained systems. I wouldn't say I am an expert now, but that's not the case anymore. That's no longer the case, because of course there has to be some kind of connection to the outside. But I do believe that the OT area in the critical sector is better protected than the normal IT area. That's in the nature of things.
Klaus Mochalski
Perhaps I would like to add one more point. We regularly carry out security analyses for all our new customers and also for many existing customers, where we actually look at how securely the infrastructure is set up. Are there any anomalies? For example, is there any exposure to the Internet? Is there data communication with devices that nobody can explain and that should not be taking place? Is there unauthorized access to certain systems? Are there any systems that communicate that nobody knows about, that are not allowed to communicate? All of these are potential targets for cyber attackers. And they do exist. We actually find them every time. This means that if we had to take on the role of an attacker, we would know where to start with almost every one of our customers. In other words, it's technically possible, but it just doesn't happen.
Stefan Grützmacher
Yes, that's what I wanted to say. As I said, we are already at a somewhat higher level of security. But as you say, I also know that from our time together, when you look at the traffic in communication, there are many protocols, many devices that nobody knows anymore, that have somehow been in the networks for years and decades. So, as I said, we are at a slightly higher level of security. But I think we've also been very lucky. I am absolutely certain of that.
And to be honest, I'm not sure - as I said, I haven't experienced it - whether we're also getting public information in this area. Well, I mean, we haven't had any widespread outages in Germany. Yes, that's the case, but I can't say whether there have been any incidents that could have led to this, but I'm assuming that something like this won't necessarily be published as a press release. Of course, if I know that my ISU system is down or my entire ERP system is down and I can't write an invoice for weeks, then I will have to communicate something to the outside world. If I now know that my PSI system [note: PSI is a provider of the OT core for energy companies] is infected or about to be infected, then I won't necessarily put that on the Internet as information. As I said, I think it's a compositum mixtum. In fact, I'm surprised that nothing has happened yet.
Klaus Mochalski
Yes, let's keep our fingers crossed that it stays that way. We are actually working on helping our customers to keep it that way. Of course, that's what we all want. Now let's look ahead. If you were given a mandate and part of this mandate was the task of systematically increasing the level of security for a medium-sized municipal utility, as I mentioned earlier, in a municipality with a population of 300,000. This would be an organization that is well aware that it still has some homework to do. But which has so far been below the critical threshold. In other words, there have not yet been any strict legal requirements and there is still a lot to do. There are basic first steps, but overall relatively little has been done so far. How would you proceed with such an organization? What would be your recommendation for the first steps? You always have to be a bit careful not to overstretch resources and goodwill. How can you make getting started with more cyber security palatable and not make it look too big and difficult?
Stefan Grützmacher
Yes, it's exactly as you say. So, of course, the sledgehammer method doesn't work. There are experts sitting there who know what they are doing when they operate the networks. As a rule, you don't have the specialist know-how at management level to discuss things in depth with these people, these experts. Nevertheless, the cyber threat, the potential danger is there.
I can say that we tackled this issue in one of my last jobs. Fortunately, too. You at Rhebo also took a good approach, namely to start a project together with the control system manufacturer and a university to do exactly that. Because with a university, there are still a few external resources, including neutral resources. I think the control system manufacturer is an important player in this, because at the end of the day they also ensure the functionality of the control system. I know that in many of the discussions we had back then, Klaus, that was always an issue. How does the Rhebo solution work with the control system? Are there any interdependencies? Yes, no. You always rightly said that it's a passive system, so there can't be any interdependencies. But it was still an issue. That's why I think it's good to tackle it together with the control system manufacturer and, if necessary, a neutral institution, and to approach it carefully but consistently. Because if you imagine possible damage events, then of course any effort I make ex ante is minimal. And I think that's crucial. I have to do it.
And last but not least, the costs. After all, we are in a regulated business and are even paid by the grid fees. It is probably a problem of resources and mentality, but we should no longer rely on our luck.
Klaus Mochalski
That's a very good overview. I would like to emphasize the topic of simplicity again, because I believe that is what customers will ultimately demand. No operator of a critical infrastructure can afford to introduce new complexity into the network, into the infrastructure. This means that whatever additional functions are implemented must be easy to implement and operate. And I believe this is also something that has developed significantly in recent years.
You mentioned control system manufacturers and of course it's an integration that is very, very important. My control system shows me the status of my infrastructure or important network elements, and my security system has its own dashboard, an overview of the current security status of all these elements. Of course, it makes a lot of sense to integrate the whole thing, and that's what we've actually set out to do. That we bring this integration with us from the outset, so that no matter what the infrastructure of a specific company looks like, a solution can be seamlessly integrated and generate no or as little additional effort as possible.
And that's actually what the title of this podcast says. OT Security Made Simple, i.e. simplicity in installation, implementation and operation, is actually the biggest challenge and the most important one for us. We have a duty to our customers, not only as a product supplier and solution provider, but also as a service provider.
And I think that if the solution is simple and at the same time offers effective security, then there should no longer be much of a question as to whether or not to invest in this area. Because these solutions are also becoming more and more standard and, let's say, are offered in reasonable packages on the market.
Stefan Grützmacher
Absolutely right. Absolutely right. That, coupled with a proper service, can only be the future.
Klaus Mochalski
How do you see the market developing over the next few years? Do you think that we have reached a saturation point in this market? Or will we continue to see dynamic development?
Stefan Grützmacher
Not at all. We have just worked out that, surprisingly, this topic has not yet received the attention it actually deserves in terms of practical implementation. As I said, it has become relevant with words on Powerpoints, but I don't yet see it being implemented to the extent that it needs to be.
And when I think about the fact that the networks are becoming more and more complex with more and more intelligence, i.e. digitization, it naturally also means that they are becoming more vulnerable. And the less it can be operated manually in operational mode and the fewer threats can be detected. If I previously had ten substations in my grid and saw a few more stations, and I now think about the smart meter rollout, then I suddenly see tens of thousands and hundreds of thousands of intelligent devices in the grid, which also have to be operated somehow. That no longer works manually. It has to be automated.
Regardless of the fact that we broke our own legs a bit with the complexity of the law on the digitization of the energy transition. Fortunately, things are now getting a little easier. But the system is becoming more complex, more digital. It has to be operated and monitored in an automated manner. There is no other way.
Klaus Mochalski
In other words, to summarize, there is still a lot to do here. The market is developing very dynamically and there will be more and more networked devices that also offer an attack surface. You mentioned smart meters. This means that the challenge here will also be to map this complexity, this increasing complexity, in the security systems in such a way that it does not make operation more complicated and complex. And that is where we need to invest in order to be able to provide our customers with a managed care system with a manageable service in the future.
Stefan Grützmacher
That's exactly how it is. That's exactly how it is. And that was and is the exciting and intelligent thing about your products, that they recognize the normal state very, very quickly. Of course, there will always be anomalies that - and let 50 messages pop up, 49 of which - are not critical because it's just the famous technician in the substation or whatever. But it's better to say 50 times: "There's something there, take a look", than to be wrong once. There will never be 100% security, but we should still do everything we can to reduce the probability of an incident as much as possible.
Klaus Mochalski
A very good conclusion. Thank you for the exciting discussion. Let's look to the future and work together to ensure that we can continue to develop simple solutions despite this complexity. Thank you very much for the interesting insights, and I really enjoyed discussing with you here today.
Stefan Grützmacher
Thank you, too. I wish you continued success.
Klaus Mochalski
Thank you, Stefan.
