This episode sees Hans-Michael Krause from Bosch Rexroth and Dr. Frank Stummer from Digital Forensics talk about how to integrate OT security in new and existing automation environments without having to wait for all vendors to comply to security requirements. We dig into how Bosch Rexroth make their industrial automation platform ctrlX secure and discuss why the ROI of an OT monitoring should not only be evaluated in terms of cyber security but also production availability.
Listen on:
Transcript
Klaus Mochalski, Rhebo
Hello and welcome to a new episode of the OT Security Made Simple Podcast. I'm Klaus Mochalski, Founder and former CEO and now advisor of Rhebo. My guests today are Dr. Frank Stummer and Hans-Michael Krause. Hans-Michael Krause has been working for Bosch Rexroth for quite a long time. Bosch Rexroth is a provider of automation solutions. And for these solutions, security has become a more prominent topic over the years. That's why we thought it would be interesting to have him on the show. And also with me today is Frank Stummer. Frank has been working on these kinds of partnerships for Rhebo. He has worked for a long time with Bosch Rexroth. With this, I would hand over to you both for a brief introduction. Maybe, Michael, you want to start?
Hans-Michael Krause, Bosch Rexroth
Yeah. Hello, everybody. Nice to be here in your podcast. I'm Hans-Michael. I’ve been with Bosch Rexroth in different roles for 16 years. My current role is that I'm Director of Ecosystem. This is a partner network of additional solutions around our automation platform, ctrlX AUTOMATION [pronounced: Control-X Automation]. I guess we are going to talk a little bit about ctrlX in this podcast, so I will present it later on. I'm an automation guy. I'm not a security guy, so I'm looking forward to this conversation and also to learn something from you guys.
Frank Stummer, Digital Forensics, Rhebo
Thank you. My name is Frank, Frank Stummer. I'm a co-founder of Rhebo, together with Klaus, and I worked in particular in partnerships like Michael, for quite a long time too. On the other side, I'm also coordinating research projects for Rhebo, really for the future of what we are doing, what we are integrating into automation platforms, for example.
Klaus Mochalski
Thank you both. Michael, you already mentioned ctrlX. The topic of our podcast is OT Security Made Simple. My question to you is how do you believe the two go together? Because I believe that the partnership between Bosch Rexroth on the ctrlX platform and the Rhebo solution for OT security is a very good example of how to keep things simple. But how does it work from your perspective? Give us maybe a little bit of historic background how this partnership came to fruition and what you were looking for as Bosch Rexroth.
Hans-Michael Krause
Maybe let's start with a little bit of context. We are working in factory automation. Our main work is, I always say, to make machines move. To make machines move within the factory context. When we compare the OT world to IT, I always say we are lacking. We are 20 years behind. When you take a typical automation system that is widely installed in the field, you can say that those devices are not at all secure. Because a PLC, let's name it a PLC, needs to cover different needs, operator safety and machine safety. Security was not the focus for a long time in the OT industry. Security is something for the IT guys and not for the OT guys. The OT guys need to program a PLC. To give you an example, it's pretty easy to log-in into most of the widely spread PLCs that are available today in the field with standard passwords. Taking the brand name as user name and as a password works for or has been working for many of the old PLCs.
Being 20 years behind is not very good when now you meet this famous Industry 4.0 or the IT-OT convergence and you want to connect machines to the internet and you want to read out data from the machines, right? This opens up a huge field of possibilities also to attack these systems since they are not state-of-the-art from a security perspective.
That was one of the reasons why in 2017 we [Rhebo and Bosch Rexroth] had our first contact. Thinking about, what can we learn from Rhebo or what things can we do together to help this industry, to help machine-building, to help factory automation. To reach this, let's say, new security levels because there's a demand for it. Because everybody would like to connect their machines. That, I think, was the starting point of our discussions back then.
We also wanted to learn a bit how we can secure our PLC systems to secure our production. We are producing our PLCs, servo motors, servo drives, mainly in Europe, but also in China and the US. That was the starting point of this partnership. Even in our industry, we see a lot of attackers are more and more approaching PLC systems. It started with Stuxnet, but it continued with others. Now there's even a toolbox called Pipe Dream to attack PLCs and industrial controllers. Even our industry is more and more affected by ransomware and these attacks. There's a clear need to do something.
Klaus Mochalski
That's a very interesting perspective. Thank you. Before we dive into how we actually help machine builders and factory automation people to secure their modern IT-OT, converged implementations, like you called it, maybe let me play a little bit of devil's advocate. Why do you believe or do you actually believe that we have to secure components like PLCs that you named? Or isn't this the other way around? Like you mentioned before that, well, in the past we had the concept of the air gap. The devices were simply not approachable, they were not reachable from the internet or from the outside. This is how they were protected. Today we all know they are digitally connected, so we don't have the air gap anymore. But isn't this now the simple job of IT to make sure we have a virtual air gap? So basically separating all of these critical components, for instance, PLCs from the outside world by IT security measures, and then having the automation people not have to deal with this issue at all. Could this be a solution?
Hans-Michael Krause
Yeah, I mean, still today in 2023, many industrial controllers are actually air-gapped. Even after more than 10 years of Industry 4.0, I would say 90 % of the machines are not connected to the internet. There's still this air gap. But if you would like to, let's say, enjoy and go to the next steps of increasing OEE [Overall Equipment Effectiveness] and improving the production, you actually need to connect the machines. There are companies, like we have in Bosch, who have these strict, you call it virtual air gap solutions. We have structured our production IT into different zones. There's the production IT and the machines protected by a virtual air gap, as you named it, from the IT. This is very strictly regulated within the Bosch plants. We have a lot of Bosch plants, more than 280. There is even a central IT security team who sets up these rules, and that's well managed. But what about a smaller customer of ours that only has one plant? Do they have an IT security expert defining virtual air gaps? For the smaller ones, it's difficult to set up all these standards and to follow up all these norms. For them, we need to provide simple solutions to help them because they cannot afford a team of five or ten centrally coordinated experts.
Klaus Mochalski
That's really what makes the difference here, simplicity again. That's very good. Maybe that's time to explain a little bit how the integrated solution works. The general idea here is, of course, to have a platform, a solution platform like the ctrlX platform, where you can pull in building blocks for different applications and where security, specifically OT security, becomes just one of those building blocks to make it as simple as possible and basically drag and drop items, functionality items into the solution. Can you give our listeners some insight in how this actually works on the ctrlX platform?
Hans-Michael Krause
Yeah, but maybe take one step back. Again, I was talking about all the legacy PLCs and the state of legacy PLC security. Seeing this status quo made us come to the conclusion in 2017 that we actually need to design a new automation system based on IT standards because we see more of these convergence. We set back, we defined certain rules for us when we designed a new platform. Typically, in a company like ours [Bosch Rexroth], it happens every 10-15 years that a new control platform is designed. We set certain rules that we would like to achieve on this new platform. Out of these characteristics, it must be connectable, it must be extendable, open. But security was one of our defining principles that we wanted to integrate into the new automation platform.
Two years later in 2019, this became ctrlX AUTOMATION, a new platform for all different kinds of automation tasks from the controller, servo drives, and the corresponding software. CtrX AUTOMATION was the basis then to further explore. We followed strict security guidelines on how to develop such a system, processes, and state-of-the-art technologies coming from IT as an OT company. This basically was the starting point to further explore the terms of security and it made total sense to make a new platform, which also has versatile use cases that we talk about later.
Frank Stummer
And maybe to emphasize exactly this point, the first basic level of security in OT is to get more security or higher level of security directly in the devices. This really is a change in thinking about how to design and how to develop devices in the OT environment. That's what I found out during the last few years. And that's good because at the end of the day – and this is exactly the same what happened 20 years ago in IT – security by design means that the devices are becoming better. Not only more secure, but also better. This is really a very basic principle to obtain in that field. Then we will come to other measures, maybe like Rhebo’s, for example, in a complete system of more secure devices by itself. That's exactly what we did together with the CtrlX AUTOMATION Platform.
Hans-Michael Krause
To give you some examples of what we did to underline Frank's statement here, security by design. Let's look at the hardware, e.g. industrial controllers. We selected components that are secure, so we integrated a TPM chip where customer's keys can be stored in a secure way. We introduced a secure boot chain that only trustworthy software can be executed on the device. And looking at the software level, we choose, for example, the operating system Ubuntu Core with their security features. One of the nicest features that we see is that you can run software in a sandbox. When the software is attacked by somebody, it's only the sandbox which is attacked and it cannot influence other software or other apps running on the platform. Those are just some measures that we learned from the IT guys that were implemented 20 years ago. We now finally also implement them in OT to make our devices more secure from the beginning.
Klaus Mochalski
That's very interesting. I had a lot of discussions lately with potential customers and partners about OT security by design principles and what they are, what they mean, how they differ from IT security by design principles. I think this is a very specific topic and I think it's probably worth doing a deep dive on this specific topic in a later episode. Today, I would like to turn the attention a little bit towards the customer perspective. I think by now our listeners are understanding what security of the CtrlX platform means. It's secure by design. Then also it provides security controls, so that can be implemented or installed as an add-on on the different components as they are rolled out.
But if we take a step back and look from the customer's perspective. If I'm planning a new factory, say of a medium-sized production company, and I'm interested in the steps that take me through this journey so that I end up with a modern, let's call it industry 4.0, compliant production system – if something like this exists. How do I tackle security? When do I start taking this into account and what things do I have to observe and pay attention to? Is it only the right security controls? Does it involve the design phase? So really, Michael, from a customer's perspective, how does my security journey with a new installation of CtrlX start? What steps do I go through until I'm ready to actually start production in my new secure production facility?
Hans-Michael Krause
That's a very good question. Basically, I would start when I plan a production, of course, you need machines. So what we call a machine operator or end-user needs to select the machines that are needed, for example, for a packaging line. There are certain machine builders that he can choose from. And mostly this is done by specifying what is needed in a spec sheet and then going to different machine builders.
For example, if I were buying machines, I would specify that my machine builder integrates a control system with state-of-the-art security features. On the one hand, connectivity is needed, OPC UA, but on the other hand also that he implements a state-of-the-art security control system with security features. That specification is the first point. Then I would only choose machine providers who can offer this. I mean, in the best case for us, of course, the machine builder should use CtrlX Automation.
But let's assume there is no machine builder which offers state-of-the-art controllers because machine builders also have their innovation cycles. They can be maybe quite long, seven years. Then I could, for example, think about adding additional protection to these machines that are maybe not secure. Using secure gateways or ctrlX AUTOMATION which can also act as a secure gateway. This is from the machine perspective, the selection that I would choose.
Of course, you also need to think about factory IT and how to plan and how to connect these machines. Then you must think about how to protect the factory IT network. Maybe this is something where Frank can join from his perspective.
Klaus Mochalski
Yeah I was going to ask, sorry to interrupt. I was going to ask what kind of security frameworks are important here? Because it sounds like you should look at what's already existing in terms of security frameworks like the IEC 62443 or the ISO 27001. There are quite a lot of these frameworks. As a machine builder, let's assume using the ctrlX platform for security too, what frameworks should they look at?
Hans-Michael Krause
Yeah, as a machine builder, I would definitely search for controllers that support the IEC 62443 norm. That is already a selection criteria on how to choose a secure industrial control system. That's an easy guideline, I would say. We are also fulfilling this norm with our main controller, the ctrlX Core. That's the easiest thing. Obviously, you also find some guidelines on how to implement secure networks in your machines. Because machine builders have more and more Ethernet devices in the machine, around the machine, and they need to cope with all these network participants. There are certain very good guidelines available on how to design these networks. This ethernet based networks.
Klaus Mochalski
Frank, would you support this? What other frameworks are there that you could look into? Are there sector-specific frameworks or is it just this one? What does the landscape look like here?
Frank Stummer
Yes, I would add or maybe I would disagree a bit with you. There are not that many different regulations. We have indeed some very general regulations, standards like the IEC 62443 standard. For me this is a basic standard because it gives a complete framework on how to sync security and then how to get to a higher level of security and what the levels of security are and all that really basic regulations. It's a very pragmatic standard from my point of view, indeed.
Then, of course, in the different sectors, you have a bit different standards to adopt. But basically, you have a basic framework and then you have to adopt it like in automotive with a T-SEC. It's a bit different from the ISO 27000 to 270001 standard, or it's adding some more standards, and that's okay. There are not that many. It's not that high of an effort, I would say. Many think, Oh, it's a high effort and it's too complex, too complicated. No, it's not. That's really my personal experience and my personal view. These standards give you a very good path, a very well-defined path, to reach a high level of security. Your example was a new factory. In the IEC 62443 standard, you have this approach of the three different roles of the component vendor, of the system integrator, and of the operator. That's perfect. That's the real world. That's like it is, and so you can really use it.
Klaus Mochalski
So a clear recommendation. If there's nothing you've done so far in terms of security, IEC 62443 is the standard to look at first and then maybe cover the more specific, let's say, industry-specific regulations or requirements later on. But always start with the IEC 62443
Frank Stummer
Yes, indeed.
Klaus Mochalski
Good. So while talking about regulation, today there are many industries that are regulated in a way that they are forced by the regulator to implement certain security measures in their OT infrastructure, specifically critical infrastructure operators. We know this from utilities, power plants, but also many other sectors. Is this playing a role for CtrlX customers today, Michael? Or do you see this as something that is not yet influencing your business today?
Hans-Michael Krause
It is influencing because there are ongoing discussions for factory automation or for machine builders and machine users that there might be some regulations in the future. We have certain customers. I talked about these long cycles of innovation in machine building. When we talk to customers implementing ctrlX in their machines, they actually want to introduce the machines in 2026. Also in three years time, they want to plan a new machine. They are currently already designing, for example, the control system because the cycle is so long. These customers who look a bit more into the future, they all ask us about security. They ask us: Can you provide security updates for your software? That's already a selection criteria when you select a control system. Even though it's not regulated yet, I see many customers looking into these questions.
Klaus Mochalski
So what is causing this mindset of the customers? Is it their awareness, their understanding of the risks? Or is it that they observe regulation in neighboring industry sectors and that they say: Oh, this could happen to me, so I want to be prepared?
Hans-Michael Krause
I think it's both sides. I think it's the examples of ransomware attacks that go through the media that also happened to the machine-building industry in recent years. They observe, of course, what's going on with the Cyber Resilience Act, for example, which is not yet decided. It's both ends. I would say that they are looking at these machine builders with the long innovation cycles. But I also have to admit that there are machine builders and machine users who unfortunately still have not so much know-how about these topics and that becomes dangerous when looking into the next years and into their business. I would always recommend to every machine builder to have a look at the standards and to get familiar with these topics. Otherwise, it will be really difficult in the future to sell machines which are not secure.
Klaus Mochalski
Yeah, and – we spoke about this in the previous episode [with Thomas Menze from the ARC Advisory Group] – they should probably also have a look at services and consulting services with people knowledgeable in OT security. There still are not too many of those today, especially at smaller customers, that are skilled in planning, setting up and ultimately running these solutions. Because we've seen this quite often with many of our customers. They know that they want or even had to implement OT security solutions, but especially smaller utilities always had the challenge that they lacked the people who were able to run these solutions. Of course, they may have IT staff, they may even have someone proficient in IT security, but that's not the same as OT security. There will be a staff shortage even in the future. So potentially service companies, consultant companies will play a very important role in helping the smaller customers, e.g. the typical German middle-sized customers in production, with setting up these new infrastructures and making them secure.
Hans-Michael Krause
Yeah, for sure. I think there's a huge potential in helping the customers. But it's not only security where, for example, monitoring network traffic can help. It's actually also – and Frank, maybe we come to this example – interesting for machine operators for detecting faulty Ethernet devices that might stop a production. Maybe that's something, a use case, we can talk about here in the podcast.
Klaus Mochalski
Yeah, absolutely. I find this very interesting and we've been talking about this and we call this a surprise benefit to many of our customers because many of our customers decide to acquire the solution for their security capabilities, but they end up using it on a day-to-day basis for troubleshooting. Because many of these solutions, including Rhebo’s, work by detecting anomalies in the communication, which are caused by a security incident. But it turns out that also technical faults like a broken network cable, or a malfunctioning network controller generate the same type or a similar looking anomaly.
Many of our customers end up using our devices for day-to-day troubleshooting. That's very beneficial. But the challenge also is that usually it's different departments at the customers who work on either problem. The challenge here, organizationally, will be to bring the two together, like the operational knowledge for OT infrastructure and the security part. Quite often they are still not working well together. I guess there needs to be some more grease on these cog wheels in the future. Maybe Frank, you can chime in here and talk a little bit about observations that we have done recently.
Frank Stummer
Yeah, indeed, the majority of anomalies, what we are seeing or what our customers are seeing in their network has nothing to do with cybersecurity, to be precise. It has something to do with the network itself, bandwidth issues, broken cables. You said it before, and that's true.
What I can observe is that the organizational challenge is becoming better and better. Also we have this history together, Michael, at your own production site and for quite a long time now. I can remember we sat together on the table with the IT and OT and it was very interesting because – and now I'm a bit harsh maybe – I felt that it was the first time that these colleagues were talking to each other about really concrete issues, what they've seen. But it was a good starting point at least. This was years ago, and nowadays it's becoming better and better, I would say. Right, Michael?
Hans-Michael Krause
Yes. Maybe to dive into this example to make it a bit more concrete for the listeners, I already explained two facts.
One fact, the Bosch plants obviously are very secure, and we have more and more Ethernet participants in our production IT network. This was actually the starting point of this project that Frank was referring to, where we said, okay, we use the Rhebo system, obviously also to detect anomalies about possible hackers, but also to detect what's going on in the network.
You may not believe it, but a lot of network traffic can sometimes cause real production problems. So when you have a faulty device – and it's not only PLCs, it can be label printers, and so many Ethernet devices that are now in production – it can really jam your network communication. It can actually stop the important information – like for example, what needs to be produced coming from the MES – reach the important participants, i.e., the machines. You can have production problems, even production downtime due to that.
I think that's the lucky side effect, as you call it, Klaus. This lucky side effect can actually also justify a system like the Rhebo system to be installed and also bring a fast return on invest already when you consider how much a production stop costs. You know, when you have a production stop of one hour, of one day due to a faulty Ethernet device. Crazy thing actually, but I heard that it happens. The costs of this [installing an OT monitoring with anomaly detection like the Rhebo system] already pays off with one avoided production stop. It's security on the one hand, but it's also guaranteeing production on the other hand and proper network communication.
Klaus Mochalski
That's a very interesting point here. To me, this sounds like a perfect key takeaway for our listeners. Security is important. Making it simple for our machine builder customers is a key requirement. Without this, it's not going to happen.
But also keep in mind that the system can help you with all the things to keep your production running. Combined you suddenly have your return on invest [ROI], which is otherwise always difficult to calculate for an IT security solution. There have been attempts to calculate ROI on IT security. You can do this. But the way you described it, Michael, like preventing an outage due to a broken component or a simple thing like a broken cable is, of course paying many, many times over for the installation cost, for the operational cost of a solution like we're describing. That's very interesting, and I think that's a very nice closing remark that we saw here. As I said, I'd really like to dive into the security by design topic in a future episode. This would be very interesting to discuss in more detail for our listeners. But today I'd like to close this and I thank you very much and I hand over to you for your final comments.
Frank Stummer
Maybe I will start. It's indeed what we talked about, the simplicity and this partnership between Rhebo and Bosch Rexroth. Our solution and the ctrlX Automation platform, it's a good example because at the end of the day, it's just an application. For the end user, it’s just an application to be installed very easily, and then you can use such a solution. This is true with all other applications too, with the automation applications and so on and so on. This is a perfect real-life example of such partnerships, and therefore I'm really thankful to Michael for this opportunity.
Hans-Michael Krause
Thanks for inviting me. For those of the machine-building customers or machine users who ask themselves “How can I start simple?” I recommend the following: Get a ctrlX core or controller from us and go to the ctrlX store, our App Store, and download, for example, the Rhebo app, which is available there. Start playing around a bit by, for example, using the controller as a secure gateway for existing installations, for existing machines, and see what you can do. Like separating networks.
Listening to this podcast is already a very good thing [laughs]. Now apply what you’ve learned from Frank and Klaus and maybe also a little bit from myself and start applying it directly. It's not very difficult. You don't need to be an IT nerd to start implementing security in your production. Congratulations, listening to this podcast is already the first step. But now do the second step.
Klaus Mochalski
Thank you, Michael, for the perfect closing words for today's episode. We will make sure to put the links to the App Store you mentioned in our show notes so that our listeners can click on it and continue to put it in actual work. I thank you very much for participating in the episode today, and I also thank our listeners for listening. Bye-bye.
Sources
Explore the Bosch Rexroth ctrlX AUTOMATION platform: https://apps.boschrexroth.com/microsites/ctrlx-automation/en/portfolio/
Visit the app store’s Rhebo page to find out about the integration of OT monitoring and anomaly detection / intrusion detection capabilities in ctrlX AUTOMATION.
