In this episode of OT Security Made Simple, Klaus Mochalski welcomes Digital Forensics Managing Director Dr. Frank Stummer. Frank talks about the importance of digital forensics in security incidents in OT, the challenges he faces and how he was able to thwart another security incident worth hundreds of millions of euros at a logistics service provider.
Listen to us:
Transcript
Klaus Mochalski
Hello and welcome to a new episode of the Rhebo podcast OT Security Made Simple. Today I'm sitting here at a Rhebo partner company, Digital Forensics, and with me is founder and CEO Frank Stummer. Full disclosure: Frank Stummer is also a founder of Rhebo, has been working for and with Rhebo for many, many years. So we know each other quite well. But Frank, tell us a little bit about yourself and what you're doing at Digital Forensics.
Frank Stummer
Yeah, thank you very much Klaus. And not only that we know each other very well, also very long. But this is another story. Yeah. What I'm doing here at Digital Forensics is indeed digital forensics, meaning that we are coming into projects or into cases where industrial companies are attacked, hacked, or whatsoever. So if there is a security breach and security event or something like that, we are coming in to detect, to analyze what happened. And the best thing would be also to get some more details about it, maybe also to bring the attacker, the hacker, before court. That would be, of course, the best resolution.
Klaus Mochalski
Very interesting. So, before we talk about what forensics, digital forensic means for the field of OT security, maybe you can give our listeners a little bit of background about yourselves and how you got into the field of digital forensics.
Frank Stummer
Yeah, that's a funny story. So, originally, I'm not a forensic guy, I'm not even a technical guy. I'm more of a financial guy. I studied economics, and in our first company, I was the CFO, actually. But then very soon, I became also part of the sales process, and also of the product management process. And so I was part of the product solution development, but also of the use of such solutions. And very, very soon, I became a part of the forensic part of our company. And this is something that really caught me, because it’s not only about data, with technology, or whatsoever, but it also has to do with people. Indeed.
So, forensic analysis always also has to do with thinking about, okay, who is maybe the attacker? What are the processes behind such an attack? What happened? And this is something which I am always very curious about, not only the technical side, but also, let's say, the people side, the human side of such processes.
Klaus Mochalski
Yeah, well, as they say in security, at the end, it's always the people and not the systems or the tools. It's about the people. So, very important. But now let's move away from general IT security into OT security, operational technology, which is this part of the IT that is managing and controlling industrial control systems, industrial automation processes. So again, we are in an environment where you suddenly have less people and more machines and systems. So what are the specific applications or use cases you have seen over the past years of digital forensics in OT security? And what, if anything, makes forensics special?
Frank Stummer
In OT security, there are two specialties, I would say. One makes it more simple to do forensic analysis, and the other makes it more complex. The one thing which makes forensic analysis in the OT environment much more simple than in the office IT environment is indeed that we are talking about machines that are talking to other machines at the end of the day. And that means we have a very high grade of determinism. And so based on that principle, [communication] is comparable and quite easy to see what's not normal.
Klaus Mochalski
So it's more predictable what's supposed to happen and what not.
Frank Stummer
Indeed. And that's the one thing. But on the other side, of course it's the OT side. In reality, it's much more complex and complicated than the IT side in every company I've ever been, meaning we have much more assets there. We have much more nodes in a network communicating with each other. We have much more IP addresses. Many do not know that. I guess five years ago or so, the pure number of IP addresses in the OT sector was much higher than the number of IP addresses in the office IT sector.
Klaus Mochalski
That's probably interesting to look at. So people tend to forget that in the IT environment you have a few surface systems, and then on the other side you have a computer sitting in front of each employee, and that's basically the amount of systems you're up against. Maybe a few printers, too, but that's mostly it. While if you go to a factory, to the shop floor, you have all the machinery, all the controllers of the machines, but also the building automation systems, the IP cameras that you're running, you may have many sensors, each of the sensors connected to the network. And so you can easily see how this sheer number starts to explode.
Frank Stummer
Yes, that's true, but there are also a lot of similarities also between IT and OT. For example, everything is configured by people, by IT administrators at the end of the day, right?
Klaus Mochalski
And you're still using Windows, although we hope we will get rid of it, right?
Frank Stummer
Yes, of course, on the OT side we still have a lot of older assets and some old firmware versions, but for some reasons it's okay to use old stuff there if you have other possibilities. If you get in safe environments, so to say in a secure environment, then you can of course also use all the firmware versions, that's okay. And by the way, it's not that old, as many people think. So it's older and we have more assets, and sometimes we have also older assets, but it's also modern. Of course it's modern, like the IT sector is too.
Klaus Mochalski
Yeah. So let's look at the specifics of forensics analysis. What does it mean? And maybe to our listeners to explain what the Rhebo system is doing. So Rhebo is building an anomaly detection system. We are monitoring network traffic and look for events which look suspicious, diverting from the ordinary. And then we are not just creating alerts about unusual events, but we are also always recording a snippet of network traffic. So that in forensics analysis, one can go back in time, like with a VCR, with a recorder, and look at what really happened. So it’s not just about relying on our system, raising the right alarms with all the necessary context information, but collecting raw data which we can then hand over to an analyst and they can do whatever they want or extract whatever they need from these snippets. That's for us really the handover point and the start of the forensic analysis. From a services company view offering these services, how does it continue? What are the next steps? What are you usually doing with this information and what do you take out of that?
Frank Stummer
Exactly what you said. So, of course I get, let's say, a first picture of what is the system, what is the network, which assets are there, directly out of the Rhebo system, for example, or other systems. And that's good. For the first overview, so to say. But exactly what I need to analyze then really in a deep detail is I need raw data at the end of the day. And in network communication, raw data means PCAPs, for example, so really packet captures of the network communication, completely original and not manipulated. The raw data as it is or as it was. And exactly one of the best functionalities of the Rhebo solution is this little function to snip, to catch the snippets of the network communication.
And this makes my job easy at the end of the day, so that I can just get the PCAPs, it's not that big, this data, and I can really go to the last bits and bytes in the old data. And then our job is really to look, what happened in detail. Is there any, not only suspicious communication, but what exactly is suspicious? There is, for example, data. Is there wrong information in there, or is there malware in there or something else? And this is exactly what we are doing. So we are looking manually in most cases using tools, of course, but at the end of the day, it's still a lot of manual work. We are analyzing each bit, each byte in the raw data.
Klaus Mochalski
Well, it's a bit disappointing to hear for me representing Rhebo, because I'd like to see our system as a system that provides most of the information that you need to analyze an incident. But I can also understand why this is not always enough in every situation. And that's why having access to the raw data, I see how this is very valuable to analysts for this post mortem analysis, especially after an incident. So maybe for our listeners to better understand how this process of an incident analysis looks like. Can you think of a recent example where you did such an analysis and where you used such a detection system?
Frank Stummer
In an OT environment, yes. But beforehand, what is really necessary to understand is that forensic analysis is always really the last step. In most cases of security events in a company, you do not even come to the forensic analysis.
Klaus Mochalski
So you only get there if things have gone wrong before. Yes
Frank Stummer
So really post mortem, I like this latin expression very much. You are doing this [forensic analysis] with a lot of effort sometimes, but only if it's really important. For example, really to look what exactly happened there, or to catch an attacker. I would say, very roughly in only less than 1% of security use cases, or much less than 1% of security use cases, you are really doing a forensic analysis because it's not needed for the other 99%.
Klaus Mochalski
So the less customers see a forensic analyst, the better for them.
Frank Stummer
Yes, indeed. If you need my services, then [your situation] is not nice for a customer. But to give you an example, we did a forensic analysis two years ago, three years ago. Of course, [I won’t say the name] but it was a bigger company. A 2-3 billion euro group in the logistics area.
Klaus Mochalski
Were they part of critical infrastructure, according to any local legislation?
Frank Stummer
Not at the time. It's becoming more and more part of the security laws now, but not at that time. But what happened there was: We got a phone call. In normal cases, we do not get a phone call from the direct end customer, so to say. But we got a phone call from an IT security firm, our partner, and they said, okay, something happened here. And indeed, in these words: “Something happened here.”
Klaus Mochalski
It was a service company providing services to the end customer.
Frank Stummer
Yes, to the end customer. This company is also doing SOC services, security operations center for the end customer. And that's good. But also, sometimes they do not know what exactly happened there. But of course they said they had seen a security event. But the security event was exactly in these words: Something happened here. So [this customer] was a manufacturer of logistics systems, with a lot of customers all over the world. And very suddenly, at two or three end customer sites the complete production stopped. The complete logistics lines, they stopped.
Klaus Mochalski
So really the worst case that you never want to see in a manufacturing environment.
Frank Stummer
Yes. And within just a few weeks, there were losses of something like 200, 300 million euro. So we were talking about a very important production cycle.
Klaus Mochalski
It was a major incident, really?
Frank Stummer
Yes, really a major incident. But they did not know what happened there. They just saw the results and the results were: Okay, two, three of the sites just stopped! But they did not really know what exactly happened there. So there were some ideas, so to say, at the first beginning, it's only ideas, neither negative or positive. Only ideas, neutral ideas.
One was, okay, there is a misconfiguration, something was wrong with the solution itself. That can happen. And the second idea was that there was an attack. There was an attack ongoing, with some different subcategories of what could happen there. And [Digital Forensics] came, of course, for this second path, so to say, of the analysis into this project.
And what we did there was that we analyzed everything we had. But it was not that much. We did not have a lot of raw data at this time.
Klaus Mochalski
So you looked mostly at log data, I assume?
Frank Stummer
Yes, some log data. But in the OT sector, and in particular three years ago, there is not that much log data available. [In the IT you] have much more log data. But not in the OT sector, because many of the servers do not have log capabilities. So we were not able to see what happened there at the first instance. So what we did was to say: “If this was an attack, maybe the attacker will attack again sometime in the future.”
So we installed a system using also the Rhebo [OT security monitoring] system to monitor the complete communication from the headquarter to all the customers and customer sites, to monitor the complete communication. And of course, we had some ideas where such an attack could happen. And then we just monitored it. And indeed after eight months this attack happened again. And we monitored [and detected] it directly on the fly in real time. And of course then we stopped the attack immediately so that the 200 to 300 million euro incidents couldn't happen again. But we saw this attack again, and it was really an inside attack. So it didn’t come from the outside, it was not a competitor or a foreign state or whatsoever. No, it was really from the inside, this attacker.
Klaus Mochalski
So to better understand this. It's obvious that the customer lacked the detection capabilities. Initially the log data wasn't sufficient, and then they implemented an [OT intrusion] detection system. And the system did not only help in providing forensic data, but it also, I assume, detected the anomaly surrounding this suspicious behavior and raised the first notifications. So the assumption would be if the customer had deployed such a detection system in the first place, then they would have been able to detect it when it first happened.
Frank Stummer
Yes, indeed, because the attack itself was quite simple, I would say.
Klaus Mochalski
Nothing fancy.
Frank Stummer
Nothing fancy. And indeed it was not, let's say, any malware or stuff like that. What happened was actually that one of the engineers there in the headquarters had just sent out a “Shut down” code to the systems outside the normal remote maintenance periods.
Klaus Mochalski
So the person used his inside knowledge and simply shut down the right systems, knowing that a few of them would bring down the entire production infrastructure.
Frank Stummer
Yes. And he didn’t try to hide anything. It was really an open communication. It was not encrypted or whatsoever, it was just a communication. The shutdown to the systems out there went via the normal communication path. We recorded the raw data, we recorded all the smoking guns, so to say.
But what happened afterwards was a bit of a pity.
We were able to really see, this was exactly this server with that username at that time, at that date, sending out the shutdown. But afterwards – and this is unfortunately not part of our job anymore, but of the company – they found out that 300, 400 people – they didn’t even know exactly the number – have the same username. So they had no good rules, so to say, procedures for individual accounts, for individual acknowledgments at least, or something like that.
Klaus Mochalski
So it's a typical case of these well known security loopholes that everybody knows about, like shared username, password, [USB] stick to the monitor.
Frank Stummer
Yes, for years the password was in the intranet. And the intranet was not very secure, at least. So it's not funny, it's really serious, but something I see many times, very often, unfortunately. And then afterwards, of course, they started a big project to set up real good procedures, individual accounts and all that stuff. And this is needed, of course. And this is what I have said. It has always to do with people and procedures and rules and all that stuff. And all the technical systems, the monitoring system, the analyzing – what I'm doing – comes at the end, so to say.
Klaus Mochalski
It's interesting to see such a specific case. I mean, there has been much talk about insider attacks and how they can be dangerous and hard to prevent. But here's a specific example and it shows that it's not easily solved with systems like a firewall. This clearly does not help here. And many companies also, from our perspective, still assume that if they do their perimeter security, so securing the entrance to their infrastructure, then they've done their homework. And here, it shows clearly that that's not the case.
So it's about the internal organization, the people and protecting the network segments. And lately we hear a lot about segmentation projects, micro-segmentation projects, and this really helps. But it also shows that you need to have the detection capabilities in place. It could be log monitoring through your SOC and a SIEM system. It could be a specific anomaly detection system. But you need to have these systems tied into organizational structures.
Frank Stummer
Yes, indeed. At least, privilege [management] is very important here and all that approaches.
Klaus Mochalski
Maybe even to have zero trust, which is a very current topic indeed.
Frank Stummer
Until now, not only three years ago, but until now, there is really a lack there in that regard in most companies. It's starting to change. Also in the OT sector together, or based on new regulations like the new NIS2 in Europe or NIST in America, the IT Security Law in Germany, to be specific. And this is changing now based on standards like ISO 27000 and ISO 270001. So it's changing now.
Klaus Mochalski
I think it's also a matter of awareness. Awareness is growing. And here legislation helps. So thank you very much for the insight. I think that's really helpful because, statistically, what we observe at our customers are usually much less exciting cases. So in our regular [OT vulnerability assessments], we find incidences of outdated firmware, outdated operating system software. We find communication, which is not supposed to happen, but not really critical events or incidents. Which shows that there's either a high level of security standards already in place, or that the things that can happen don't happen too often. And so it's even more interesting to learn about such a specific case.
Frank Stummer
Yes.
Klaus Mochalski
So if you had one takeaway, one learning for customers in the manufacturing or even the critical infrastructure domain, looking to secure their environments against insider attacks, what is it they would have to do?
Frank Stummer
If you would allow, I would have two takeaways. It can also be three.
Klaus Mochalski
Probably, one is too simple.
Frank Stummer
Yeah, the first very generally is indeed security procedures. And also safety, by the way, starts with good procedures, with good processes, and with the awareness of the people. Always, that's the most important basic thing you should do.
Klaus Mochalski
Indeed, without this, any technical measure is most likely worthless.
Frank Stummer
Yes, indeed. And the second takeaway for companies like Rhebo, but also for the end users is: You should be prepared for forensic analysis too, because if something very serious, very important occurs, then it's really important to know what happened, maybe to prevent this in the future, right? And also, if possible, to catch the attacker.
Klaus Mochalski
And you want to catch the attacker in the first place and not wait for another eight months, like in this case, for it to happen again, right?
Frank Stummer
And therefore, I can just say as a forensic analyst, please provide us the possibility really to analyze something, to analyze the data. And therefore, [an OT intrusion detection system] is not only about monitoring, but it's also to get the raw data, the PCAPs, the logs, and so on. These are very valuable data and information which we can use in our job, but sometimes [companies] forget to think about such information possibilities and sources.
Klaus Mochalski
So again, a mix of organizations and the proper tools. And on the tool side, please don't forget to make sure that you are always able to collect sufficient data to do a, as you called it, post mortem analysis.
Frank Stummer
Yes. So fingers crossed, nothing should happen, of course, in the first instance. But if, then at least you have the possibility to analyze what happened, to actually be prepared.
Klaus Mochalski
Yes, I think that's a very good takeaway. So thank you very much for this interesting discussion and the presentation of this specific case. It was a pleasure to have you.
Frank Stummer
Thank you very much.
AFTERNOTE: The described security incident is also available for download as a case study (pdf).
