Cybersecurity Threats And Attacks In Modern Manufacturing

Patrick. >> Yeah, thanks a lot. Um, yeah, idea is to give you a little bit of an idea about an application space for cyber security that is maybe not that well known that for couple of purposes is a little bit specific or at least not easy accessible for a lot of people. I'm more trying to give you a high level introduction. What are the parts that make it so specific? If you look up what is actually a manufacturing system, that's kind of the textbook image that you get and you get those nice keywords. And to be honest, I don't took them from a textbook. They're from a company that sells smart manufacturing systems. They use IoT. IoT is industrial

IoT. That's essentially instead of using a Raspberry Pi, we use a rack one. Um, big data. We have autonomous robots, cloud computing, everybody is doing AI and machine learning these days, of course, um fast network connectivity, edge computing, so I guess that's also a couple of Raspberry Pies. Um simulation digital twin, I will get back to that a little bit later shortly. Um an additive manufacturing, which in layman's lay person's terms is 3D printing. Um so everything looks very modern. And I think for each of those individual topics we had either this year at Bites or last year at Bites a talk how to secure them. So I'm done. Uh we can start with the Q&A

part. Um not really. Let's have a look how the reality looks like. When they talk about automation, they talk about those type of devices. These are proper logic controllers or PLC's. They exist essentially, at least since the ' 70s, um 1970s. They're rather stupid things, large relays essentially with a little bit of logic that can switch on and off heavy machinery. very simplified terms. But these days they also have a nice display. You can see here LAN they have a network connection u but they still need to talk also to all the other systems in there. We have scattera systems control and data acquisition systems usually for controlling some manufacturing systems or chemical plants like here on the screen.

We have EIP systems. So classical enterprise systems. Um screenshot from a very widely used piece of software. Um essentially UIS over glorify databases. Um and logic that tries then to implement processes on top of that mostly for knowing where do you have your resources? Do you need to order new stuff? Um and on top then systems that drive the actual manufacturing where you have the instructions how to build actually physical stuff. So quite a number of different systems which leads then in fact not to some nice modern system but to a very complex and usually very heterogeneous u software architecture. So this is essentially a reference software architecture. So not this is real life how it looks like. This is how

essentially a well-designed system looks like in that space. Um proposed by one of the companies that um builds a lot of components in those systems in particular ERP systems. SEP is also building um manufacturing suite so the other part of POP but not doing that scattera and PLC stuff and that's all very complex pieces of software that communicate over a large number of different protocols some of them based or at least wrapped within TCP IP some of them using different protocols and essentially all of them designed at a time where nobody could imagine that you connect such a system to something like the public internet. Um to give you an idea about the importance of those systems, the

statistics that I was able to find and it's really hard to find uh really concrete information in that space because the companies don't want to reveal it is um that's from the US. The average automotive manufacturer loses round about $22,000 US per minute if one of their production facilities stops. So think about what that means in terms of oh I need to shut down our systems for a couple of hours for doing an upgrade. the uh salary for the people doing the upgrade becomes negligible essentially and those systems operate for a very long time. That's one of the things that is completely different to most of the more common security topics web security traditional application

security. Um so SAP Europe 6 was released in 2005 so 20 years ago and will still be supported for another 5 years. So think about you write software now essentially you have to live with the bucks of that software pretty much to more than half of your career. Um and that's not uncommon. the seaman's SPS which is something like one of those PLC systems um they on average are being sold for 12 years and 10 years additionally then guaranteed uh spare part availability. So you still get the physical components if there are some defect and of course you get software patches at least for security issues. So we are talking also about 22 years of support and that's the

support that essentially every customer can buy for a reasonable amount of money um at least in the SEP case. I know that there are then also special customers that do individual deals to extend the support for them by another number of years and I'm sure Seammens is doing the same. I know that Microsoft is offering also individual support contracts. But then if we go down the line to the more what we know as an end consumer type of software, Windows embedded or Windows IoT how the earlier versions were called. So not the Windows version that we run on our desktop PCs or on servers. This is a special stripped down version of Windows for embedded

configurations with super extra long or support life cycles is supported for 15 years. Um, now let's get to something that maybe many of you will also still consider to be a rather old technology. Java is a programming language and that is a language that is being used a lot in this um application area. the long-term support releases, the LTS releases are supported for around eight years and then I try to find out what is the support life cycle for web browsers because of course all those systems these days are using web applications as the final uh end user interface. I haven't found any information for Chrome or Safari. Firefox has extended support life releases where they state and I quote

more than a year. I guess that means 366 days on average. Definitely not 10 years. I looked them up for the last couple of releases. They were in the range of 14 months plus minus. Um so we're talking about a year to one and a half years. And now think about securing such a system where 2005 which UI technology was modern at that time for web front ends if you wanted to do a little bit more interactive development. Who remembers flash silver light? um if you operate such a system essentially you kind of bought the right to use that until 2030 neither Adobe in the case of flash nor Microsoft in the case of silver light uh

really want to be associated with those technologies anymore that gives a very interesting space in terms of securing those systems and also if you go into uh such a system landscape as a penetration tester. Um the knowledge that you need to have in terms of the technology stack um archology is part of the business essentially. And we can see the problem that um yeah core technologies the systems are using or built upon are way uh less or shorter time frame supported by their um manufacturers than what the enterprise systems uh promise and guarantee their customers not only promise they actually guarantee it. Um and I can for example confirm that since Java is open source

um SAP is essentially maintaining a fork uh of old Java versions that then matches their support promises which is quite an additional effort. Let's have a look at a couple of example attacks that we have in that space. This is a message that uh one of the largest aluminum companies um Morris Kyper got um don't need to read all the details. Um I guess it's also one that you guess the idea very quickly. There was a sign significant flaw in the security system of your company. You should be thankful that the flow was exploited by serious people uh and not some rookies or maybe they mind trustworthy uh or that is a form of responsible

disclosure I don't know um but essentially we encrypt your data with RSA and AES um and without getting the keys from us uh you don't get access to your data and that means also in the case of such manufacturing systems that usually uh configuration files databases that the manufacturing systems are accessing are being encrypted and that means that production um is in still stand and that said I don't have data for aluminium production but for the car industry we've seen um costs of those $22,000 US yeah $22,000 per minute. So if you then suddenly recognize that your production stands still, you get such an email that uh definitely gets your attention. And what happened here? Um in December

2018, the hackers weaponized an email. So essentially they crafted a fishing email with a special attachment. um really well done in the context of a legitimate uh communication between an employee at North Kro and I think it was a supplier or a customer uh of North Kyro. Um that payload installed a Trojan on the employees PC. So nothing special there. Um the antivirus software detected that Troion and quarantined it within a couple of days but sadly that was too late. Um those couple of days, most likely only a couple of hours, uh if at all, given how well the first step was prepared, were sufficient for that attacking group, for getting control over that PC, move laterally within the

infrastructure and over time gaining admin privileges, which meant they could then really encrypt a large amount of data, which they essentially did. Then around March 1819th, this is where the uh company recognized that attack because the production stopped shooting the encrypted data. And we can see from um December to March. So we are talking about round about 4 months where the attacker were in the system with security patches installed and antivirus software being installed. Um interesting here maybe from a a product perspective. The company had a super open communication process what was going on with regular uh um press updates as well and um interviews they gave um and they refused to pay any ransom.

Um and the estimated damage for the company is more than 45 million uh British pound. But I said essentially this is one of the examples where we don't see any special uh attacks really on the manufacturing systems. [Music] This is also an interesting one uh from a slightly different perspective. Um reads first very much standard. In May 2016, the US Department of Homeland Security released a search alert based on actual exploitations going on uh and compromise of unsecured internetf facing SAP applications. And one vulnerability particular CVE was highlighted in that alert. And that CVE was the CVE 2010 5326. And for those of you you that know how CVE numbers are being assigned, there is already something interesting

going on. CVE 2010 means that CVE has been assigned and the vulnerability has been discovered and known publicly in 2010. So if we assume that that happened in December 2010, we are still talking about nearly five and a half years in which that publication was known to the public. And usually those CVEes are also only published after patches um have been made available. A patch has been made available was available when the CVE was published. I checked that. So we have here the situation of a critical patch and this patch was uh classified as critical which at that point in time in 2010 SEP wasn't using uh CVSS scores which most of you might know which is a

standard way of measuring the criticality of vulnerabilities. They use their own system and critical is the highest category in that system and essentially that means if you don't install security patches closing critical security vulnerabilities on paper you lose the support for the system that you're operating. So the support contract essentially obliges customers to install um patches for such critical vulnerabilities. And if the US homeland uh Department of Homeland Security is involved, we might also speculate that the customers affected by this vulnerability weren't some private sector companies. It's very likely that that there were some government government organizations where we usually assume that they have a very high security standard. So even those are not able um to

prioritize patching in the right way and install patches within 5 years. Um for consumer hardware, we don't even get patches for that long. Um and we essentially cannot operate systems for that long. They have patches and don't are not installing them. And most of you have heard about this one here, stuck snap. This is a state sponsored attack or it's believed at least to be state sponsored on uh uranium u Iranian facilities for um uh processing uh atomic material and essentially that has a very long kill chain and here the interesting part is it starts with essentially standard application security stuff um here also with with a USB stick that was distributed and essentially hoping that

some employee um puts in that USB stick into their machine um then exploiting a couple of vulnerabilities including one in the printing system of Windows but then getting access and only being active when they were on a system connected to a very specific scattera. system provided by seammens that essentially is only used in this kind of facility. So that was a super targeted attack which um is one of the evidences that it was most likely state sponsored and then um this makes it here really interesting in the manufacturing area. Um it really was an attack exploiting a vulnerability on those scattera systems and then also showing to the operator. So if I go back um can think about a scattera system is

one of those systems where you have your physical processes running on your factory floor of maybe secured maybe it's not even safe for humans to be next to them and you have your screen with all the valve pressures and which valves are open and which ones are closed to control that system. And essentially they discoupled the uh display here with the physical system. And while the physical system was running into a critical condition that essentially destroyed the system, everything looked nicely and well for the operator. But this is essentially the attacks that we know about on those systems. And except for the last one, uh none of them is really manufacturing specific in terms of the systems that are being

attacked. Um a couple of years later in 2017 there was also an attack on a Middle East prochemical facility um this also similar specific um also attacking a very specific industrial control system this time one from Schneider SE which is a French company um most likely also state sponsored. So based on the publicly known attacks, um the exploit vulnerabilities are usually in rather old systems. Um essentially when the systems that are being attacked were developed under the assumption that they run in an airgapped environment, no direct connection in network connection to public facing networks. uh criminal gangs seem to focus on ransomware using more traditional kill chains and to a certain extent the shutdown of a production

system is happening by accident. They would most likely be similarly happy if they just encrypt um the financial data or the planning for the next year or similar similar operation critical data. Um but of course stopping production is for them a nice benefit. Suspected state sponsored attacks um are on a different field. They really try to attack very specific systems.

Okay. Now what is the elephant in the room here? We have all those enterprise systems essentially implementing business processes and we know business project process logic falls that's also a four category for web applications. Um, and the most famous one, the standard example so to speak, is the shopping cart web application that for example has a bug in the process implementation and might allow a customer to cancel the credit card payment that is usually done in an external system and still confirm the payment and the goods are being shipped without payment. We call that a business logic vulnerability because the buggy thing, the thing that has the vulnerability is the actual business process being executed

and not many people look into those. That's already a problem. And now we have those old systems and we have way more complex business processes than only operating a shopping cart. Um, so what about those? That's actually a problem as well. And that's currently a rather hot research topic as well. And there are a lot of different impacts that you can have as an attacker when you try to attack those processes. You have things like um there might be actions on a good that you are producing that are non-reversible. You execute them and if you do that wrongly, you can throw that product into the trash and no longer use it. there's well nowhere way

of repairing it. Um or it might delay the process a lot if you send it to the wrong production plant. It might be not available for 3 weeks which happens then results in a long stand still. you have those digital twins where essentially that's marketing speak for I have my physical goods and I have a digital model on it and I operate on the digital model and expect uh similar effects then to happen um on the physical model as well if they get out of sync your quality control is no longer working and you run into problems so you have all those kind of And I explained that essentially already that gives a lot of problems when you

need to achieve the positive effects of them. So you don't want non-reversible actions in your systems at least for large parts. You want to have timing constraints to being uh adhered to. you want to have um being aware of the provenence of goods that you produce and ship. Um and an attacker can willingly of course try to infiltrate that and invalidate those uh constraints and that's exactly what people are trying to do. The only problem is we are not aware of that. Not because the companies are not necessarily aware of that. Of course, if the production goes into a standstill, we learn about that. But usually companies don't have a big interest in making that public. Uh we

mostly learn about the situations where there is no way of hiding it. So in some countries there are legal requirements to publish such attacks or if it's end consumer goods. If a car production goes down, cars are delivered to end consumers. everybody recognizes um if they have to wait for their car suddenly a couple of weeks longer than promised. Ah and the other challenge here is a lot of the requirements the things that we need protect our systems against are specified and informal documents processes are not all digitalized. So sometimes the tracking of goods is done by QR codes that you need to secure as well or model securely. So a lot of new additional challenges and data sources

that we usually don't have in traditional web applications. And that's not an completely um academic thought of problem. I said we've seen a couple of those issues happening but not well documented because usually again um those arts are not being documented for the public. We have one refer I have one reference but also don't know the details to be honest. that was in an aircraft manufacturing um system for a US military jet where a business process flaw resulted in the fact that in one part of the manufacturing chain in a different factory. Um essentially that part believed that there are no airframes available and that started to order more and more new airframes and the other one they were

producing them but they were not collected and that resulted in a shutdown for two weeks essentially and that nice quote of the VP of manufacturing that airframes were stacking up like cordwood. Um I mean they are not small. Uh one would expect that somebody recognizes that but that results really in expensive uh attack vectors. Um and it's not only those kind of obvious things. There are a lot of new threats that we need to think about. Um in a project that we are currently working on. We are looking at those kind of issues and one of the use cases is aircraft manufacturing and you have suddenly attack vectors like talking an individual screw because if you do that wrongly that aircraft

might crash. Uh this is a safety alert from the national transportation safety board. That's essentially the organization in the US that investigates every crash of an aircraft. regardless if it's an autonomous one or one with passengers um or a cargo plane. Um essentially evidence in recording that there's one screw in the aircraft engine. If that is not being taught correctly, uh that aircraft is likely to crash. And if something like that happens, somebody doing a mistake or an attacker for example, modifying the instructions for the person talking that screw, the impact on the real life is pretty severe. And that's the research where we are working on where we're trying and this is an old prototype. The current

prototype that we have is mostly command line based that doesn't look like that nice on slides. the UI something to be developed at later stages but essentially trying to extract the information from real world systems and I know this is the red track um I still rather sell it in the way that uh the tool allows companies operating those systems to find the vulnerabilities to fix them before somebody else does. uh but we all know that those kind of security tools are of course always on the uh fence of being used in both areas or at least potentially can be used by uh people with different uh intention. So we're trying to extract the information, extract the processes that

are running on and looking then for areas where requirements are not being uh satisfied or where small changes result in a severe impact. This is here for example a very small process where people can improve their own work. um things that we also seen in some manufacturing scenarios. Um if you read the press where people essentially did their own quality uh inspection and this is something that we can detect with a tool and then also visualized to an auditor or to a security expert looking at those kind of issues or here even to a safety person that look if you run that process as here specified or implemented in the system the person doing the work can also improve and do

their own quality check. That's most likely not the thing that you want to hap happen or checking that high level requirements that you have extracted from the processes even are also propagated into the actual source code of the implementation and report violations there as well. That's all from my side. Timing works nicely because I wanted to have time for Q&A. Um the thing that I didn't tell and I thought about putting it into the motivation but I thought it's also nice to have it as a hindsight since 2021 in the yearly X force threat report from IBM manufacturing is the most attacked business sector it overtook finances nevertheless we hardly hear anything about cyber attacks on manufacturing and

that's a little bit essentially the part of the motivation for the t talk to raise awareness of that problem. As usual when designing systems and that holds essentially for all systems that we are designing or processes we should only focus on the expected behavior. of the positive outcome, but also what are the things that we don't want the system to do even if somebody is trying to push the limits and force the system to do it. Um, what holds for business logic vulnerabilities and that's often something that maybe results in deprioritizing them a little bit. They are usually vulnerabilities that need an authenticated user. So they are often put down as yeah but you need

to have an account that's an insider. Uh yes that's on the one hand true. On the other hand in particular in the manufacturing scenarios the impact that they have are severe. They are lifethreatening in many situations. We are no longer talking about oh yeah we are leaking a little bit of data that might be annoying for people. It's really life-threatening in the uh original sense of the word if you think about for example a crashing aircraft or an exploding chemical plant. Um so we really need to be careful with assessing the risk when looking at impact and likelihood and think about Yep. But if somebody somehow overcomes the first authentication barrier or maybe isn't actually insider

or if the um uh attack is interesting enough to uh uh bring somebody into into the system then we need to think about the impact a little bit more careful as usual. It's all from my side. Happy to take questions. >> Yeah. Do you think it seems surprising to me that there haven't actually been more attacks similar to if it's such a popular Q&A is not recorded right?