How To Pentest Industrial Systems (OT/ICS) And Still Live To Tell The Tale!

excellent thank you very much and uh first of all thank you very much for having me as uh I have really looking forward to do this presentation um so um one of the good things about being the second one who tells something about industrial security is that some of the things that already been mentioned in the first thing so just a quick show of hands anyone who work in a company with the industrial Network or production and then the next question would normally be who have done anything security related and then the bonus question is if you actually have tested and you got stuff you know going really wrong I have done that and that's some

of the experience I really would love to share with you um and there's two thing I actually would like to apologize for the first one is that during my presentation I might be a bit harsh on it security dudes and toads and I would also like to apologize for some of the colors for whatever reason this uh you know the Blaster cannot like red and that's a problem when your logo is orange so apologies up front for for that exactly exactly so this is me blah blah blah I have 25 years of experience for the last 10 years I've been working only within industrial security I've been working in transmission energy and done a lot of uh other you know fun

stuff um I work for I'm a part of the representative in Denmark for NATO and have done a lot of other uh different stuff and then last but not least sometime I will refer to something called IC range and IC range is one of our uh know services and basically you can see it's a OT Centric Capture the Flag platform where with real security and I'll I'll just briefly tell into that uh a bit later oh sorry is it open source or it uh no it's uh it's uh the stuff we use is is open source but uh basically when you lock in then you would be accessed to real uh industrial equipments so so it is a

service uh but it's not purely open source however we use a lot of open source and our contribut back to the communities very often whenever we get you know real industrial things we pentest them and then I always ha because my my research syst called me said hey Michael we found a bunch of vulnerabilities here so I know there's always a lot of talk with the the vendors afterwards so that's our kind of contribution so per se not open source but I hope that answer your answer your question cool so what we do we are breaking stuff as uh I just shortly mentioned we have this platform uh one of our main goals would basically be to

do that everyone would be able to learn security testing and Pen testing because just as uh on the on the panel here we have a huge gap in the talent pool for o for it security and it's even bigger within OT security um yeah and uh basically also we had to a lot of assessment and this is like the knowledge and the what the takeaway from those assessment I would love to share with you to today so first thing first I always have this in my presentation everyone who comes with a it sit security background like I did I had 15 years of experience when I started to look into OT security and I thought you know I know everything

about this to paraphrase Joda from Star Wars whatever you have learned in it security you must unlearn what you have learned because it might look the same it might feel the same but there's very often a huge difference especially when you're doing security testing when you're doing pen testing so uh this is something I'm not going to go too much deep into when we are talking about it we are talking about stuff that interact with data when we are talking about OT security which is like the overlapping arch of the description we something that interact with the physical world it might be the local power grid it might be building uh facility like this what keep the air

condition running and basically you see the biggest difference is that in Enterprise security you're looking into protect the data no whenever you're doing a pin test how can I get access to the to thing but when you're looking into IC security or OT security you're looking to secure the process by itself because nobody have ever G been promoted for having updating a system or pen testing a system then suddenly then half of Albania is just having a black out so so again we need to do things uh a bit different the brief story of uh security is that if you look into the security poster you would probably for everyone who have been working here for more than

5 or 10 years you would feel like this would be a a blast back to the to the present there's a lot of the security models that is very old because original it was s that this system should be separated and just imagine that having a system where you designed to it to be separated one day suddenly find itself directly faced on the internet you know then we might have a a a slightly problem here of course the data must flow but that doesn't necessarily mean that we should be able to be finding them on showen and last but not least a lot of the protocols are rather old so there's no encryption and I have been

that if it's not broken don't fix it however for The Last 5 Years this you know this has been changed really much much um particular because the vendor has P out that eventually security is something they need to to take seriously yes if we look into traditional it security everyone who have been on a one-day security crash course have learned something about oh we're looking into confidentiality we're looking into integrity and we're looking into availability that's all good but however when we're going over to the OT side of the house the scale change then we're just suddenly looking into uh safety and safety is not the same thing as security as one of my cooworker once said that

one of his prime concern was that his employees was able to come home to the family safe and sound and as you can see then you just have confidentiality in the lower part so again it's another ball game but it's something that we still can can use to our advantes so just a little quick quiz if we're looking at into what have done the most outages in power grits the number has been taken from the US power gits but I'm seeing they're probably pretty comparable all over the world so besides you know force of nature what would be the top three root causes for power Bridge what kind of creature would you think that was the number one reason for

power outages and it

guesses sorry once again please physical yeah you you're are almost there if we're looking into nature you know we we we normally oh thank you uh we normally say that at one point when the hackers and the adversaries learn to train squirrels we really do have a problem uh because that's the most uh you know the most predominant way of of our tities all right if we then look at humans what kind of creature would you think that would be the second most responsible for all outages any guesses perhaps you have a good idea no that's fair enough I'm not going to put you on on the spotline ladies and gentlemen the second most responsible

thing for outages is the it security the person who think I know it security now I'm walking into another facility and I take all my knowledge and particularly all my tools and use them again this is a really really really really bad idea um I can give you a very good example there was a large incident in the US uh back in 2019 and basically they said we know our it network has been compromised perfect we are not sure if our industrial network has been compromised so they called in a bunch of high- paid consultancy doing incident response and gave them that task and one thing that is very predominant within industrial security is that the network

drawings might not be accurate and they might not be you know from this sensory so the guy said it's not a problem that your network is your documentation is not up to par I have a tool something called metas sploit which can help us a lot let's just put it that way out of all the tools this person could have chosen it was absolutely the worst because that when he did run that if they hadn't had an incident before in the production they particularly had it afterwards ladies and gentlemen the point of this presentation is that you're not going to be that person who's doing that kind of problems and then last but not least uh we see the all

time in the news it's advanced persistent tread state sponsored XYZ fluffy bunny or whatever so isn't it actually quite interesting that our own it security people are responsible for more outages than uh everyone else so we have uh sometime we also called the OT like industry 4 or uh and what I think is quite interesting if we're looking into industry number uh three is back from the' 70s we haven't figured out that security model yet so now we just go into four and it's going to be really really awesome it's going to anyone who's going to be interested in this field I can guarantee you would have job until you choose to uh to retire so the strength and the weakness

is that it's built to last it's very simple to it environments but I would like to quote one of my rather good friends he's a Godfather of my daughter he said if Michael can find vulnerabilities in those kind of systems believe me anyone can you know with friends like that you don't need any enemies right so if we look into the LIF side prediction and why is this so hard actually to to pen testing on if you look into your average uh mobile phone laptop probably about three years then you're going to replace it if you're looking into uh you know different product uh environments it's going to be 8 to 10 years and then when we are

looking at energy and other kind of equirements it's something like 20 to 35 years and then just a little sort imagine that you're sitting and doing architecture reviews and you're doing just imagine that when you're doing a design for industrial security you need to foreseen what kind of problems are we going to have in 20 years good luck with that ladies and gentlemen that's why even today we see okay there's a lot of interesting findings and things this is the reason why because nobody can predict the future at least not so far so with that short very short introduction I'm going to talk on the main topic like pentesting o and then knowledges from from The Real

World so how to break stuff never ever please doing industrial testing in production unless you really need yourself a new job and you're too lazy to write a reignition later to your boss because this is a really excellent way to get yourself fired ladies and gentlemen don't do that um one of the thing is like nmap d o- a as some of you might know nmap is you know like the default support scanner when you're doing A- o you're seeing what kind of operation system is it running and the a is like what kind of applications thing and the way that nmap find out what kind it is it sends a lot of different packages a lot of invalid packages and

based on the response it would actually see okay based on this profile it's like a Windows 2016 server or whatever the biggest problem is that a lot of those it equipment it's perfectly fine but when you move into the OT environment then the implementation of network is very very very strange sometimes and giving a lot of you know invalid uh request would actually Break Stuff I didn't believe it myself until I started and doing you know some testing when I started back in 10 years ago and then suddenly I had equipment who just went boom on me and that's at that time I was really happy that it was in lab testing so it wasn't

in running in production because otherwise things could have gone really bad uh again don't bring a a USB drive with all your fancy hacker tools and M where uh and then last but not least the testing is possible but we need to do it in in a good way and we need to come prepared so if you have any slides you would like to uh I would ask you to remember it's like please this one because before you start to doing penetration testing start to have a okay from the senior Engineers those people even that there might not be technical he or she would be able to say okay this equipment would most likely be able to

withstand this one would definitely not and then preferable if possible test the devices you know in individual uh nonproduction networks it's not always possible and that's you know one of the uh exciting things about it but you know if it's possible start to do it in a test lab before you break stuff and then last but not least particularly when you're doing onsite testing triple test the client for the IP address you have been given I know this sound really modan this really like simple but very often we have been out on the test we got a list of IP addresses and we start to look into it and then we see okay this do

10 I it it doesn't seem to be whatever scope Target we have been given and then the engineer when goes all wide and said oh my God it should be 110 instead so again you know stupid human mistakes but again always you know don't always trust but also validate um when you're doing live testing ensure that you actually have seor people around in order both to test the systems and to recover the equipment because stuff might fall over even with the best of your preparation uh I can give you a good example we have built a internal 12 steps guide for whenever we doing pen testing and basically means like we start with the least um likely things that can go wrong

and then allocated a lot of time between them so that means in example we're sending you know just some simple things as ping packages and then we said okay now we're waiting 15 minutes did something break no it didn't cool then move on to the next step and this 12-step guide basically it takes a day to do it but uh we have good uh you know good experience with that another thing you always should ask the client on because again that's a big difference between it and OT you said okay why would you like us to do pen testing very often the client would go back and said I need to know what kind of vulnerabilties say is and then I'll

let you into a little trick just by doing passive pen testing meaning that you would capture the network traffic from the environment you can do that via span or mirror import that was considered to be quite safe then you would have that information and then you can work on that rather than to doing active intrusions on the test very often the pen test would be able to stop just there because you would in normal circumstances you would get a lot of vulnerabilities just by looking at okay this is like a seens S7 1200 PLC it's running firmware XY Z and then you would be able to see a lot of vulnerabilities so compared to it pen testing this is a

very easy way to do your job and um again uh very often then the pentest would actually be able to stop there sometimes I'm so fortunate that particular vendors comes out and said well we're not interesting about the CVS course that already known we would love you actually to do testing like zero day hunting and then we would go in and uh and do that so yeah all right just to give you a little example on how easy it is to do tread hunting and ioc's in uh in such Network we have something called mbus it's very close to being a protocol that is 50 years uh old now it's a client server model so it's

you know predictable how it's supposed to communicate it doesn't have any kind of encryption and it doesn't have any kind of context and with that I mean that if you have the right commands you would be able to interact with mod bus because it said Well if you know how to communicate with me you're probably a good guy or girl welcome to the security of OT right it's interesting uh it always uh and and other simple trick is that whenever you're doing pen testing uh if you're looking into nmap or some of the other node tools it would always look in for for the protocol called 43 which is basically read device identification and a good thing is that

if you're ever going to do blue team in industrial thing this is one of the first thing you should look after if you're running MBS and believe me almost all networks are running this kind of of service because no engineer he or she would her sold word would ever run this module but it's the first thing that a hacker would do so you have something clear text no no valid reason to use it but this would be the first thing a ad hacker would do so look after this please and this is when things get really scary um again back to the uh IC uh range we have sometimes vendors approach us and said hey would you like to resell

our stuff and I said no it's it's not what we do and then they said hey can I can I donate you some stuff yeah of course you can do that this is something call coming from China and uh you can buy it on Alibaba uh basically the price is $120 which is roughly about 10 times the 10 times reduced price of what Sean s and everyone else does and basically it can communicate to everywhere so just imagine then suddenly you have stuff where you s thought it was being protected and then you just have a direct internet connection straight out when you're doing pen testing look for those kind of devices because very often

it has been attached to the network but the asset owner might not know it because this is um there was a big incident in Saudi Arabian some years ago and part of this thing was there was a vendor who put up some equipment that definitely should not put up and so yeah um as one of the previous speakers said it's quite interesting out of like you know this 10 12 kind of malware we know the last three came after Russia invaded Ukraine so of course it shows out that there's a huge interest in uh in those kind of aial and also make really really good sense because that if you can you know attack a city's infrastructure it's

much easier rather than sending soldiers you know CU just said hey someone could just take better care of the power gits in example and um another little resources when you're doing pen testing you know go for the easy apples right the low hanging look at this list it's available on GitHub uh and someone have taken the time to go in through like a lot of Dusty manuals and looking into like what's the user credentials and passwords for a lot of different systems if you find this well you know you can just log on and the good thing is most of those equipment are single user so you only have to guess you only have to

log in with admin and then as you probably can guess the password would very often be admin as well so uh again save yourself some time show you some of this as the previous speaker talked about there was this attack back in Ukraine 2015 uh the uh black energy attack and let's see I can see we're a bit short on time so I'm not going to but the really interesting part was that in this attack one of the most uh 90% of this was it attack and 10% was you know industrial attack and they attacked something called a little moxa which is uh this little thing here it's been used like everywhere globally and um I've never had such

thing so I went out and bought it for 50 and within a evening I started to find vulnerabilities there you can see this one year later oh sorry uh one year later uh uh then uh me and two other researchers was being credit for you know postmortem finding the vulnerabilities there so again back to my good friend saying that if I can find vulnerabilities is just way way way too easy um last but not least if you run end map and again do it in a sensible way there's a lot of old scripts and again if we go from the IT world you would say oh but those scripts they like five or 8 years old believe me they

still work because this is a word in OT like it it doesn't change a lot it doesn't work very fast so again and another resources if you look into that you have like we had the I protocol that was mentioned this morning there's a a script for that that we have something for mud bus you know the one with Section 43 S7 is for semens one of the most you know deployed plcs around the world so plenty of stuff to go out and uh and start to do have fun and if you really want to have systems start crying there's a a a a little attack called black nurse which is basically a ping attack uh I uh icmp package and

basically says this attack just say hey I don't know where I'm going and I don't know what's wrong believe me even today most Industrial Equipment they would just die on this kind or be really really really sad on it so just imagine this simple attack would bur if if you have like if you have been granted a a a you know OG penetration testing if you can't find anything else this is your Hail Mary you would find something just by this so please remember every machine is a smoking machine if you operated wrong this is particular correct in OT and um but again remember you can actually do pen testing in industrial networks you just need to lose the it

security attitude like forget about talking patching 24/7 Zer trust and all the other things very often you can turn them the weakness of OT security into your own personal magical powers an example unencrypted traffic is a really good thing on those kind of networks because it tells you what kind of devices what kind of firmware do I run so it's really really easy to to do pen testing in that limit the exposure to networks and then uh start looking at funing uh if anyone are interested and have skills in that we're always looking for skilled people and uh Freelancers and with that I would say thank you very much particular to you thank you much for

the