Risk OT for the BiscOT

thank you all right good morning uh thank you for coming to my session i know there's no others running at the same time so i don't have a lot of competition but thank you anyway uh today i'm presenting riscott for the biscott which i think may be the best pun i've seen in the talk titles so far so i'm gonna self-appoint myself winner of that slightly disappointed they didn't serve biscuits at morning tea but anyway we'll move on so as a bit of an introduction hello i am cairo i used to work as a cyber security advisor at one of the world's largest mining companies up until july of this year and it was my job there to assess

technology across both enterprise and the industrial or operational parts of the business i'm currently working for octopus deploy so i have moved from digging into the dirt up into the cloud if you're interested in devops and code automation you can probably don't talk to me because i don't do the code automation stuff i'm in the cyber security team i also have a cis and a bachelor of arts uh and if i seem familiar it may be because i have been involved with a whole bunch of acronyms which you can see up on the screen there so to get into what i'm actually talking about today i'm going to start with some really basic definitions just to make sure

we're all on the same page then i'm going to talk you through the controls that my team would be interested in when we were doing risk assessments on operational technology and then i'll take you through a slightly vague not proprietary case study from my time at my previous employer so i'll start off what actually is operational technology this is hardware or software that controls or monitors physical processes so this is what we're talking about in terms of manufacturing mining utilities it's not you know logging onto your laptop on your corporate network sending some emails writing some word documents playing in excel we're actually talking about things that control big crushes conveyor belts anything kind of in a factory setting

those kinds of things and also these safety and monitoring systems that surround them so there's a lot of quite dangerous things going on in these environments and there's people in there that need to interact with um that technology as well so there's monitoring and safety systems that look at things like air quality that look at seismic activity if you are drilling down into the earth and all of those kinds of things that are going to help keep people and keep the processes safe so this presents a slightly different challenge to traditional i.t generally speaking the systems are quite a lot older and they're also quite a lot harder to patch because they are usually ridiculously old

and also you've got really small change windows and you know if you want to take something offline for 10 minutes sometimes you're looking at 10 million dollars of lost production in that time so very very small change windows you're also heavily reliant on vendors in this space there's a lot of technology in this space that's basically built by two people in a shed who probably used to work for a mining company and thought i can solve this problem and they've built a little black box they've put their software on it and they stick that in your environment you probably don't really know what's on it and they're probably the only people that actually make that product so you

don't actually have a competitor that you can go to um and finally and this is probably the most important one when we're talking about the impacts of cyber security incidents when it comes to operational networks and operational technologies the impact of those incidents is quite a lot greater than in a traditional enterprise environment we're talking about millions and millions of dollars lost very very quickly and even more important in terms of people and safety your impacts are looking at injury or possibly even death so for example i did a risk assessment on a mine in mongolia it's an underground mine it has a shaft with a lift in it that takes people underground there's 200 people in that lift at any

one time and if something goes wrong with that you've got 200 people who could possibly die if something goes wrong so there's a little bit more pressure to actually do things right in these kinds of environments so what is a risk please do not actually read the picture i'm not a mathematician but i don't think that checks out um so in really brief terms a risk has three components you have an asset which is anything of value you have a threat which is something that wants to have an impact on that asset and you need a vulnerability which is a weakness so to steal a metaphor from casey john ellis an asset is your face

a threat is someone wanting to punch you in the face and a vulnerability is your inability to actually block that punch and when we're assessing a risk we're also looking at both the likelihood of a threat occurring and the consequences so this is how likely is someone to punch you in the face and how much would actually hurt and obviously that's going to impact how you judge that situation and what you do to avoid it so performing a risk assessment in an operational environment these are the eight domains uh that my team would traditionally look at um a bit of a disclaimer we don't we haven't actually based this on a specific framework or standard or anything like

that it doesn't map strictly to anything it's just what we put together and what we worked with and what we found was a good thing to do so a bit of a disclaimer there so i'm going to take you through the key controls for each of these domains and i'll kick off with network segregation and segmentation which is not a tongue twister at all so first up one of the major controls that we're looking at in an ot environment is actually having it segregated off from your enterprise environment and a big part of that is making sure and this sounds a little straightforward that all of your operational systems are actually on your operational network

it can be really easy to plug that stuff into your corporate network and then you know there's there's different considerations that you need to have uh and a an enterprise network probably has less of a consideration of up time it can go down without it being a big deal it's more of a big deal when something goes down that's an ot system so making sure your ot systems are on your ot network making sure your ot networks are not actually connected directly to the internet because then probably a whole bunch of you folks in this room would try and get to it usually you want to have access through a dmz or a data diode or a

unidirectional gateway rather than having a direct connection from your enterprise environment into your ot environment and going along with that node your home devices so don't go to all the trouble of segregating your networks and then plugging something into both networks and wondering why things go wrong uh the next step then is segmentation so within your operational network actually having segments within that network to separate things out that don't need to connect to each other if you actually have that in place then if you have an incident with you know it's let's say you get malware on your truck you don't want that malware to then get to your drill seems pretty straightforward so segmenting that out

with vlans having things separated out rather than just all in one big mishmash of a network always a good idea and finally one thing that we would tend to look for if you've got low value assets so like little monitors that are looking at things that you don't necessarily need to have real-time data on or that's not high-value data keeping those out of your actual internal network and pushing them onto public 3g or 4g even 5g now that we're all getting vaccinated it's going to make it a lot easier to keep those kinds of slightly dodgy maybe not the most secured sensors off your actual internal network and this is a big thing now also with a

lot of vendors when they create sensors and monitors they want to be streaming data out to their cloud service so it needs an internet connection you probably don't want that in your operational network so just push it out to the cloud stick a sim card in it let it send off to the vendors cloud and you're not going to have too much of a drama so domain two vendor and third party management the first one is making sure you've actually got endorsed remote access solutions to have your vendors actually accessing your network you do want vendors getting into your network because nine times out of ten they're the ones providing the support for whatever black box it is that they've

sold you and you want them actually patching and updating that and doing support when they need it but you don't want them just logging in you don't want to have teamviewer on all your engineering workstations that's not going to end well like a particular florida water treatment plant yeah yeah that was a good one again you don't want direct access to your operational operational networks um actually make sure your remote access solutions abide by your network segregation requirements don't have third-party devices connected to your operational network this is a really big one with vendors that want to come in and patch a piece of hardware they're going to bring in a usb that they've taken to 100 different

mine sites that they've probably plugged into their laptop that they're on while they're perusing whatever websites they do in their spare time and then they're going to bring that into your operational environment and plug it into your hardware that you need to keep your trains running generally not a good idea and this also goes for your internal staff as well because a lot of them will have a mobile phone that at some point they're going to want to charge and they're going to plug it into any usb that they can find regardless of what that actually connects to and finally all vendor and third-party access is monitored and logged this also goes for physical access so if

you have people coming on site in person actually doing work making sure you have them sign in you have someone accompany them and it also goes for their logical access as well i think a uh a few hackers that i know often like to use the phrase no logs no crimes you want logs so you can tell when there's crimes so domain three identity management and infrastructure access now a lot of this does line up with enterprise um access management so a lot of it's gonna be pretty basic access management the stuff we all say is really easy and then none of us actually do very well so first up you want to restrict physical access to your infrastructure

take a look lock leave approach don't leave doors to data centers open don't have you know engineering workstations out in the middle of a workplace where people can just log in and start messing with your trains all that kind of stuff and then your basic access management so only grant access to those that need it remove it when it's not required review it on a regular basis and then things get a little bit trickier so shared accounts we all know we want to avoid them where possible in an operational environment there are often legitimate needs to actually have shared accounts one of the ones that you know i would see quite regularly is a lot of the

safety monitoring workstations where people are watching trains down the rail corridor uh it's not actually possible for those people to at the end of a shift log off and have someone else log on because that means there's that window where no one's actually monitoring that system so those people have a legitimate need to actually share an account so where you have those legitimate needs and make sure you double check them because people will say they need it and they probably don't but look at stuff like limiting the access of the shared accounts make sure those shared accounts are locked to that particular workstation so they don't have a dynamic login that can be used elsewhere

make sure you have those shared workstations in a physically separate part of the building and make sure you're actually logging staff who are on shift so that you can attribute any actions that are taken on that account at that time passwords i'm not going to go through basic password requirements because you folks should hopefully understand them but one of the big things is changing of default vendor passwords in a lot of operational technology contexts vendors won't actually let you do that and it will be a violation of your support contract if you change that default password because they're probably lazy and they want to be able to log in with the same password to all of their customers

so double check if this is in fact a requirement of your vendor if it is in a support contract try to get it taken out and be able to actually change that default password you also want to have separate identity management to your enterprise environment so don't go to all the trouble of segregating everything out and then have one active directory managing everything have a separate active directory for your enterprise environment use a centralized password management platform for your system and your infrastructure credentials and also make sure your service accounts are non-interactive because they should be for services and not for people so number four configuration management and hardening so this is kind of a mishmash of

different things um you want to be looking at limiting and restricting removable media so that usbs people are plugging in you don't want those you want to make sure you've got version integrity monitoring for your plc's you want to disable services and components that you don't actually need again that's some basic hygiene stuff make sure you're not storing credentials in clear text particularly within config scripts and that kind of thing if you can have a standard operating environment that you use for everything in your environment that way it keeps it nice and consistent you know what's happening you don't have to actually think about what's switched on for each thing disabling insecure and legacy protocols

and use secure alternatives where possible so we all know this ssh for sftp not telnet and ftp this one's actually a really difficult one to do a lot of the time so if you can do it good job i envy you and finally where you do have exceptions to a lot of these things because you probably will particularly with those legacy protocols actually record them weird idea have some governance around it actually know where they are keep track of them if you can get rid of them down the track then do it but you're not going to be able to do that without actually having a record of where they are and why they're there

so this is some pretty basic threat and vulnerability management stuff but it can be quite difficult to do in operational environments because you are working around all of that segregation as well so have monitoring in place for your mission critical ot systems um there's actually a bunch of vendors these days that sell appliances specifically designed for operational networks um so you've got nozomi clarity dragos are all selling that kind of thing so those are specifically designed for those environments which makes them a better product they're not designed for an enterprise environment then you just stick it in there and hope it picks something up patching again quite difficult when you are not connecting to the internet and

you're trying not to implement too many changes and you have very small change windows but patch everything that you can and also make sure that your vendors are patching the things that they're responsible for because a lot of the time they will try to get away with not doing it and also make sure you're in a lot of cases you will be reliant on vendors to actually test that whatever they sell you actually works on a new operating system make sure you're working with them to make sure they do it in a timely fashion and you're not sitting around waiting for them to do that testing before you can actually roll that patch out and finally actually deploy antivirus

and make sure you do update it and have some central management within your ot environment again not from your enterprise environment i'm going to keep hammering that point home so end of life systems actually identify them because i can guarantee you they are there they are 100 there they're hidden and in a lot of places you might not be able to actually upgrade them so actually identify them first to start with make sure you have plans to actually replace and upgrade and actually put those into the budget i know a lot of us don't like having to do the management side of things actually you know add the numbers up and see what you can afford to do

but you need to do that for these end-of-life systems because otherwise you're never going to be able to get away from them and finally where you can't and i can guarantee you there will be ones that you can't because that particular vendor only sells that product to run on windows 98 and they have no interest in upgrading it so you're stuck with it actually look at what you can do to act to limit its exposure so have extra restricted access have targeted monitoring extra extra extra isolated and segmented away run it in a vlan whatever you can do to actually try to put some extra layers of protection around it so resilience and recovery here we are

looking at single points of failure there will be some try to make them not be very good advice make sure you actually have processes in place to perform and test backups and this should go for both data and for configuration so if you've got system logic or plc logic converter firmware make sure you're actually backing up the configs for those devices because if you have to configure them all from scratch you are not going to be having a good time so actually test that those backups will work make sure you've got bcp and dr for ot systems and actually test them which is the step that most people forget um and actually run through those tests

and then work any findings back into your plan and finally make sure you actually know who to call which isn't ghostbusters in this context if there is actually an incident i mean maybe you've got ghosts in your ot network i don't know but make sure you know who to call the thing with a lot of these ot networks is they are very physically isolated a lot of them are out in the middle of nowhere i don't know how many of you have been to the pilbara but it's a big red nothing with a mine in the middle so make sure when something goes wrong the people out in the middle of nowhere know who to call and if you're in the

office looking out make sure you know who's on site who can help you if you actually detect something because getting to talk to the right person is probably 90 of your battle in a lot of these situations and finally i know the governance piece is all of your favorite one um so actually keep track of some stuff have an asset register have a vendor register make sure you have contracts in place include cyber security within your contracts include ongoing support and maintenance in your contracts make sure you actually tell your vendor that you have cyber security requirements before you engage with them and then make sure that they can meet them and then come back later and make sure

they're still meeting them and make sure you have processes to capture any exceptions so where things can't abide by the best things that you want to do keep a record of it come back to it fix them later if you can put extra protections where you can't and make sure people actually know about them because the worst thing is is not actually knowing where you have any of these vulnerabilities you can't secure what you can't see so my somewhat vague case study i have taken a couple of different risk assessments that i and my team worked on while i was at my previous employer and i've combined them into something that's indicative of what we saw but isn't

giving away any commercially sensitive or proprietary information and there is someone from my former employer here and i can see him watching me so i will be careful but this is very common and this is what we were seeing in basically everything that we did so let's say hypothetically you had a drill a very very big drill in the middle of nowhere and this is a very fancy very expensive autonomous drilling system so there's a few things that go into an autonomous drilling system first of all you have your big big drill then you have a human operator who is controlling that drill and that human operator is probably thousands of kilometers away from the drill

is controlling it through a computer and is probably controlling six to ten drills at any one time so for them to be able to do that there's an on-board system on that drill that tells the drill what to do from the operator you've also got your control room where your operator's working and ideally you have a backup control room and then you've got connectivity between all of those things so what are we concerned about with this setup well first of all drills are very big and very scary the internet is also a very big and very scary place so if you have a drill that's controlled by the internet you can kind of see where i'm going with

that and there's some things you need to consider so for example when you stop drilling you stop making money and we're in wa i don't need to tell you how important mining is for our economy well i'm not going to get into that but if you lose one week of production it's actually going to take you more than a month to make that up and that's actually millions of dollars i mean if you think of the price of iron ore and how many millions of tons of iron ore we are shipping out from wa every day it adds up pretty quick and those big big drills are also very expensive to repair if something goes

wrong um so some of the parts on that drill cost three hundred thousand dollars just on their own and that doesn't include the cost it takes to fly a technician from perth out into the middle of nowhere probably drive a couple of thousand kilometers into the middle of nowhere and they're going to charge you quite a bit because they don't want to do it um so that those dollars add up pretty quick as well so let's say you did a risk assessment on an autonomous drilling system these might be some of the things that you would find as part of that risk assessment so for example in the category of segregation you might find a stray corporate server

has direct access into your ot network and i'm pretty sure i've said 27 times already that that's not a great thing you might also find that some systems that are responsible for software configuration or change management or infrastructure management they're actually located in your corporate network and have been given direct access into your operational network you might also look at the firewall rules that you have and realize that some of them are quite permissive and they're not actually reviewed very regularly so moving on to segmentation you might look at your ot network and realize that there is a not a lot of segmentation actually going on there um so any of your systems and devices

can communicate with each other even where they don't need to so as i think i mentioned earlier if you get malware on your truck it's very easy for it to spread to your drill and no one wants a malwared drill if you were to look at access management you might find that you have some backup workstations that are left logged on at all times they're always unlocked they're unattended a lot of the time so anyone could wander in you might have a physical estop switch which is your big red button that absolutely stops everything in the case of an emergency that might be left unattended and it's pretty easily accessible as well you might have a lot of vendor default

passwords in that environment you might have some service account cred stored in clear text i don't need to tell you folks what that can be used for and you might also find credentials written on whiteboards and on post-its within your control room and you get bonus points for that one if there's cctv pointed at those whiteboards and post-its as well so you're probably gonna have some bad security hygiene you haven't been washing your hands for 20 seconds so you've still got some legacy and insecure communication protocols there's going to be some telnet some ftp some http you've probably got a legacy ad integration method lurking in there somewhere you're probably not great in patching and you probably don't have any

antivirus on those servers and when it comes to actually looking at what's happened and what's been happening and what has happened you probably have no idea because you haven't been logging and you haven't centralized that so you don't amalgamate all of that data to give you the full picture and no one's really looking at it anyway so once you know all this what do you actually do uh the simple solution is to fix it um and if you go to some stakeholders and tell them that they're probably not going to give you a very uh happy response so it's not actually as easy to fix a lot of these things as we might think and a big long list of to-do's for

people who are already overworked and admittedly they're probably fairly well paid because i'm talking about the mining industry but they're very busy uh and they're just trying to keep the lights on so having these security folk roll in and go actually you suck here's some stuff to suck less uh isn't gonna go down very well but you can actually do some things um so let's say we start off with the process side of things we can probably put in some processes for account reviews we can actually do some firewall rule reviews and actually fix them once you you've reviewed them and pointed out what's wrong you can change your process so that you have password updates when you are

commissioning new hardware you can start doing some patching within your ot network within those tiny change windows that you have you can go and mop up all of those default vendor passwords that you can you can roll out something like cyberark for password management you can get rid of that naughty legacy id integration module fix that one up you can deploy antivirus and you can start doing centralized logging and monitoring with one of those um products that i mentioned earlier so you've done all of this stuff and things are actually looking a bit better but what's actually left network segregation enforcement very very very tricky that's why i keep saying that you need to do it because

it's actually really hard particularly if you haven't done it from the beginning going through and clearing out all of those connections and re-engineering existing solutions so that they don't need those connections is a very messy and expensive job so you've done some of it you're doing well but there's still a lot left to do and if you haven't done that segregation piece the chances of you actually getting to that segmentation piece are pretty low and removing all of those legacy and insecure protocols you're out of luck on that one it's just not going to happen um so this is what got done in three years and there's still probably another three years of work left

so what did we actually learn during this process well number one security is hard particularly in large and complex environments but there's really only one way to deal with that and a good friend of mine often says you just have to eat the elephant which is just start eating like bite by bite you'll get there eventually the second thing i learned was that getting to the right people in your organization is half the battle and when you have an organization that's about 50 000 people and you are spread across the world and you have a division between office people and site people it's even more difficult to actually find who you need to talk to

get them on board and actually start getting this work done but once you find that person and there's always the person they know where all the bodies are buried they know where all the connections are that don't need to be there once you find that person you get them on board happens and it does happen and finally sustainability really is the key to a lot of this stuff there's no point actually fixing a problem if you haven't gone and actually created a process behind it that will actually stop it happening again because unless you're a contractor who wants your one-year project to be renewed at the end of the year you don't want to get to the end of

fixing all this stuff and then realize you have to go and fix it in all the other places that it's happened again so actually making sure you have those processes to stop it happening in the future drawing that line in the sand and saying we're not going to do it like that anymore and then we're going to mop up will actually mean that you have sustainably fixed that problem so that's the end i don't think i'm going to have time for questions oh cool i do have time for questions remarkable that was a very quick quick hand up

specifically for operational technology there is actually um if you're looking at mitre there's a specific miter for operational environments which i find really handy um i'm not a huge fan of the stride stuff but i just kind of mash everything together to be honest i find that's what works for me um but mitre for the operational side of things is probably the best bet

yeah it's it's pretty much the same so operational technology and ics's industrial control systems so operational technology is probably more of an umbrella term and under that you would have ics and then you know different other forms of technology but the terms generally are used fairly interchangeably and there's a lot of other terms some places will call it industrial and operational technology some places call it production technology it just depends what vernacular you want to lean into yes

not well to be honest there's a lot of it's not a perfect solution um but it's better than not having absolutely anything to be honest um yeah yeah um but yeah there's things can go wrong but for the most part it if you've set it up correctly and you've done your testing you should be able to run it there's a guy over there who can talk to you a bit more about antivirus and operational technology okay all right lucky last right down the back

satellite

i don't think there will be a move just because it i mean once it gets cheaper probably but what a lot of um those companies are doing is just building their own mobile networks at the moment so they'll have their own one they're not working through telstra or anything like that they'll build that out in the middle of nowhere and that's what they use and sometimes they won't even connect back to the internet it's basically just a lan that runs on 3g or 4g but hopefully you know if satellite i mean if elon musk brings us all the cheap satellite he's promised it might be a proper solution going forward but i don't see it happening anytime soon

cool all right thank you so much everyone i'll be around the rest of the weekend

thanks cairo