Batch & Breaches in OT

okay ladies and gentlemen welcome back to the second session uh of today's bides conference and I'll pass you straight along to buan Alika vasovic who will talk to you now there'll be time for questions afterwards if you have questions please raise your hand if if you don't have a question now you can always reach boan during the course of the day okay thank you John for your introduction uh thank you very much hello to all I'm not fluent with Hungarian you know Budapest and I've learned uh during my visits to Budapest I've learned the most important words on Hungarian such as gulash langos kosach sure thank you okay uh today I'm talking about batch and breaches in OT

environments I will share some of my experiences in the last few years and of course you can ask the questions during the presentation of or after the presentation I will be around so let me introduce myself at the early beginning of the presentation it doesn't work Wonderful

H I know proper Hungarian word for this situation [Laughter] but okay I don't know I don't see I think I'm not so lucky today what give me a few seconds okay oh oh finally okay I currently I hold the position of principal cyber thre intelligence in in Figo company in Croatia uh and most of my daily tasks are related to intelligence analysis sometimes to reversing various kind of stuff and incident response most usually as a supporter not as a in leader and you can find me and contact via LinkedIn this is my profile feel free to ask any questions to send connection invite Etc so I will be I will be happy to answer all of your

questions okay so let me introduce you to the today's agenda I will talk about IC environments I will uh talk about you know just from the just I will mention just the basics about the industrial control systems and I will guide you through or lead you to my experiences from incident response in such environments uh and I will uh also uh go through the brief history of Windows operating systems so today's lesson will be also a history lesson for younger generation in the audience but don't go please don't go I I hope it will be very very useful for you and for all older generation Generations in the audience will be something that they already know I hope

so you can use also command line CMD in Windows to scope in incidents this is especially useful if you are in a very closed very strict environment where you cannot install anything you want during the incident and this is pretty characteristic for OT environments it's quite normal okay okay so let me introduce you into the industrial Control Systems just few words and comparisons between between it and OT systems if we talking about it systems we are talking about data protection you are protecting the databases documents file shares uh something some stuff in the cloud Etc but if you are talking about industrial Control Systems you are protecting the process every single process production process for example for example prod

producing the electricity is composed is comp ation of uh many uh micro processes which are and they are uh interdependent inside of the very big one very big uh process so this is the most important this is the ultimate goal to keep the process 24/7 fully functional in it environment for from the defender perspective if you look look at the network logs network uh user Behavior Etc you w't be able to uh predict some patterns such as internet surfing Etc but if you look at the uh network connections and protocols and the patterns inside of OT Network it's very easy to predict uh uh every single connection every single uh Behavior cavor because everything is programmed

exactly to some rules you know and it's very easy to monitor such systems it's very easy to detect anomalies in such systems regarding the hardware upgrades it's quite normal in Enterprise systems in it systems to upgrade Hardware work station service every three four five years it's quite normal but it it's abnormal or it's not usual to OT systems to upgrade every s every single year or every three years uh the complete Hardware it's it's uh it's unimaginable uh because you have to plan all whole the process of replacing the hardware software everything because we are talking about processes 24/7 and high available and high reliable everyday processes right and you can the the final conclusion

will be that you can uh see frequent change in it it's quite normal on every on daily basis but in OT you won't see uh frequent change on uh hourly basis or daily basis Etc okay and of course all devices in in it nowadays are secure by Design at least most of them right but in OT it's quite normal to see unsecure by Design devices protocol without authentification uh plain text protocols uh devices without any kind of login forms you can directly open via browser uh controls uh control panel via Ethernet Network and you can change anything you want it's quite normal in OT systems okay and uh the first rule that I've learned during my experience watch don't

touch you can watch anything you want but don't touch don't touch it and if you've been to the museum it's the same approach and there is another one uh connection between OT systems and museums OT system is a museum you know because you can find very old operating systems very old uh devices still working nowadays properly without any any failures and in OT systems the final conclusion is it's everything is uh it's all about availability okay uh does anyone anyone here you know audience working with OT systems one two three okay wonderful okay so up time is measured in years do you do you agree yes I've seen devices with up time more than eight

years without restarting without shutting down shutting in the meantime so this is pretty pretty good okay so let me let me go through Windows Bri briefly to Through the Windows history just to remind you uh about major updates or major uh functionalities introduced by the Microsoft through the years in the last 25 years if you're talking about uh Windows XP almost uh 20 24 4 years ago uh we are talking about introducing the firewall for the first time as a host-based firewall on Windows operating system only in One Direction you can configure XP Windows XP only in One Direction right uh and we are talking about NET Framework introducing the dot NET Framework version one it's pretty pretty

old and shitty version but it works right okay and few few years later uh Windows introduced uh Windows Vista and Windows Server 2008 with firewall look at that with firewall denying or allowing connections the traffic in both directions so you can configure firewall finally in outgoing and ingoing Direction on host base on host basis right they introduced Integrity levels uh UAC Powershell 1.0 this is the first operating system Windows operating system with a Powershell etc etc so Windows 7 the 2009 in Windows 7 uh you can find the newer version of pow shell pow shell two and Windows 8 was the pretty uh pretty different at the time was the completely different operating system from the user perspective definitely

compared to Windows 7 uh with a lot of security uh builted in features such as Windows Defender Etc and I won't go to the all the versions because it doesn't doesn't matter so much what I want to tell you and I want to show you one case this is it was the first case in OT at least from my perspective I will introduce the case it's uh it's a realistic case Okay I uh I U make a name AV driver anti-s driver because attacker took the the driver of avast anti virus as a as a attacking tool to stop antivirus in whole system it was uh back then I don't know five years ago it was vulnerability

in Avast operating uh in avast antivirus systems so it was very very easy to stop the antivirus on the computer okay so let me introduce to the case narrative in the beginning put yourself in uh in early beginning of uh of this incident so my colleague one one more morning my colleague incident leader called me and he said we have an incident very serious one without serious impact on environment but we know that in this moment the attacker is inside of the system doing some stuff connecting on computers the problem is that we have few hundreds of Windows servers with different versions of it you know from Windows Server 2003 Windows Server 2008 even the patch level was

different Etc and all of them were old and unpatched which is horrible without most of them without uh antivirus protection without any kind of protection on it you know just uh plain default configuration with install with installed software such as scada or or HMI to control the system inside of the very big industrial control system in Croatia and uh we spoke with the technologists and administrators and they said uh immediately shutting down is not an option you can install anything on it and restart the any of those servers because there's no impact in this moment and we don't we we we want to avoid any side effects of your incident response in in in our environment which is quite

normal for OT you can expect this on every single incident response uh engagement installing anything is not an option either if you want to install super modern EDR on very old operating system such as crowd strike it won't work in every single situation and it's it's unimaginable because you are interrupting you can interrupt the normal ongoing processes inside of the operating systems which are responsible for microprocesses inside of the OT Network okay remember the first rule watch don't touch and you are incident responder there are some expectations you know from the customer side and you don't you don't have much options and especially overloading anything in terms of CPU Ram inside of the Windows operating system

is not not an options you know if you if you overload because you install something if you or if you uh took the tool for forensics and you overloaded the operating system it will cause some damage okay so after the first two hours of incident we uh we SP we we discussed the findings initial findings inside of the uh system and we found one infected computer with very specific indicators on it after a few minutes we found another two computers uh two computers um in in the network and after performing some forensics reversing disc discussing internally we uh we have finally we have a list of the trade craft of the attackers trade craft you know uh all

the indicators very specific indicators IP one IP address uh three or four uh file names on exact locations installed on the exact locations on the every single Windows operating system and it was enough it was enough for us to inspect all computers in the system but how can you inspect and find indicators without any tools right it's pretty hard especially on older versions of of windows but we have Central provisioning system such as Microsoft s CC CM fortunately it was great great news for us because from one point we can distribute anything to every single server inside of the um inside of this big industrial control system on every single GE location remotely without going to location you know and checking

manually infections artifacts Etc so I came to to I came up to a great idea let's build a bot script and use natively builted in Windows tools to detect malware and to distribute this information to uh data collector inside of our environment so I will show you step by step the main functionalities of this script and um the way you can use um regular Windows commands in the uh that are built in in in the last 25 years in the windows so uh everything is the same for every single Windows uh operating systems system from XP till Windows 11 okay it's very very very simple but if you use them uh in more creative way you can help yourself

during the incident okay okay so uh if you want to search for specific files on specific locations inside of the windows you can use exist okay and you can put I don't know 10 files 10 pads all you have to do to put space in between them and this is like a operator or either this one or this one one file okay in case you the script founds your your file on the operating system you can use variable to increase the number of found indicators okay this is a this is the variable that I used uh in in the script and the second variable will be concatenating variable with the letters inside of it is a concatenating

string variable indicating that I found the file on the this system okay we found the aast dll on this path on the system root path okay in case of finding that the the script will add F letter capital F letter at the end of the string okay if you want to search for network connections you can use netstat this works on Windows XP exactly the same way as in Windows 11 on Windows Server 2003 on Windows Server 2008 Etc that's why that's why I took very simple commands Okay the story here this U uh this part of the code is the same as the previous one from the previous slide okay you can find if you in case

you find uh IP address we will add we will concatenate another one letter signaling that there is a network indicator on this computer okay you can find also specific ports instead of the IP addresses you can put 100 IP addresses in inside of the inside of the F string command it's very easy of course you have to catch the established connection right on the PC at the moment of of uh of valid and fully functional TCP connection on the on the um on the U infected computer okay IP config with ip config you can look at the DNS cache just displaying displaying the DNS in with this syntax and you can search for very specific internet

addresses as a indicator and is the same approach you can add one letter as a indicator that you found that the script found uh in this specific indicator on the computer okay task list you can list all the uh processes currently available currently running on the operating system is the same on all windows versions it's version independent and you can search for very specific process names with the schedule tasks you can look at the configured schedule tasks on the Windows operating system it's very simple command nothing super special without any philosophy right okay if you use Query for uh uh query and fo table uh sorry uh Flags you can list in very very readable and very uh clear uh shape

all the all the uh configured uh uh schedule tasks and we are searching for very specific super super specific name of the schedule task on this computer okay wmic it works on XP perfectly but it works with very narrow set of commands and this is one of those commands that you can use it's prod get name version and you can list all installed application on the on the on the computer unfortunately for example with wmic if you run this command with usernames something you can list all the configured usernames on Windows 11 but it doesn't work on Windows XP this one works on Windows XP and uh as I said already we can add in

case we we found U indicator specific indicator we can we can add this U append capital A as a indicator as a descriptor or something like that in this variable I will show you at the end of the of the of the program or the script uh detection variable and uh its usage okay so with net start you can find all configured Services is the same for every single Windows operating system this is very important so you don't have to you don't have to deal with uh various kind of versions Etc you can query Windows registry also with requery it's a builted in tool nothing super special about it okay q w insta Q WIA it's a very simple

command in Windows if you type in this command you will see currently connected users via RDP it's very valuable information during the incident if you want to catch the attacker and his his activities inside of the network and another one net user you can list all the users currently configured local users on Windows operating systems as I said already you can use wmic unfortunately but it doesn't work on very old operating systems it works it works on uh on brand new windows versions such as Windows 10 Windows 8 uh Windows 11 Etc but it doesn't work on server 2003 because it's it's very very uh simp uh uh the wmic it's uh installed and configured in very simple form in in

those uh older versions okay wonderful the script found a suspicion in suspicious indicator so we took the script with uh specific configuration in it we distributed the script through all the system on few hundreds of servers but the question is how can you uh how can you collect the feedback from the servers let's call the feedback you know okay the first option is to in case of finding the indicator you can ping the infected machine will ping your computer inent respond computer inside of the network wonderful but how to differentiate all the pings from Network scanners Etc you can play with time to leave indicator and you can search for time to leave lower than 30 for example okay okay

that's how you can differentiate infected pinks from healthy pinks inside of the network another one option is to send DNS query with a host name in it as a subdomain with a host name of infected computer you can ping computer without installing any tools you can send DNS query without installing any tools and the third option you can send s a very simple txt report via FTP why because you don't have to install anything on Windows XP Windows 11 Windows Server 2000 to use FTP client it's built it in if you want to use for example tnet you have to install it if you want to use SSH there's no SSH client on Windows XP there's no such

thing okay you not you are unable to utilize such applications okay so you have to send some kind of the beacon to in that respond computer from infected machines right and this is in this is a conditional part of the script one just one part if we if the variable counter is greater than zero then send then ping this IP address this is IP address of incident responder five times just to differentiate pinks from all others machines all other machines from the network and ping with time to leave with value 30 or lower it depends on the network of course if you know how time to leave Works inside of the network uh you know why it's important

to track lower uh time to leave where values so this is very specific kind of ping with you using natively builted in tools inside of every single Windows without Powershell without you know super super scripting engines just with batch scripting okay so if you want to send DNS record from the infected machine you can use NS lookup is builted in application inside of every single windows and you can send automatically from infected machine um internet or sorry internet addresses such as these but let me explain little bit further so here you can put inside of the the subdomain you can put as an info computer name of the infected computer in the sub inside of the subdomain and

you can put all the capital letters inside of the subdomain as indicators as a let's say a description of indicators that are found on the infected computer and you can use your domain company's domain I uh I adise you to use you know privately registered domain not your company's domain because if you are running this script inside of the network where attacker is currently connected and uh he see uh he see your um your domain that's that's pretty pretty bad you know so uh the the most important thing here you can register domain and you can put a subdomain such as call collector or something else I don't care as authoritative server so put your IP address public IP address of

your company's DNS server publicly anywhere in the world but put your subdomain of your of this domain as authoritative DNS server this means that if someone anywhere in the in the world sends uh any kind of DNS query with this subdomain and your company's domain always in every single situation this DNS query will reach your DNS server anywhere in the world okay it's very simple trick and uh this number here is random number its purpose is to avoid DNS caching inside of the companies what is DNS caching if you have uh network of end hosts and servers and in between there is a proxy and here you have DNS local Enterprise DNS server DNS caching will be cache the the the

most common DNS queries inside of the proxy you know to avoid a bursting of DNS queries towards the your corporate DNS and if you want to avoid DNS caching if you want to make sure that every single DNS query will leave your company where you testing the environment you can put a random number and that's how you can avoid DNS caching I mean DNS caching is good good thing for everyday use but for this scenario is pretty pretty pretty bad because if you send multiple DNS queries in very short period of time from the same server proxy will stop your queries because it it has already uh memorized the the the translation proper translation of your query that's why if

you put the random number every single DNS query will be different and you can avoid you can bypass caching inside of the Proxy Systems and that's good this is the functionality that we need that we need in in such uh approach okay so uh I defined few ioc's just uh to show you how script does work in OT environment so the uh those indicators that you see currently on the on the this on the slides are the very specific uh indicators for agent test light to Cent Tesla as a sample sample scenario for this presentation so we are searching for very specific file name we are searching for network connection towards 58 uh 87 Port which is specific for email

translations email uh um email communication we are searching for DNS record such as these SMTP agent blah blah blah something this is specific for this kind of malware we are searching for very specific name of the process and u persistent mechanism inside of the registry okay so let me show you the approach give me a few moments

is it visible okay so this is the computer this is clean computer without any infections on it okay number of indicators found is zero in this moment I double clicked on the agent Tesla malware we are waiting for a few moments to make sure that everything is told properly from the perspective of malware such as files registry modifications Etc okay and then I will rerun the same script and you can see number of indicators found file network indicator registry indicator process indicator sending DNS Beacon from PC blah blah blah to my domain I'm I'm listening for DNS beacons sending ICM Beacon to to local IP address of incident responders computer with time to leave value 30 or

lower it depends on the network topology and uh let me reverse okay and sending results via FTP you can use of course only DNS you don't have to use all three um possibilities you you you can use all only DNS or or only icmp for for sending uh beacons you know confirming I'm infected this is the DNS record that I'm sending you to let you know that I'm infected this is the the purpose of of it okay and uh if we look at the process name it's very simple approach you know we can use process hacker or task manager we see that the malare is here active if if you look at the configuration the the most common

configuration for persistance inside of the registry is automatic run to survive reboot of the Windows operating system it's very specific for this malware the script already found all those uh indicators inside of the computer this is the exact location and nothing super special about it okay but from the side from the incident uh responders side let me show you we are waiting for beacons from infected computers okay on the top screen you can see fake DNS waiting for DNS queries and on the lower screen you can see TCP dump filtering only icmp uh icmp Echo requests with time to leave 30 or or lower and I will in the meantime I I uh starting the script we have to wait for few more

seconds okay this is the domain from the malware okay this is how it looks like from the incident responder side you see five five pinks from an infected computer and very specific DNS requests that came from one company to my public DNS authoritative server and this is the uh folder from my FTP server containing the very short report from the FTP stating all the indicators that are found on the remote computer everything is automatic written in batch with without any without any Powershell without using uh any Powershell so let me show you let me show you the hex value that I mention let me translate it

okay okay you saw the hex value inside of the subdomain if you translate it to the aski format you will see the host name of the infected computer and you will see the capital letters concatenated inside of the one string and uh configured to send DNS sarey as a subdomain so that's how you know the host name of affected computers that's how you know um indicators that the script found already on the computers and from my perspective it took me I don't know 4 hours to build the first version of this script to test inside of our laboratory to make sure that everything works exactly the same on every single windows from the last 25

years and and we distributed this script by using Microsoft SEC M provisioning system and after just 15 or 20 seconds we received 18 computers without with zero zero false positives sending DNS re uh DNS queries to our DNS server sending pinks to incident responders IP as address inside of the infected Network and that's that's pretty you know pretty um I love to say pretty you know what is kiss keep it simple and stupid without using any super Advanced tools you can solve very I I will not say very complex problems but if you have such arsenal of tools at your disposition you can use them wisely to detect Mals to detect any kind of U suspicious

activities inside of your network without installing anything without uh without uh CPU or Ram uh molesting which is especially uh uh uh let's say dangerous in in OT environments and if you want to download this B script you can download down uh it on you can find it on my git you can play with it this is not super special script super you know uh super Advanced finding zero days Etc it's very simple very very well and very structured written script that you can use for your environments without any danger dangers that you will um you will um crash something you will impact on your systems and um I think that's it I mean thank you thank you for your time I

think I'm uh right on time yeah I finished right on time uh thank you very much for your time thank you very much for your attention uh and that's it for any questions thank you uh buan um do we have any we do who's over here uh what tools did you use before writing this uh B script sorry uh what tools did you use before uh you wrote this uh batch uh file um for incident responses uh not pet Plus+ ah and my knowledge I gain my knowledge about these tools I mean this is nothing super special you know if you ask any older system administrators they are familiar with those tools unfortunately newer Generations in it doesn't know all those

functionalities because they use Powershell I I mean it's quite normal you know but you don't have to use any kind of super specific tools to program bash script notepad Plus+ is enough yeah but I mean uh what like uh before you had the script uh what tools did you use to uh respond to incidences what what tools yeah like did you just issue the same commands without making this this uh script uh you mean in inside of the OT incidents or oh yeah I mean for every single incident you have different situation because every single OT system is a little bit different than others you know I I've seen for example uh environments where you can

find on every single workstation or ser EDR Solution that's perfect I mean that's super super from from the the from the perspective of incident responder or sock analysts and you don't have to develop such scripts it's stupid to to to waste time developing such things you know uh I also have experience for example in uh environments with Alan Bradley systems hman systems and it's perfect from from the perspective of monitoring Network because you can replicate the network connections on your uh computer you can use for examp for example cyber lens this is free tool and it's wonderful tool to visualizing all connections inside of the IND industrial control system you can use it on the Fly you

know just connecting your your PC on the switch you don't have to have uh uh span Port as a port mirror to mirror all the traffic inside of the network because there's there is one very specific uh the there uh most of the protocols inside of the OT networks such as ethernet IP it's not related to ethernet it's ethernet industrial protocol or bunet or I don't know uh um there's a ton of Step seven Etc they use natively multicasting to communicate between just two devices it's stupid approach but as I said at the early beginning of this presentation unsecure by Design those Protocols are written by guys originally or that came from OT networks not from it and sometimes if

you connect your PC in switch in the ethernet switch inside of the industrial control system and and you open wire shark you will see a lot of broadcasts and a lot of multicasts and you can map just in first five minutes you can map a lot of Ip Al live IP addresses inside of the network especially if you're talking about Windows Windows has mbns multicast BR protocol and llmnr translation protocol which is a newer version of uh uh mbns protocol supporting IP version 6 Etc so if you just listen inside of the network uh with cyber lens or with wire shark or with TCP dump you can gather all IP addresses just by listening without any scanning without any

additional configuration that's OT unfortunately thank you yeah okay uh if there are thank you for the question if there are any more any further questions please reach out directly uh to buan either during lunch or anytime during the rest of the day once again boan thank you very much thank you kissing him