Living on the Edge: Evicting Threat Actors from Perimeter Appliances

and why and defense tips. >> And a round of applause, please. Thank you. >> Thank you.

>> That's one to four. >> Yeah, perfect. Thank you very much. So, thank you all for being here for the last talk for today. Uh it's been a great day. Great a lot of great talks and we have to make a great finish for the end. So my colleague M and I want to talk today about living on the edge evicting threat actors from parimeter appliances. And first of all before we go into depth um and talk about some forensic stuff and some exploits and what's happening right now. Uh we first need to define what exactly edge devices are. So we all know what what we are talking today and basically what we're referring to is edge device is something

that sits at the on the edge on your of your internet between your company and the internet something which is reachable from the internet like it should be uh like a parameter firewall for example it should be reached from the internet obviously like virtual desk environments asset managements and stuff like that so there's a lot of things which are considered edge devices and additionally we're also considering edge appliances so appliance again something which you get from your manufacturer like a box which is plugged in which works and it does its job and it's secure more or less and to put it all a bit more into perspective um what we want to talk today about is edge devices

exploits and initial access so basically what you can see here is a slide from Florian R you maybe know him and he summarized all the common entry vectors so on the top you can see fishing I think you all know what fishing is at the bottom We can see brute force like passwords guessing credential stuffing all stuff like that but we are targeting now the middle box the exploitation and to be precise we're targeting this middle box of vulnerable appliances like Ivanti Fortnet Cisco and stuff like that there are some other names we're going to talk about them in a second but this is the focus of our talk today so how can you get as a threat actor initial

access via a vulnerable appliance Thank you very much. Okay, let's first establish some context on um why this topic actually matters. Um first from the perspective from an attacker. So uh edge devices are usually quite easy to compromise. Uh yet it is difficult for the defenders or the the victim to detect the actual actors uh to internal resources uh and that they're present. Also a big issue with these edge devices that their operating system and software are very outdated. Um I brought an example from Ivanti connect secure which is a VPN appliance. Um it has a pearl version of version 5.6.1 32bit which was built in April 2001 and I'm pretty confident there are people

here in the room that are younger than this. Uh so that tells you everything I guess. Uh also most of these devices lack basic exploit mitigations. Um so this is far off from those mitigations that we see on modern desktop platforms. And lastly, these appliances are extremely volatile. Um a reboot or software update could wipe all forensic evidence that an actor even accessed the device um any time in the past. And forensics is very challenging and timeconuming on these devices. So yeah, that's why we're here. From the defender perspective, uh from our point of view, it's an imperfect target um because we usually don't have file system access. Uh some vendors even encrypt the data addressed. So that disk

forensic is not an option most of the time. Um and in that in that case normally the OS resides in memory. Um, that's why the artifacts are so volatile and also most vendors don't allow direct operating system shell access. Um, so you get these weird challenge response shells where you have to contact the customer support first and they peer over your shoulder so you don't do naughty things with the appliance. Um, which is an issue as well. Uh, dumping memory therefore is also pretty difficult as well. Um and most vendors use some kind of proprietary encrypted form um in case you can do dumps. And the last problem are the trust me bro inte integrity checking tools. Um

Ivanti is a big offender in that regard. Um basically they have a tool running on the appliance itself uh which you can tell to run an integrity check. you have to trust the appliance that it actually does that. And even if you get an resolved, you can't trust that because it's running on the compromised device itself. Um, and this tool is mainly implemented through checking hashes and some sanity checks on the file system. So, it's not really a good indicator that the device is clean or compromised. Uh, and even if got that message somehow somewhat um and it was kind of a mind-blowing um discovery for them. that some fair actor has actually actively manipulated the tool. Um yeah so there's

that right so now we come to our four uh main actors for today uh that'santi fornet citrix and cisco uh in no particular order but um somehow most of them are uh involved in some way if we are called for an incident so yeah let's start off with the case one uh that's Cisco IOS XE and we'll take a short trip back in time uh to late 2023. Some of you might say, "Okay, that's a bit far off. Um might be old news, but trust me, it's still very recent news." I will come to that in a few slides. Um but basically back in 2023 there were there was one or two we didn't really

know vulnerabilities in Cisco IO6E devices and Cisco published this to generally raise a awareness that these issues exist but didn't provide any actual proof or uh indicators of compromise and we suspect that this campaign was highly targeted. Um but after Cisco published this information in general uh the actor switched to a internetwide exploitation campaign to kind of muddy the waters and um that's where we came in because we didn't have any information or indicators what to look for. So what we did is we set up honeypotss um using actual Cisco devices and monitored them for exploitation. Um yeah so as a short introduction um what is Cisco IOS XE? Uh it runs on a lot of devices which equals an enormous

attack surface. We have routers, wireless equipment and switches for example and the general precondition was that the management interface had to be internet exposed um for the exploit to be thrown over the internet. Um we also include data from shadow server in all of these cases. Um so big shout out to them. Uh this is the tracking data that they have for these Cisco vulnerabilities. Um they there are two big gaps in there. That's because of the um fingerprinting evasion that the threat actors used. I will come to that when we talk about the implant. Um and that's why there are annotations for implant version two and version three because it changed over time uh as

details were uncovered. Yeah. So onto the log and traffic analysis. Um as I said we set up pony pots to monitor the situation and to monitor what coyote that's what the threat actor um is named internally um that secure infra is was doing. Um we saw in the logs um already a couple of strong indicators of what they were doing. Um at the top the font may be a bit small. I'm sorry for that. um they created a new administrative user um named uh Cisco support to kind of blend in and maybe prolong their persistence for a little bit more. Um it's also quite obvious that they're using a specific API path um the web UI WSMA

and you also see in the SE Linux logs that they're writing a particular enginex configuration file and yeah at the bottom there's a screenshot from a packet capture uh that we created uh which basically shows the um exploitation activity uh in progress and based on these packet capture that we did. We worked together with ET Labs and um a couple of other network security vendors to create early protection rules for these vulnerabilities to hopefully uncover some some more. Yeah. So onto the vulnerabilities as well. Um we first have the authentication bypass um which grants the actor um access to the iOS D binary which is a monolithic binary inside of the appliance which controls the command

line interface and all the functions basically. Um you could talk to this um binary via an API that is called web UI WSMA HTTP and through a clever encoding trick as you can see in the first line at the top right um they could bypass certain security checks in the engine X configuration um that allowed unauthenticated access. So basically it didn't really matter what kind of uh credentials you provided. They weren't checked at all and you could run um arbitrary commands as a um administrative user on the IOSD subsystem. And from there if you have access um to the unauthenticated API uh you can again abuse a different API path uh which is called software management install ad uh

which via exploitation grants you access to the lower operating system components. Uh there was a particularly nasty pausing bug in the install method call where they uh check for an IPv6 address, but they only check if there are three um colon signs in there. So there's a um a command injection vulnerability in there. Um which looks different on depending on the version of the operating system that's running on the device. Um but works either way. All right, let's get into the implant or what the actual actor actually dropped after um exploitation. We call this implant boulder drop internally. Um Cisco Talos called it bad candy in their reporting. Yeah. Uh so this is the implant. Uh

maybe some of you expected a binary um but this is actually a um tempered with engineext configuration file. uh the actor established what's called a nobody but us back dooror here um which basically means they have a direct channel for command injection which is authenticated by file hashes and these hashes are unique per exploited device so they're generated each time and only the actor who knows these file hashes or like hashs can access the uh this back door um so this is what's tracked as lo on hash in the in the header um artifacts and yeah um the spectto allows um arbitrary command execution again and depending on the parameters you provide um it returns different HTTP responses.

Uh so this was used by for fingerprinting and when the act noticed that the defenders were on the tracks they switched up their implementation a bit and provided different responses so fingerprinting was no longer possible. That's what these dips were in the tracking. Yeah. Uh we were and still are very cautious about attribution at this point, but um as it turns out, the activity um that we saw is partially overlapping with reporting on Salt Typhoon. Uh there was a report by Recorded Future earlier this year. Um they call them rep mic. Um but uh yeah that shows some very striking similarities in the exploitation of these vulnerabilities and more recently in June um a Canadian

telecom organization was u compromised through these exact vulnerabilities um because they still weren't patched and as we heard in the previous talk uh thanks for that uh transition there was a um report from um multinational um joint group of many um intelligence agencies uh on the China Nexus activity um surrounding these vulnerabilities uh on Cisco devices. So if you need something to read for the weekend, I recommend this. Yeah. So on to case two uh the vanti epmm. So what is ivante epmm? Uh it's short for endpoint management mobile. Uh it's was formerly called mobile iron if that maybe rings a bell for you. Um the appliance runs on a base operating system of Redhead Enterprise Linux and

basically what the PMM does is it manages mobile device configurations on Android and iOS phones and it also integrates with Microsoft 365 online services. Um the tracking data from from shadow server is pretty spotty here. Um but in total there are about 4,600 systems out there at the moment. Um not all of these are vulnerable but um these are this is the general number. So onto a short timeline to showcase how quickly the attackers move in some cases. Um we had the advisory published by Ivanti on the 13th of May. Um and mere 2 days later um Watchtower Labs who are usually pretty quick about discovering and uh replicating these vulnerabilities published a blog post

with a um proof of compro uh proof of concept and a mere hour later uh we saw the first exploitation attempts locked on the customer appliance that we analyzed afterwards. And from there on it took only uh three weeks uh for the customer to realize whoops or I think you forgot to pass that points. Uh so that's where we came in. And yeah, that's how we get to talk about the uh payloads that we saw in that time. So that's a plus. Um in total we saw about 1,000 requests per appliance and there were if you boil it down about 80 unique export payloads that we saw. So this ranges from uh simple payloads like um HTTP

um DNS or even ping callbacks to the attacker infrastructure just to establish whether the service is vulnerable. Then we had a pretty popular one where they dumped the internal MySQL databases um for the credentials for example of the active directory and LDUP users um which also contains the office or Microsoft 365 credentials. So you could take over arbitrary accounts. Um we also saw some crusty loader payloads. Uh crusty loader if you are not familiar is a highly evasive rust vector targeting edge devices for example Ivante but also those of other manufacturers uh and is generally attributed to the China Nexus AP realm. Uh then we had some uh run-of-the-mill mai based botnet stuff that was floating

around and some hands-on keyboard activity where they downloaded a network scanning tool in Chinese language and try to probe around the network a bit to see what's up. And lastly, there were also some more or less script kitty level activities going on where they started a web server on a non-standard port and it was blocked by the firewall of course. So they had to come back and do weird things, but I'm not sure what they were trying to do. Uh yeah, so uh investigating these appliances, um we have um operating system level shell service sections that can be established, but they're restricted to 30 minutes each. So that kind of restricts us in our investigation.

Uh but on the plus side, the the boot disc is not encrypted, so offline investigation is possible. um aka that disk forensic and we collected most of the artifacts using Unix the Unix artifact extricctor collector. Uh so the primary artifacts that you will find on these appliances are for one the HTTP server logs or Tomcat logs which show the um exploitation attempts. I will not go into the vulnerability in this case um more directly because it's it's Java code and I I don't want to go off on a rant but basically you could inject code um through a yeah code injection vulnerability and also um there was an authentication bypass in there again. So that's kind of the combo

you always find in in these cases or most of the time. And the other artifact that we didn't see documented anywhere as as far as we know um is a proprietary log file that the appliance creates um and that logs user login into the administrative portal. Um so if there was unauthorized access, you might be able to see something there. And also a big shout out to the colleagues over at Profero who documented some of this workflow previously. Yeah. And so um as a last slide on the vanti um there were also cases where these appliances were um although with a different vulnerability breached um in case of Norway's government for example um most likely also um by nation state

actors but yeah right let's move on to case number three test >> yes perfect so case number three studio Netcala ADC. Probably everybody knows what netscaler is. I'm going to skip the definition and we're not going to talk about one or two vulnerabilities. We're going to talk about four in total. Step outside. So basically to make a bit sense here, we have on the right side we have some remote code executions. On the left side, we've got some memory reads where you could read session keys from the Citrix memory and basically replay existing sessions and therefore skip things like login and MFA and all right obviously you can run unauthenticated code and also we can look at the top

it's like the old older uh exploit or vulnerabilities and on the bottom we have the newer versions. So you can see like Citrix speed one and two it's kind of similar and also the right we have two rces which are kind of similar in the sense that if you investigate it's basically more or less the same um to give you a bit of understanding so we just saw a timeline about how quickly um we have a pock and then we have exploitation and to make things a bit more clear here also for CRIX I've tried to compile all the dates for first exploitation for the bulletin released by Citrix for the public pock release and for the Mars exploitation. Now

take these dates with a grain of salt because uh obviously we rely here on public data and not not each and every company is going to u disclose that they were breached. So the security bulletin that's fixed but the other dates take a bit grain of salt especially for the PC because we just looked at public source like GitHub and stuff like that. There's also like dark web source. We didn't look into it. And to summarize it a bit because you don't have to look at everything, you can look at it at home. Um, basically to summarize it, which of those vulnerabilities were used as zero days? We have two the speed one and the

unnamed rce. And what especially interesting is here obviously the point the date on the bottom time from PC to mass exploitation which is like two days, two days, one day, seven days. So as soon as a pock drops max exploitation starts on edge device or here in this case Citrix but also like dates from bulletin to exploitation is also really short because of course if you read the bulletin you also have some mitigation steps and you can read out maybe where the vulnerability lies in the developer pock. So we have quick exploitations which just showcases again with the two days also that just patching is not enough. If you patch and even if you patch in time, you still need to do IOC

hunting or investigation and stuff like that to make sure that you're not compromised before the Bolton was released. And even if you patch this one week too late, it's maybe already too late. You still need to investigate it. So to showcase a bit of the attack surface which we're dealing with because Tedrix is a well product which is widely used again we're using shadow server data here. Now uh one thing to to point out is for Cutics Bleet one and the rce they're bit older and this data is just one year back. So we didn't see like the big drop. Uh we can see that some instance are patched. We can see here CIX speed 2 the blue one which uh drops

at the end. So people are patching which is good but we still have a lot of vulnerable CIX appliances reachable from the internet worldwide. This is worldwide not just Germany or something. It's really worldwide. So all those device are vulnerable and for example Citrix split 2 we are now roughly at what's it 4K or something basically 4K device we can just read session tokens and retake or overtake sessions now um as we have instance for both uh vulnerabilities or both categories of vulnerabilities we're going to group it up a little bit so on left side we've got remote code execution so how do we analyze it from a forensic point of view and on the right side we've got memory

reads so basically Basically first steps for attacker as soon as he exploits the vulnerability what are his first steps on the left side it's always webshells you just drop web shell you have the persistence and execute commands even after patching or something like that if does not remove the webshell on the right side we get session keys so we use them to overtake existing sessions basically we can then use those machines which are deployed in CRIS and we are already mostly in the active directory now post exploitation so we did our first steps what are our next goals on on the left side for the rce as we are on the appliance itself we try to deploy

some kind of other mo even some crypto mining which we saw or we try to dump the NSP there's SMS missing core memory which also contains those session tokens to then laterally move inside the active directory on the right side we're already in the ad we try to move to the DC as our goal want to deploy ransomware something like that if we're ransomware group now where do we investigate like if we look at the remote code execution we obviously try to investigate on the appliance itself because we want to verify was it attack was it exploited is the attacker still on appliance itself or has it already dumped the memory and has gone further so we have to focus on

appliance and as you already saw analyzing appliance is not that easy it's not that common we need to pick up some tricks from our sleeves on the right side as the attacker is already inside your machines and already has those sessions we need to investigate those sessions we need to investigate are there any suspicious sessions are there many sessions using multiple IPs or sessions using weird IPs, something like that. And that's a bit hard for us as from a forensic point of view sometimes because we can't really know is this IP legit for this company or is it not legit? Obviously, if it's like 3:00 a.m. and it's coming from Russia, it's probably not really legit, but we

need some context and we don't have it. So, the customer needs to help here. And then lastly, how to investigate it? Well, on the left side, as I said, we need to analyze the appliance itself. also need to take a disk analysis, a disk image or something like that and take a look at it to confirm if there was successful exploitation and if we need to continue our investigations into the AD or the environment itself. On the right side as already said we need to check for suspicious login patterns which is not that easy sometimes and we can dump the NSP memory again as the attacker did as we do it ourselves because we can just check there for

requests. Now you might ask yourself on the right side why do you just just check logs or something like that and look for like requests to the malicious endpoint. Well because those logs are those requests are not documented by CRIX. So um there's a blog from Mandant and they basically had a report regarding this um uh regarding Citrix speed one and as they say those requests edge is not locked. So basically if I take a look at the appliance uh I can't I can't confirm it was exploited by CIR speed one. I can look into this memory dump which I take myself because it contains some requests but it does not tell me if those requests were

successful or not. So it's bit hard to verify was it successful or not because I'm really sure about it. Also on the bottom as you can see there's small note which is really important. uh confirming past investigation a past compromise also really difficult because there is one log which might help us but it's rotated after 24 hours. So if the compromise was a week ago we well yeah it's a bit hard. So let's take a step back and look at the disk analysis. So basically someone calls us or someone is compromised from either sherix or the other unnamed rce. uh what can you do to verify it was a compromise or not? So this appliance so the Citrix appliance

is basically running a hardened FreeBSD kernel or customize and we can't really run stuff on it. So there's no EDR running out, no AV, no security solutions. So we need to get to the disk now as we can't really run stuff on it. we need to use another machine then use SHFS and basically remotely mount this appliance to another analysis machine and then we can do something like using four or four live for some quick wins because we're looking for web shells and basically the directories where those webshells are stored are mostly the same so we know it and we can just look there or else if we want to do a deep dive we

can do a UAC image so Unix artifact collector and look at the logs and artifacts to get a bit more insights when what happened what and what can you do? And basically if you do not patch in time what do you get? You get web shell number one, web shell number two, web shell number three, web shell number four and as I said at the beginning we had four Cricabilities. Um allow me to proudly welcome member number five which is just three days old. Uh it's called Citrix Dalp. It's basically a bleed but in reverse as you might have imagined was released three days ago. Um, Citrix Netcale is not affected by using the default configuration but affected if

you use it as a gateway which is sometimes the case. It's again rce. So if you use Citrix call the it now probably or you can have a nice weekend and yeah so that's again data from shadow server. We have 4.3K affected device in Germany alone. Maybe someone of you has one. I don't know. Which brings us to our last case. 40inet 40 gates. Again, we start with the attack surface. So what are we looking at? And 40 gates. So in this case, we're looking at two CVEes which are basically the same. So if you patch the one, you already patch the other one. And they just let's just say they're the same more or less. So we

can see here against year worth of data we started like 45k affected devices probably a bit more because the cut off and it gradually drops because people are realizing oh I have an rce I need to patch now it takes some time as you can see but we're getting to the end more or less so this vulnerability affects or targets the often exposed webbased management interface you don't need to have it exposed to the internet it makes your life easier as administrator but you also allow vulnerabilities to affect you so don't do it and basically if you abuse this vulnerability you can take over SLBN sessions so again here a small timeline to showcase when was first um compromise

detected when was a bolton release stuff like that so on the left we can see from Arctic Wolf on the 16th November 2024 uh we saw they saw first probing for this vulnerability. So Fred actor was probing was checking of uh if some device are vulnerable. It was no exploitation yet just probing. Um some weeks later we had some first exploitations on the 4th of December. And roughly one month later we had a public adversary advisory from 40 gate detailing this vulnerability and offering a patch. Now again guys from Watchtower they're pretty quick. They took a look at this advisory, did some patch diff stuff like that and came up with POC, they released it and mass exploitation shortly

followed. Now again here we have a bit other dates but as you can see like from advisory to the pock it's like roughly two weeks and we have two months from the first vans to the advisory. So basically at least two months where at least one fractile I don't know uh took advantage of this vulnerability. Now back to our case. So basically our case involved a threat actor called Keing. They are ransomware threat actors. So like to um ransom your systems and steal your confidential data to make a double extortion. So you paid twice. Now we had a issue here because as you can imagine to verify if you have been exploited by vulnerability you need

some kind of logs. Now those logs they exist they are in the 40 gate itself. Foret also sells you something called 40 analyzer. It's basically data lake where you can push all your security locks from all your foret data inside. You got like all centralized is really nice. The issue here is this 40 analyzer system or this VM. It's mostly not really considered in the security posture policies. It's not a tier zero system for many companies. So it can also get ransomed which means all locks are destroyed. Now the foric itself it stored some blocks for a few days but it didn't store for like months back or dating back. Um so we can still use them

a little bit. Um additionally which was really funny the customer had defender so Microsoft Defender using Defender for Cloud and unknowingly he pushed some locks to Defender for Cloud. He didn't really knew it but we can use it for investigations. So really nice because without it would be pretty hard. So um yeah here logs and forgate uh we can use them to uh look at those logs and verify if this exploit was abused. Um if you have a 40 gate and foriz are like considered place as a tier zero system and securely deposit your locks somewhere safe. Now in our case the attacker abused this vulnerability took over some uh VPN sessions or create some

VPN sessions for unused administrative accounts. So we didn't create new accounts but already used uh existing accounts which is bit better for blending in sometimes uh create new VPN tunnels and modified some policies to access critical systems um from this VPN account. So um then using some locks like from the defender for cloud uh like the 40 gate locks in the defender for cloud and the existing 4gate locks inside the 40 gate itself we could reconstruct the timeline a bit. So when was the attacker active and what was he doing on the endpoint stuff like that and using that we could create like a timeline that's obviously not the full timeline it's much more in detail and

it's also obviously redacted but as you can see on the top you have like this 4 tunnel IP also have the two tunnel IP and they're working simultaneously so at least two guys working together for Khen and ransoming the environment. Um there's a small bonus. Um there's a nice report about um code hanger or umbrella stand which is a mware specific for 40 gigit firewalls. It's really stealthy so it's made for stealth and long operations. Uh it can hook sys calls. It can survive firmware upgrades and reboots um which is really nasty. So also a lot of attention here on 40 gate for threat actors. And lastly maybe third times the charm, I don't know. Um, there's a post. Just be

cautious. This might be a fake. We don't really know. There's apparently a zero day rce of on sale for 40 gates. It might be true. It might be false. We have we don't have more information about it, but let's just hope it's well false because yeah, I value my beacons to be honest. >> Yeah, I will keep the rest really short. Um, couple of measures to protect your edge infrastructure. Um, don't expose the management interface. uh create some visibility by storing logs and deploying a proxy or web application firewall. Install updates immediately. Don't wait. And installing updates isn't the silver bullet, so it doesn't m magically fix it if it's already compromised. Um there are some

solid recommendations by the Dutch. Um here are a couple of links um to stay up to date and a couple of tools that we mentioned. And with that, in case you want to stay up to date on our latest research, consider following us. And thank you for your attention.

>> Any questions? >> You guys have questions for now? >> There's one. In case there are a couple more questions, find us after the the event is over. Um, yeah, we're a bit short on time, I guess. >> Oh, yeah. Um, on the uh on the cases that you've worked um with the Citrix lock that would actually be helpful, how many customers have locked that actually on a remote location where you could have used it to be effective in an investigation? >> Could you repeat a question again, please? Um with the Citrix vulnerability we said okay um Citrix is not logging um the uh connections or the loons >> um you mentioned that there is one log

specific log type that would help. Yeah, >> but nobody like >> stores it >> which is rotated after one day. You mean? >> Oh, okay. And if it would like the if you would store it in a remote location, you could like use it later on, right? >> Normally you could use it. So another thing you could do is obviously you can place something before your CRIX which would uh record requests like a buff or proxy whatever which would make your life I think a bit easier because you also maybe have indication was successful or not. Um and you can look back further in the past than just one day because one day is not enough.

>> Yeah. >> Yeah. >> Like um did you encounter customers with like or was it usually that they would not use it or >> Yeah. Pro mostly don't use it like default Cric just standing there and yeah what happened we don't know most of the time. You also have of course this locks um which you can uh customize and forward to your Zim or something like that which is also sadly not really used but that's something I would recommend like in all your appliances because your appliance are reaching the internet so you don't know if there's a new vulnerability tomorrow so just be prepared have your locks you don't always need to have some use case

something like that for it but just store them at least because your forensic guys will thank you for that and your insurance too. >> Perfect. Thank you. That's not

>> but quick. >> Hi, I wonder I didn't saw any forensic analysis based on memory forensics on the FreeBSD devices. Why not? Um so first of all there was a propri I think it was a proprietary uh memory image on the on the CRIX you mean probably and most of times we didn't really need it in our case so we everything we need to know we could find out why UAC and like forensic analysis of the disk image but sure it makes sense in some cases to also use your memory images. Okay, thank you. So in case the other questions, we'll be drinking beer in 50 minutes down there. So just join us. >> Okay, thank you very much.

>> Thank you.