How persistent is an APT? Battling Three Threat Actors in a Single Environment

you thank you okay hi everyone so welcome to our session today we will talk how we battled three different threat actors in one single environment okay so let us start um I'm onon kusher as a wonderful Karen introduced St I'm the director for the incident response in siga hi my name is or bman I a red team term blue I'm a foodie and a coffee addict um and also an incident response team leader at signia and signia is a Consulting FM specializing in helping organizations that have experienced security Bridges Our Story begins when a client contacted us and requested our assistance with responding to a financial extortion attack now when we pick up the phone and went on the call

it seems like a very regular activity extortion attack we encounter many of those through the year but at this point little did we know that we are in for a roller coaster ride what started as a simple I turn out to be a battle against three very capable and very persistent threat actor but let's start at the beginning okay with threat actor Alpha threat actor Alpha is a financially motivated threat actor they are the kings of cloud-based attacks and this client was working with different Cloud vendors they had a single identity Federation that ruled them all and the threat actor managed to compromise a highly privileged administrative account and using that that user they connected

to the cloud environment and compromised it now this threat actor was very creative they knew how to bypass multi multiactor authentication they knew how to bypass conditional access rules and they executed almost every attack in the Playbook on their Cloud environment in addition to these attacks they also searched for it related documents and this is how they found out how to install and operate the organizational VPN solution and they use the administrative account to move to the on premise environment and also compromise that environment one of the novel techniques that they leveraged was to deploy an EDR solution as their CNC mechanism now put yourself in our shoes we are incident responders we came to investigate a

compromised host and we see that there is an EDR solution installed initially we were very happy right we have Telemetry we have data we can now investigate so we went to the client and we say hey can we please get access to the ADR console and the client says I'm not familiar with the solution I have no idea what you're talking about okay so we dig a Little Deeper start to investigate and then we discovered that the threat actor was the one who deployed the ADR solution and they leverag the Tool's ability to execute commands on the machine in order to have remote access into the environment so overall the attack of Thor Alpha was very impressive but we

wanted to kick them out of the environment right so we started working on a remediation plan we basically gather all the ioc's on the ttps and give recommendation to the client how to clean the environment and as part of the remediation plan we also implement some postbridge monitoring the reason we do that is that we know that threat actors will respond to the radication efforts sometimes they will try to re-enter the environment sometimes they will leverage dormant persistency mechanism but they will respond and you want to be prepared and you have you want to have monitoring in place and when we monitor it's easy to monitor for the ioc's right it's easy to block and monitor IP and hash value but

it's also easy for capable threat actors to just purchase a new IP address so we always want to monitor for ttps and we also monitor for the implementation of the eradication plan why we want to monitor The Blue Team of the organization because we learned that in large and complex environments one it team can Implement a rule in the firewall and a different team will revoke that rule because they are used to how things worked before it was easier to connect to the network so we also monitor the blue team and and here in our story we completed the remediation we completed the post Bridge monitoring and now we just sit and wait now several days later we see a very

suspicious activity when the user the high privilege user that orang just mentioned was re-enabled okay we contact the client and then ask them hey guys are you familiar with this activity they immediately say no and we understand here that we might see threat actor Alpha returns to being operational now it's true it was the same high privilege user but with several differences the activity was originated from a different server that was correlated with the known scope of compromise of threat actor Alpha Deep dive investigation of This Server revealed an I module installed on the server but it serves only one specific single web page which is a China Chopper webshell now this is another something

that we did not correlate with threat actor Alpha at all Deep dive investigation of the IIs access logs on the server revealed that the access to the China Trooper webshell dated years back this leads us to the in eminent conclusion that we had just just identified a new threat actor a live campaign of a new AP inside the same network let's talk about threat actor beta threat actor beta is one of the China next to state sponsored threat actor with the motivation of Espionage okay different theor Alpha now they had prolonged operation as I mentioned several years within the same environment and the main ttps of this redactor was web shells web shell web shells web shells all kind of variants

of China Chopper web shells either the classic one either one with some kind of a symmetric encryption and even others that just out there so basically the stor have utilized the web shells not only as the access Vector from external but also for lateral movement and remote code execution in the internal environment they deploy it remotely via SMB and then access it in order to execute code remotely via some kind of webshell tunneling mechanism very interesting now another interesting fact about this threat actor is that they operated from home routers meaning that you cannot just simply block the IP address of the threat actor since there might be a legitimate activity origina from this IP address so

before proceed let's let's H take one step back and understand what led us to reveal this specific thre actor this is something that we called a remediation Domino okay when we implemented a a comprehensive remediation plan in a large it Network then it actually impacts the network okay it it impacts the day-to-day operation and it can also impact the threat actor operation as well now for threat actors that are operating H for a long period of time in the network they know that the network will change they don't not necessarily know why but they need to adapt now this is something we called a golden time frame when we Implement a a a very comprehensive remediation plan and

enhancing the monitor in this specific golden time frame we are able to detect abnormal activity which lead us to detect this specific threat actor okay they try to resume their operation by just Reena the the account and the access they had which is a bit blunt operation but we were able to identify it detect it and understand what behind of this activity now it's true that um changes creates opportunities okay the it changes creates opportunities but for both sides for threat actors if we are doing an incorrect remediation then we might connect for example a two previously segmented Network and create a new lateral movement path for the threat actor this time fortunately the opportunities was for us the blue team

in order to detect them now after that we understood that there is a live threat actor in the network what do we do we investigate it okay but what is the problem or the challenge with investigating an AP is that they are OBS aware they will tamper logs they will delete their evidence they will um disable install security mechanisms and they will try to do everything in order to evade detection they want to to operate of course so what the proposed solution to deal with that is what we call stealth monitoring stealth monitoring actually means that we will Implement such kind of a monitoring mechanism that will be out of the site of the threat actor one example for that

is to implement what we call the port mirroring okay we know that server a is compromised server B is compromised the threat actor operates between a and b we will go to the network level implement this kind of Port mirroring into a server which is out of ban okay not even aain joint nothing we only the uh the ones that knows about the server and then monitor there when we will monitor then then we can continue to understand what the threat actor try to achieve okay so let's go back to the investigation the monitoring that we implemented helped us to identify a another compromised host it was compromised by thre actor beta it has its China Chopper webshell but when we

investigated the host we managed to identify another tool a new tool Shadow pad now Shadow pad is not one of the tools that were leveraged by thre actor Beta And when we tried to investigate how shed got to this host we identified that it was during an RDP session again not a TTP of threat actor beta that likes to deploy webshell from web shells and the deployment of shadowed was not on the same time frame as the deployment of the web of the webshell and this is how we came to the conclusion that we have a new threat actor in the network the third live campaign within the same client environment let me introduce you

this threat actor signia attracts them as velvet an they are a Chinese AP group specializes in Espionage they maintain the prolonged operation in the Network they were in the network for more than 3 years now vetan are very creative they are the kings of dll side loading they can load DS in ways that nobody can really and they are very evasive it means that they disable security tools they tamper with the log they do time stomping they are H they even try to tamper with our monitoring and some of the tools that they leveraged during the campaign were pluging and shadowed both are remote access tro that are characteristic of Chinese AP groups they also use toway

proxy to Tunnel the their activity within the network and wmi exec in order to execute tools remotely so we investigated vetan it was very impressive but we want to kick them out right so how can we eradicate an AP usually we take the kill switch approach kill switch means that we have a coordinated remediation we have a single event it's a very big event we prepare for it for weeks we need collaboration from multiple teams it's the it Network Security application owners we bring all teams on site some of them are in data centers and with the DRP sites usually you conduct the kill switch off hours or during the weekends off hours during the weekends H and it's

a very big event the reason we Implement key switch is that if you try to clean host by host when you're dealing with an AP they will respond okay it's easy for an apt group to change their ttps in the middle of the operation it's easy for them to implement hidden back doors and we don't want that to happen so we need to clean everything in a single day we need to have a list of all the compromised h all the tools all the persistency mechanism and just clean them and when we do that we also Implement post kill switch monitoring so we'll be able to identify all the re-entry attempts and if there is some

sort of back door that we missed during the investigation we'll see it in the monitoring so we clean the network Von are out the network is clean the monitoring is in place and now we just sit and wait several days after the successful kill switch we start seeing alerts being triggered again we see that someone is deploying plugx in the network and it was very strange because we think okay everything was clean what's going on so we went to investigate the host we went to plug pulled the configuration information and identified an internal C2 instead of the external IP that was blocked and we try to identify which IP trash the host and we came to a legacy server now this

client has an entire segment of Legacy hosts none of them were monitored by the organization EDR solution because it was an unsupported operating system nobody deployed additional visibility so they had a segment that was not monitored and the threat actor was quick to identify that we went to investigate that Legacy system and it was also infected with plugx when when we pull the configuration file there was no CNC configured now this is really strange because plug X is a remote access tool so how can a threat actor connect to a remote access tool that doesn't have a CNC configure we dug a little deeper we did some memory forensic analysis and we identified another internal IP address

this IP address belonged to an F5 big IP device now F5 big IP is a firewall SL load balancer here it function mainly as a load balancer now let me take you through how the attack chain worked so we have the F5 load balancer it was compromised running four different malware families on it we have the external CNC server it's the same IP that we blocked in other locations within the network one of the tools that was running on the load balancer was a tool that we track as verting verting would pull the CNC server once an hour looking for command to execute when the threat actor wanted to regain access to the environment they

would connect to the CNC and leave there a command to be executed that command allowed the thread actor to connect over an SSH tunnel to the load balancer from the load balancer the threat actor would move to the Legacy server and connect to the pluging instance that was deployed on it and from that plug instance they also compromised additional host and deployed additional pluging now this is the point in the story that our alerts started to trigger because the Legacy server is not monitored but other sections in the network were monitored and we were able to identify this lateral movement attempt and of course that each new pluging deployment connected back to the Legacy server as

its internal C2 so now we updated vetan ID card it also includes now working from Network device deves it includes vetting and VAP VAP is a tool to perform Network tapping on the management interface of the F5 device so three three actors in a single environment all of them clean and now you probably think yay the story is over we can go to eat lunch no so the post kill switch monitoring allowed us to understand how persistent these APS are and let's do a side by-side comparison of threat actor beta persistency and velvet an persistency let's start with thre actor beta thre actor beta post kill switch tried to access the same webshell from the same C2 it was um of

course a fail since the web sh was deleted and detected by our monitoring now then they tried to change the C2 and access the same webshell in that phase they still didn't really understand what happened the access was failed since the webshell was deleted and detected by us afterwards they tried to do another thing to exploit using some kind of a de realization attack and deploy what we called a ghost webshell okay A Memory residing webshell and this was success for the threat actor however it was detected why because we monitored for the TTP immediately after successfully deploying the ghost webshell they try to create processes on the new compromise server on this specific TTP we have

monitored for the child processes that are being spawned from web server processes cool uh now let's talk about vetan so while threa to Beta AP persistency was a in its attempts to regain access to the environment velvet an was very persistent within the same environment they attempted to deploy tools pointing to the same CNC server that CNC was blocked in the security stack so the attempts failed and we had monitoring then they Tred to deploy tools using the same name with different hash values now there is an issue with edrs they are very good with blocking by hash they are not good with blocking by names so the deployment attempt was successful but we had alternate

monitoring in place we were able to identify this activity and we responded quickly another attempt is thre actor velvet an basically basically used impet to deploy tools and since impet is an open source tool it's just a bunch of Python scripts it's easy to modify them and once the threat actor modified them they were able to use them and move laterally within the network again we had alerts in place our alerting was based on suspicious service Creation in the network so it was agnostic to the change in code and we were able to quickly respond to the activity now it's very interesting to see that both threat actors were actually detected by the TTP based alert that we implemented in the

post kill switch phase so let us go into the key takeaways for this session let's start with the first one or first two which are that lucrative targets or large corporate networks might be hosting several threat actors working in parallel each threat actor might have a different motivation some can be the extortion some can be the Espionage so on so forth and the uh specific motivation of a threat actor might impact their persistency which means that for an extortion attack there will be a simple calculation of cost to effort and if the operation is starting to become H not beneficial then they might stop or halt the operation until further decis decision like just like a normal

business right but for Espionage the need for intelligence will probably always be there so H they probably will be even more persistent now next H takeaway is that the response that we will do for the AP will draw a response from the AP okay now this is a very crucial here to understand the post kill switch monitoring this was the one that assisted in order to understand what kind of a response there will be now for the Chinese Espionage threat actor the response was try to reenter the network try to change their lateral movement path within the network but H differently than that for Iranian threat actors we do see that if they um if they

feel that their operation is compromised and they are being discovered they might shift their operation into some kind of a disruption attack okay uh to hide their tracks and to do whatever they can with the uh access they had for even for a prolong time in the network so this is something to take into an account when preparing some kind of a response to a threat actor to consider what can be done from their behalf last but not least is the postbridge uh monitoring and visibility enhancement okay without implementing some kind of additional visibility mechanisms to allow us to understand better what the threat actor do and to monitor their ttps in the most efficient way we probably will not even

understood that they just resume their operation or that they are try to reenter the network so this is super critical when responding to such threat actors to implement the right remediation measures and the visibility visibility enhancement in order to support this monitoring phase now with that being said stay tuned for part two thank you thank you