Press F to MOVEit: A quest to discover how a web shell appeared

hello everybody uh welcome to my talk uh press F to move it um a quest to discover how a web shell appeared um so our quick agenda for this presentation is we're going to go over you know what this progress move it transfer application is talk a little bit about um who the threat actor got attributed to and then Deep dive into some technical discussions about how Mandy and ended up investigating via this particular um incidents uh and exploitation of this application so a little bit about me I'm uh Peter yukonov I'm a senior Mandy and consultant with the Google public sector group under in Google and we focus on various uh State local and governmental agencies as well as companies that are associated with government uh prior to joining Mandy and last year I worked at dragos for a little bit doing industrial incident response and then I started my career out with the defense Information Systems agency I a DOD sub-entity uh performing incident response for various Duty and organizations uh from and then outside of work I picked up mountain biking after I moved to Charlotte a couple years ago and then I enjoy as all nerds enjoy video games and cooking um so what is progress move a transfer um so this is a piece of software that was formerly known as Ipswich move a DMZ um and you would see references to Ipswich and DMZ and various files logs that this application would generate um so it is a managed file transfer solution um similar to like Dropbox Google Drive OneNote but for a Enterprise environment that way you can set it up on-prem and you have full control over what happens how how it happens in that application um these manage file transfer solutions they tell you know secure collaboration and automated file transfer and as we can see um in as we start investigating you know how it got exploited some of these secure collaboration features are actually their possible downfall uh one um unique capability of mfts and move it in particular is that they support multiple instances of an environment per deployment so that means you can set up unique enclaves for installation that have you know dedicated controls to access those enclaves and in case of move it uh there were multiple ways you could interact with application um some of the more Legacy methods so FTP I served up an ftp's file server that you could automate tasks to pull files upload files there's a web UI um that you know majority users would probably use and it also had guest access which would we will see that that guest access was leveraged as part of the exploit chain and then finally there was a web API for web API stuff so you can develop apps to integrate with that stuff and this web API would also be ended up being used by the attacker to successfully exploit the application now from an attribution perspective so when manual started investigating all these incidents regarding this application um many that originally attributed this activity to an ankh 4857 and what we call an unk is a uncategorized cluster of thread actor activity that way meaning you know we've never seen this potentially before um so as more and more information came in from the various investigations public reporting um based on some of the targeting infrastructure overlap and then finally when klopp um said hey this was us um this Unk got merged into a fin 11 thread group that Mandy has tracked um and there are financially motivated thread group that's been active since at least 2016. um historically Finn 11 has conducted some of the largest and longest running malware distribution campaigns observed amongst mangian tracked Finn groups today and they've recently started to monetize their operations using data Expo as we saw in this case um the thread group ended up using a zero day vulnerability in servers that allowed them to excel a lot of data and this wasn't the first time that this thread group has used a server's vulnerability uh some of their previous known uh public attacks included attacking the excelian Legacy file transfer system secure solarwinds secured FTP servers and then before to go anywhere managed file transfer solution relatively recently as well now from a timeline perspective this timeline was looks pretty basic um the earliest that public reporting suggested that fin 11 started to be interested in move it uh goes back all the way to 2021 so almost over two years at this point uh with some public reporting indicating that somebody was scanning movie instances to see what were they out there now as we started investigating the various compromises we were able to determine that back in April of 2022 there was large volumes of attacker scanning against various movement instances and this was observed by the various main investigations fast forward to a little bit over a year to approximately May 15 16 of 2023 and again we saw Mass scanning of movement instances over those two days with some instances getting scanned anywhere from like two to three times and then on May 27 2023 uh D-Day that's when the max exploitation and data theft of the various public movement instances started to occur um Keen I among you will remember recognize this weekend as potentially Memorial Day Weekend uh in the US and in on May 29th which was actually Memorial Day in the US we started to see additional scanning activity against instances that weren't successfully exploited in that original wave of attacks uh moving a couple more days later on May 31st the cve got finally publicly announced by the vendor progress in this instance announcing that there was a SQL injection vulnerability and that it was under active exploitation and the recommendation was to take down you know your move at instances from public access and to perform incident response and investigate what kind of data might have been breached um and then a couple a week later almost June 6th that's when Finn 11 klopp publicly announced that this was them performing exploitation and Performing proactive red teaming uh to secure your infrastructure um and then ever since then then um a club has been um doing negotiations with affected entities and then if somebody potentially didn't pay up or negotiations broke down they would release that information on their uh League side now how did we end up performing this you know investigation figuring out what got affected impacted we ended up using a variety of data logs uh focusing mainly on the Move It application logs as well as other hosts and applications logs in the case of move it uh there was three main logs that were most beneficial from an analysis perspective there was this DMZ underscore weblog and it that ended up capturing a lot of requests errors including full SQL injection queries uh stack traces from exploits execution failed exploit execution um fun fact about this log um depending on how you had your movement instances configured if you're using SQL Server as your backend or MySQL certain commands that the attacker was doing would fail or show up slightly differently and those would cause a log to be created um additionally there was this DMZ is API log um and that one particularly captured in certain instances full request response so you could see the full headers and request bodies that were being sent to the application and that allowed us to see what kind of you know metadata they were submitting to the application to cause the vulnerability and then finally there was a move it audit log in the application and this log uh blogs all audit events in move it so this includes file downloads login events logouts file uploads and by default it gets logged to the database that backs the movement application and what we saw with this attacker they were actually coming in after they finished their exploit chain and deleting the audit logs from the from the database so they were trying to you know make it harder for anybody to do investigations um one thing that we saw in certain instances depending on the organization there was an option to actually forward the move at audit logs to Windows event logs or syslog and that would just make it a lot harder for an attacker to clean up that log plus you can do additional detections right as uh that that data fed into your sim for example now from a OS slash database perspective um Microsoft IIs logs ended up being invaluable uh for this investigation those log standard web requests including by default how much data was being transferred as well as duration of this request so that made figuring out you know was there a download um a little bit easier additionally uh one thing that we did find out um and Kane became very helpful as you'll see uh in the next couple slides are the SQL transaction logs and SQL transaction logs are logs kept by the database anytime a query modifies data in the underlying database so any insert update delete statements that get executed get saved to this file so that you can replay it in case something happened right with the database um and then finally uh in this I would say super useful in some of some of these investigations but if you had network security tools so like netflow Zeke for example um they ended up being pretty helpful to see you know the volume of traffic being sent across the environment as well as any EDR tools endpoint detection response that way you could have potentially seen that web shell actually getting dropped on the system and use those logs to investigate if there was any other possibilities uh for like lateral movement or any other adversarial actions if it was some sort of different thread group that was attacking you now the exploit chain the exploit chain ended up being from all the logs that we've seen about 16 requests to multiple endpoints in the application that resulted in this human2.asbx page getting dropped on the server and then used by the attacker um the chain um some of the pages that were used as part of this exploit chain was this guest access that espx that was leveraged to generate valid asp.net session IDs that then the attacker took put them into this SQL injection that existed in this movered asapi that dll endpoint um and using the SQL injection they were able to do privilege escalation stage their payload for deserialization as we'll see in a couple slides and then perform that artifact cleanup including audit tampering and then the key API that was responsible and that API endpoint was this API V1 that move it exposed for programmatic access to the application and this API was leveraged in combination with that SQL injection to actually perform the deserialization vulnerability and to deploy the Lemur loot web shell as we called it and this API was used to create a resumable file upload that contained the payload and then State use the SQL injection to Stage it for exploitation uh one thing to note uh we did note that if the web API was disabled or not licensed the whole exploit chain would fail and the attacker would come in and scan it again now I talked about this web shell uh what is it uh Mandy and ended up calling it lemur loot um it's a c-sharp web shell that was specifically developed to Target the movie transfer application um it ended up getting a hard-coded passwords set for every exploit attempt um so that you know only the attacker could access it and it performed multiple functions everything from performing full scale data uh reconnaissance and configuration enumeration so getting stuff like your Azure blob configuration settings getting a full listing of files folders and users in all instances of an installation and then performing some of these uh impersonation and admin creation capabilities um one interesting thing about this file is why it was called human2. espx as well so that it blended into existing pages that existed there were various Pages called human.isbx machine.spx and there was a machine to Daddy XPS which is a legitimate page um so this way you know if a regular admin looked at it they would have had no idea that that page potentially was bad now how did we figure out how limor loot appeared first things first review all the various logs you know a bunch of logs scroll through you know somebody's gonna be like hey I get tired of looking at logs we did identify a web request um in the IAS logs that ended up returning status code 500. um this particular request was that specific resumable file upload um that ended up returning a 500 meaning something happened on the server and using that uh time frame timestamp we ended up pivoting to the DMZ web log and in there you could actually see a full stack Trace that gets got dumped to the log saying hey something happened with this binary formatter deserializer in this specific function of the movie application um so next steps were pretty much just go decompile the application take a look at where this specific function exists and this is how that specific function looked um in this red box I have highlighted this particular binary formatter that was exploited and you can see here this binary formatter deserializes whatever is in this upload State variable so poking around the application some more we find this function that sets this upload State variable um and here we can see that this data this function is backed by this file upload info table and both this comment field and State field in the database are encrypted but only the State field is base64 encoded and this will be quite handy in figuring out the actual payload a little bit more about this file upload info table pretty simple has a file identifier common field potentially saying hey you know this file is being uploaded for XYZ reasons and then some transfers IDs that are used internally and the core field the State field uh that's State field just tracks the file upload progress some some metadata now knowing this database we could ended up looking at the SQL transaction logs and what we saw in SQL transaction logs was the attacker using the resumable filed upload API containing the base64 payload in the comment you could actually see the full insert statement and here we just I just redacted a lot of the information because the strings ended up being super long um the attacker then used the SQL injection to copy the value from the command field into the State field um here's another screenshot of how it looked in this transaction log with a regular update statement where the uh the encrypted payload value just got copy pasted into the the right field um and then at this point the attacker then used the the reasonable file upload to uh vulnerability to trigger that destabilization vulnerability and then finally how does actually their cleanup look um the attacker then used the SQL injection to actually go in and delete that file that that was being tracked as a risen mobile file and that is just a regular delete SQL statement so at this point we have the encrypted payload from the database um how do we decrypt it and figure out what the actual raw payload was that was ran well at this point you should Channel your inner attacker and create we ended up creating our own what I would call Web shell um to decrypt the payload um this is a screenshot pretty straightforward uses built-in functionality of the movie transfer application to perform decryption um one unique feature about move it I've as I've talked previously about this different instances in the application um you have to specify a correct instance ID to get the correct decryption key one unique thing the -1 instance ID is the installation one encryption key that is for the entire installation that is used by the system processes there are other instances IDs we originally were trying to use those uh and the code would just error out saying you know incorrect decryption key when we ended up running this page we got a base64 encoded payload back that we saw you know the attacker inject and then the only reason it returned base64 at this point is that so it was easier for us to transfer it back to our environment for analysis and probably what everybody waiting for what does the payload look at this point we got it Quest accomplished we figured out how the web shall appeared on disk so this um serialized payload this is a beautified version of it pretty much the three things it decoded a string in a specific header then generated a good for a password and replaced a pretty fine template variable in this case change me 1337 with that specific Google password and then the content of that string ended up getting written to disk and if it was successfully getting written to disk it would responded with okay and then the specific password for this exploitation this just shows this just explains why every single instance of a recovered lemur loot web trial had a different password now from a post exploitation perspective after the attacker got the web shell on system they performed data exfiltration using two primary methods first one using native functionality they used limor Loop to impersonate an existing admin logged in with that impersonated admin session and then just clicked download all files from the page and the web application zipped everything up and provided them a single download file now in some instances and this uh was reported in some public reporting about this in these investigations uh the limo loot website was actually used to download individual files uh from a timing perspective uh data x file actually occurred anywhere from a few minutes after that webshot got dropped on the system to and we saw in some instances several hours passed uh before you know exploit uh data X will occurred um so that's how the webshot got dropped and used now how could a view potentially have prevented or detected this exploitation from a detections perspective you know as everybody a lot of individuals people recommend you know monitor your web routes for new file Creations you know typically you shouldn't have new files getting created especially aspx Pages or code Pages getting created in your webroots uh you should definitely leverage some sort of EDR especially on your key assets and specifically on your publicly facing assets and some investigations we saw some edrs create a an event that hey a potential website is getting uh created but it never bubbled it up to an alert um and then finally as I mentioned in the log section you know forward all your logs to some sort of sim so you can do some additional enrichments and analytics um that way you can have more insight into what an application is doing and here I have a couple of you know what potential things you could develop rules for uh from a recommendations perspective a lot of these managed file transfer Solutions have retention policies built into them so leverage them to automatically delete files so that if you do get exploited you know you're not releasing 30 days worth of data or like terabytes of data potentially disable unneeded features and hosted applications would have saved some organizations in this instance and then probably the big one um do not use this binary formatter that is available in.net Microsoft even says in their documentation like this function is unsecured there's no way to secure it so if you have that in your code there's probably a chance that an attacker will find it and potentially try to exploit it um so that couple recommendations detections hopefully that will you know you can take that with you back to your teams and potentially develop something for your tools from additional resource perspective Mandy and ended up publishing a blog post uh going over this specific intrusion and exploitation as well as a containment and hardening guides focusing on movie transfer so that you know if you do have mover transfer you can potentially apply this to harden your environment and then of course progress put out a technical article detailing all their findin