A Case Study of MacOS Supply Chain Compromise

[Music]

hold on you I am Michael George and I'm gonna talk about Mac OS host monitoring and I can do Incident Response using light ways lightweight open-source tools and after that Jason's gonna talk how you can use the data from those tools to proactively find power in your environment so we have a lot too content to cover so probably going to talk a little fast so sorry not sorry nothing we can do about it all right so a little bit about myself my name is Michael George I work on instant response at Dropbox I dabble in like Mac OS stuff I don't really have a super active Twitter but I really like these touched coma things from ghost in the shell' so you

can always find me on the internet by looking for those thanks Mike I'm Jason Craig I work at Dropbox also we work together you can find me on Twitter I don't we very much anymore kind of busy it's unfortunate I like unicorns in a minute for the tea files so why are we here it's a great question what problem are we going to describe to you and why should you care about what we're saying we hope to cover that we're open to your feedback if you feel otherwise how are we gonna how are we gonna find the things that we're gonna find we're gonna talk about some Mac OS tools that we use to generate generate

telemetry on our own some tools specifically that Mike built and has talked about a little bit in the past we're gonna talk about some ways you can take the data from these tools with some analytical techniques to to find supply chain attacks and other attacks that you may not have observed before it's fancy but not so fancy Analects you'll see I'm gonna resist the urge to troll you all with this screen because it's huge and for those of you that know me personally I might fail it resisting that we'll see so squad goals what are our goals with this talk what are our goals in general with security at Dropbox specifically with the respect to Mac OS first and

foremost we like to lima tree we're gonna generate and log as much data as we can about all the things secondly we want to generate our own specific threat intelligence I think that you probably would like to generate your own specific divided intelligence to your environment it's better than what anyone else can tell you in addition to that we want to decrease time to detect new malware from weeks to a day or less depending on how performant your your analytic systems are obviously we don't want to have all our secrets stolen and we want to race Patrick Wardle for those of you follow the Mac community Patrick portals pretty prolific blogger he talks about a lot of

stuff he makes tools I know if he's here but he is greets we want to beat him to a blog post so again why are we here incidents happen supply chain attacks happen don't wait for a blog post to show up in your Twitter feed to do something about it figure out your social chord was stolen your user data left your money's gone so handbrake this is a screenshot from the handbrake website when they announced their supply chain compromised for those of you who don't know what handbrake is it's a popular video transcoding software for Mac OS it was back doored last year in May and it served malicious binaries in addition to the legit handbrake software

to a bunch of people so we're gonna talk about that specific ambry compromised in the context of our Mac OS tooling and the proton rat that specifically came down with the handbrake malware so I note about supply chain attacks they're really popular right now you're gonna see a lot more of them they're great in a number of ways for an attacker because they're much likely much less likely to be detected versus say a screw phishing email all of us have spear phishing email detection either from our email provider we have our own analytics on top of that we may have a third party source on top of that still and they this will completely completely succeed

because it won't be delivered via spearfishing it's also a great a great target for mask compromise you just see who downloads your stuff you away from them to phone in see who they are by their by their machine info by their ESN if they're calling you to you select your targets it's great so again guaranteed you're going to see a lot more than so fun facts these are not fun facts but we'll call them fun facts so the handbrake server one of the to handbrake download service was compromised on May 2nd the project and announced the compromise on Saturday May 6 which is a terrible day to announce anything because everybody's at home but also the first earliest

notification that they gave it's probably better they gave it then than Monday so if you if you paid attention to either the popular security media or Twitter you would have found out about this on Monday mates in the morning also during the time of the incident there was zero coverage in VT for this implant so proton the proton package in handbrake was not signaling by anyone's security software so VT had no detections for this all the shouldn't say that the AVS NVT had no detections for this at the time so the IOC s for this the indicators are compromised when mainstream on Monday May 8th as I described patrick portal blogged about it saturday morning he has a prequel

blog post i recommend you read it a bunch of really bad stuff happened consequent to this so more fun facts how many have heard of CCleaner i can actually see this great so a lot of people so I did not know what ccleaner was so ccleaner is a Windows utility that is intended to speed up and clean up your PC and the quotes are very intentional these two quotes are directly lifted from the ccleaner website and now seeing the compromise you may note the second quote says the compromised version dot was undetected for four weeks that's super bad for everyone I personally think this is a really notable point it's kind of the crux of

what we're driving at is that no security company told anyone in the world that abt 17 was in their networks for at least four weeks it's bad news so the second quote really seems to like throw a little bit of shade of security companies which I think in this case is probably appropriate so ccleaner found this suspicious activity on September 12th and they announced it on September 18th in the blog post if they want public about it they Nancy I've seen at the same time yeah so that's pretty sad panda and alternatively you can call this rogue and depending on who you use for apt names Mike so now that we have a good idea of like

why we should care about this we should probably figure out how we can get the information we need to actually start doing something about it so I came up with this outline for a doing investigations so the first thing that we want to be able to do is identify hosts by checking for known IC known IOC s io seas are indicators of compromise compromise such as IP addresses domains hashes file name file paths etc you might be able to get these via the notice this is like the the main IOC s that were released on the Monday vs like what was released on Saturday so to be able to do this we're going to use OS

query to help with the time lining and scoping of infections so we can figure out which hosts were actually compromised and then for each host we want to establish a timeline of what the malware did on the host so we want to be able to reconstruct a process tree figure out what malicious activity happened on the host what network connections were made and what files were touched by these processes and we're gonna cover three tools to go over this os Gray Santa and audit when the Santa is gonna help us create the process trees and audits going to help us understand the network connections and what files were touched on the system so OS curry OS query is a awesome

tool developed by Facebook it allows you to check the state of the machine on a schedule using a config or on-demand with the help with some external TLS tools these are really cool a couple of providers of these are colliding window all of the current state data for the machine is accessible via an SQL interface so you can do fancy things like join the processes table with the current network connections table with the current listening ports table and see for different types of like interactions between processes it's really cool you can use those great to write the output of the queries to a file on disk or you can use a plug-in to like shovel it to external sources it's

super easy to install it works just right out of the box it's available for Linux Mac and when those windows is bit new so there isn't as much feature parity as from what I understand and the log volume for us query is actually pretty good because you can DIF only changes to the system so it's a complete state machine so if action happened it gets added and then like if it changes later then you get another log that says it changed instead of a log every like two seconds it's pretty awesome so this is an example of me running a query on my system looking for the ILC for handbrake I'm actually read it yeah

so it's just like a SQL interface so I just selected from file where passes the IOC you get a bunch of information about what was actually what the permission set was like when it was added a bunch of information like that these pack these queries are available in query packs provided by us query the immunity is really great about adding new stuff as soon as it comes out so we really think people for doing that so if you weren't able to run this query across every machine in your fleet you should be able to figure out which ones actually had the malicious application installed but it was curious only a state machine so it only gives us a time

at like an interval so every time it checks in we need something a little bit more aggressive or event-driven in order to get like process trees and in order to not miss things so for that we rely on Santa so Santa provides us with execution events from the hosts hashes from the executed binary some information about certificates it was primarily made as a whitelist blacklist engine for like Google to block egg processes or execution of processes on their hosts so you could like block a completely known malware or you can block any unknown application from running on your system they recently released a tool called up vote to help identify what you should and should not

blacklist so the right list so that's pretty like double thumbs up for Google but mainly what we're focused on here is execution monitoring and the logging of those executions so this is the first the first image on the top is a handbrake installing a malicious plist on my system it just moves it over to the launch agents file and then loads it in a second command so that's like one of the types of activities that you can just catch by looking at process execution events the second one is it actually exfilling all of my data off my machine the super fun I mean don't you like losing all of your SSH keys your passwords from your Chrome browser it

did a bunch of stuff as we're both yeah still is your one passer involves there's your SSH keys browser history form data a list like your serial number for whatever reason I don't know why they use it probably so they could have it as part of their c2 but it also tried to steal your your your password for your system so you're like local account password and it uploads it to handbrake that bids which you can see as part of this curl command it's pretty fancy also no one should ever do a curl - app that's like why would anyone do that anyway so this is another execution event which is it trying to or it's

moving or making a zip out of my one password vault right before it stages it for exfil we can see a bunch of events like this from the handbrake malware for pretty much everything so that chrome data and like opera data and Firefox data like puts them all in like one place as zips so actually monitoring this type of information is a very useful oh yeah it steals your OS create your OS keychain that's also fun especially if you store your SSH key password in there so one last really cool thing about Santa so it hashes the binary that's executed and if you're able to look in virustotal or other sources of truth for these hashes you

can get information like this this this fiber total buyer store the result is about a month afterwards and is still only half of the AV vendors in virustotal detecting the actual malicious binary from handbrake I'm sorry Mike you repeat I in the area only half of your antivirus could actually catch it along after a month it's that's important yeah yeah it's not great but you can also use the information to help you determine how good your antivirus is so you can just like look and see whether or not your antivirus detected malware by using fires total anyway it's cool stuff but it also includes a quarantine URL so if one of the files are downloaded from the

Internet to the host you can actually track where they were downloaded from and this provides us some information so we can see where it was downloaded from specifically this one was downloaded to a VM so you can kind of see the local IP but in general you would actually see the domain name and the full path to where it was downloaded from this is help you find attack vectors and be able to identify things that are downloaded from the first time from a trauma from a domain so after that so Santa doesn't actually give you network connections or file like modifications so we need to have another yet another tool to actually get that information as well so

what Apple has provided us is the audit subsystem or open DSM or Darwin DSM or DSM which allows us to monitor assist calls made on the operating system so we can be able to monitor like process execution but we also more importantly able to monitor Network and file activity so interactions as to file system so it's all it's basically a giant like it's difficult to set up it's not as easy as it makes it look when you start reading the logs but it's once you have it set up is very useful what we use it for is we use it for like file monitoring and for network connections specifically so those for this this is

actually it writing one of the this is my one password vault being written to disk before staging so you can actually see staging and different types of file interactions from the malware using audit and the bottom run is a network event so if you might notice here if so what would actually be there under bar run mdns responder would be the IP address that it connected to but if it does a domain like resolution like on the host it shows up in a different log but this is very important for you to be able to monitor the network events from processes so if you were ran into a log where you got em DNS responder logs you

could look up the actual domain and the IP resolved to at the exact time it did the resolution using mdns responder logs so for those of you that had have like older systems you actually can collect these logs in the system log file but once you upgrade past High Sierra you kind of lose all of them and you have to write your own system in order to get this data back so I provided a short like execution log for a command line so that you can go do this yourself and collect all the same logs that you would have been able to collect before they got rid of it in high sierra this is these logs are super great to have so

bringing it all together so once you have the ability to make process trees file interactions network activity you're able to get really detailed context for execution events on the system so this is an example of a malicious word dock being executed on Mac OS system you can see that it's curling down from some file hosting site definitely not Dropbox downloads to this Google Drive it writes it out to this liked impact underscore pack dot pyc you can see that that actual file writes out yet another serve file probably for encrypted traffic so being able to put all of this in context together allows me to like say that this is probably worth looking at versus just having any one of them

like a curl is not necessarily by itself malicious but seeing in the context of like word executing curl or you know executing a Python binary from the internet is like really good to know and being able to have like good alerts for it okay so back to proton this is a picture of a proton rocket it's a soviet-era rocket it still exists they use them to launch heavy stuff really far we're pretty sure that the rat is named after the rocket not the coming particle so let's talk about that thank you for the track it's really full-featured the stuff Mike described is a significant subset of the things it can do to you it does include a socks proxy capability

also which is super terrifying because if you're in the corpnet work it just you know need VPN you don't need VPN creds you don't need three factor authentication you just socks proxy while they're uncoordinated and get whatever they get it's great it's also written in Objective C which is pretty unusual for most rats today for Mac OS it's very professionally developed and maintained their response is good they have a ala carte pricing sheet that you could choose from all the way up to buying Europe buying the source code directly and rolling your own C to it was first advertised on a Russian language forum by this acronym veksler on January 22nd of last year and

keeps getting updated and revised back sellers somebody you pay attention to if you play that game there's even a great YouTube video it's not the greatest production quality but it's worth noting there's a YouTube video for the rat there's a website also this I think gone through several iterations now it used to be behind cloud flyers not anymore if you want email to author that's his email hit him up tell me wanna buy stuff so there's a press on Easter Egg which I don't think anybody's ever blogged about we saw the activity and we thought we'd I mean we've reversed it a few times now so we thought we'd throw it out it runs the shell command which is

pretty cute all it does is remove logs we're not sure why he did this echo business but it runs every single minute to arm the logs and it throws this string in and I'm not that I don't have much nerd cred so I didn't know what this was so I had to google it I was like what what is this that's this guy fn-2187 so it's an author is apparently a Star Wars fan given who this character is in the Star Wars movie I'm not really sure what he's trying to say maybe there's some deeper meaning I don't know but back to the prior slide this string is really unique in the world really really really unique you will

likely not find this if you do dynamic analysis of malware your rule is not gonna help you but if you run this through sand boxes that you can control you will not see that very many places in the world ever so it might be a good thing if you do that kind of thing so a few words about telemetry we use em DNS responder which again is built into the OS to do passive DNS analysis or passive DNS flogging through the fleet at scale it's incredibly cheap you can D dupe on your storage end it's great you should all do it we use OS query in Santa to log hashes binary hashes in addition to

a bunch of other things including signing bits as Mike was talking about this is built to be an application whitelisting solution so santa itself logs even when you're doing when you're just doing detective controls and doing auditing it will log certificate information about the designer for the software that's being run and if you recall this is required because gatekeeper requires Mac OS software to be signed by a legit Apple ID so that's another interesting thing that the Santa does and not a lot of people talk about so given that we have these different data sets about about hashes about behaviors about DNS requests and about IP connections let's enrich those things with some context and go hunting so a few words

about logging log everything it's really not that hard its foundational in many cases and it's the premise for all the way we do everything like you really have to log all the things so we log a ton of stuff binary hashes DNS requests IP address connections those are one of many many many things we log I encourage you all to do that too well get all this for as long as you can we try to keep hot storage for called 18 months is ideal if you can keep it all in one place make a key value store this is sort of a that we'll talk about in a few minutes so if you have a key-value store of a

bunch of things that you want to look at say for example hashes you can do a daily roll-up it's like if you speak a dupe it's like a daily some jam you can roll up what you've seen that day put it into a key value store you don't need to keep all the data about which host it was on etc etc etc you just know like hey on this day this hash executed on this day this DNS request is made in this day the IP connection was made so that way you can perform at least Rs reactively without having to wait 13 years for your query to finish or like you never building this foundational key

value store give you the ability to query external entities about new stuff so that's kind of like the the crux of what I'm going to talk about a couple of minutes is this external enrichment for data that you already have so you have DNS requests you have IP address connections your hashes enriching these things and acting on them is super powerful so you can do enrichment a few ways we're gonna talk about DNS and IP mostly in the next few minutes you can ask a bunch of external entities about DNS data you can ask a bunch of external entities about IP idea you can even do some like command line who is command line who is a super cheap if you have

zero cash you can't spend anything you can just do that all day every day I recommend you buy a commercial service they're not that expensive and they provide a lot of value beyond just what we're describing you'll see some screenshots we're not here to recommend vendors but you'll see some screenshots so stuff you may recognize depending on if you've seen their tolling before out of scope for this talk we're not talking about enriching binary data at all all we're talking about is new hashes and new DNS and new IPs so there's a whole different scenario for binary hash data which we're not going to cover at all so acting on enrichment use basic math like

this is not actually that hard so you can say hey on a build a function that weights a bunch of different metadata around DNS names and around IP addresses you can build a function that weights these different components that will suit your environment you can make quality assertions based on the output of these these functions in worst case don't do maths you can start with them not doing any math at all you can start without figuring out what coefficients you need for like privacy protected who is etcetera just don't do math if you really want to start start the easy way and just surface an alert or engage your IR procedure do further collection further analyse whatever your

environment does every time you see a new hatch connecting to a new domain and a new IP address or even a new house can you can do a new domain with an Argentine IP address so let's pivot back to the Hambrick incident itself Hamburg club is is one of the command-and-control nodes for the Hamburg ensign there are others and I can talk about them Henry type is is the one most people I think we're impacted by so we're gonna query Whois data to enrich to enrich this domain name and make some decisions based off of it firstly registrar's named silo that may or may not be important it may be important to you that's up for you to

decide the most important thing for me is that the registrant is privacy privacy protected and the second most important thing is the creation date note the creation date is very close to when the handbrake incident happened so if we recall back to the prior slides they started on me second domain was registered in 429 Wow gotta go fast we'll look back at this IP in a minute so this IP address is where it resolved up until five ten since five ten it's been resolving to this eighty eight address which is like Telenor I'm pretty sure that's a sinkhole with a bunch of other stuff on it this is a screenshot and Whois data for a hammer a table is

again it's a commercial tool I cut their name out so DNS key key takeaways is a newly registered domain close to being newly observed by you a super meaningful is the registrant privacy protected that could be meaningful third some register I see a lot more abusive activity like for example if you see Bitcoin DNS as a registrar you may know that some cific actors prefer that that register for the obvious reasons building context on the dns and the IP history together is also super fruitful you might find other domains that you want to enrich on given that vivid so this is the IP the average thought is was pointing to during the incident PSN not particularly noteworthy you will

note the DNS history for this IP address going back probably seven eight years now includes a bunch of sketchy sketchy domain names Carter's gate is what it sounds like it's a carding site there's a bunch of those sketchy ones that are varying degrees of carding tangential or carding adjacent yeah you'll see down at the bottom is part of this gate there's a bunch of those sketchy ones so I think a key takeaways your ESN is roughly equivalent of a physical locale so we all know it's sketchy neighborhood when we see it the same principle applies to ASM and registry or registrar excuse me so newly observed I keys in your fleet are worth noting the IP owner can be

super important don't rule out VPN providers because they they're opposed to a lot of malicious activity these days as well DNS history of an IP might give some meaningful context and when you put these things together it might be a great indicator of suspicious activity or a new supply chain attack or just a malware in general so we're gonna skip over this because I think we're running out of time another Mac OS client software called atima got compromised same proton stuff if you were to perform these techniques pivoting off of the domain name and the IP address and the new hash this would fall out pretty quickly registrar's enum for this domain name etc etc if you

pivot on all these different registrant characteristics and attributes they're all garbage there's like one domain in the world with this email address and a bunch the other stuff is actually faked it doesn't make any sense yeah here's a bunch of domains that we're also parked on this IP address CVV shop is what it sounds like that's more carding stuff so we'll wrap it up so again just to reiterate continuous house telemetry is what you need get it for all the things you can get it for including IP hash and the NS requests easily searchable data stores for have I seen this before are super important enriching those key value stores for news newly seen things is a great

technique you should think about applying and if you apply this when you see a new binary hash coupled with a new DNS request with a new IP address you're probably going to be Patrick Laurel to a blog post so extra credit we don't have time to get into this so I'll leave it as an exercise for you all this is the ccleaner indicators so if you applied this technique to see cleaners you wouldn't have had to wait for a month you could have done it like in a day or in a couple hours depending on how fast your analytics are so trends we're seeing thread actor is moving to GCP and AWS and other legit cloud services so

the IP reputational thing is not as meaningful anymore doubling down on on DNS and hashes is a good idea don't trust Google just because it says school GCP I I have malicious stuff in GCP like I run I mean so these guys anyways thread actors have a trend towards living off the land this is not just for Windows it's been super super super hype on Windows last few years it also applies to Mac Python is installed by default as our other interpreted languages other interpreters threat actors moving towards these techniques five PowerShell and Python and other scripting languages so doubling down on DNS and IP nourishment is super helpful there and you might want to get better

logging for PowerShell microsoft includes that natively you just have to turn it on actually collect it I highly amend it pythons really hard maybe Mike will talk to you about yarra about that this is you Mike sorry well we'd like to thank a couple of people for helping us get content for this particular presentation Brandon Dixon and Bryan can Kenneth delicious fabulous candles fans we'd like to thank for their talk alphabet soup burning three actors with data writing floor and David Herold for their sands CTI talks oh s query team you guys are amazing Google Santa team you guys are all so amazing and whoever the saint is that works in open BSM so I shot it

at Apple I would really just like to thank you personally if I could ever find you but uh please fix our bugs yeah and I PC and I think we're slightly over time oh sorry that's it we're gonna go thanks if you have any questions to the stuff we're happy to catch