Tim Crothers - Investigating Macs at the Speed of Compromise

welcome to 10:00 a.m. on a Saturday I alright thanks for coming and and I want to if you're all not aware go out of your way sometime today to thank some of the folks in the orange or the black besides Augusta shirts if you don't know they do all of this volunteer just so we can have a cool conference crazy amount of work goes into a conference like this so please take take a moment as you're passing by just shake their hands and tell them thank you for enabling us to do this so this morning I'm going to talk about investigating max as most of my talks this starts with the story about a year and a half ago I was

starting my journey into increasing my ability to reverse engineer Mac malware and you know it was one of those weekend mornings down on like this blood level in the caffeine system was a little too high and I may have inadvertently self infected my laptop that I was doing the analysis on one of the interesting things with Mac's in particular is it can be done you can create virtual machines with Mac OS X but Apple goes out of their way to make that difficult and so a lot of times I just use an old Mac that I can easily reload for this but that really got me thinking wow how do I really dig in to understand where

all of the forensics artifacts are my background for the last many years has been a lot of my time spent on the ir side of the house and well I have a fair bit of the classic forensics training I really like the sniper forensics approach so if you're not familiar with that term coined oh it's been a while now probably close on ten years Chris from over it over it spider Labs coined the term and really the idea here versus the old-style forensics is all about picking out the data points that are most interesting to us right if we are doing a particular task what is what are the forensics artifacts for instance that are most

likely to help us get just that so that we can investigate quite quickly we can always take our time and do a full forensics workup on a host but especially in a situation where we've got adversarial activity suspected it's good to have techniques and as I dug into this again the the reason why I ended up putting this talk I I assumed as I went on this journey that there were lots of resources available this was pretty straightforward mac OS is kind of a combination between mock and bsd kind of a hybrid so i figured well a lot of my linux investigation skills will work kind of is the answer and so what i'm hoping to do this morning is

just consolidate a fair bet so if you're looking to investigate especially in nir triage situation hopefully i can jump start that journey a little bit for you and point you in some of the directions this is probably an intermediate level talk in that i'm assuming you've got some ir fundamentals assuming you've got some things like basic skills if you don't have those you want to get some background i do have a bunch of references at the end to give you to help with some of those so let's start with some basic stuff first and foremost this isn't actually exactly germane to the talk but one of the more interesting things I discovered as I as

I dug into this is that so Microsoft's still showing my screen let's try there we go so the Mac operating system has a built-in antivirus called X protect and interestingly enough X protect as it turns out uses a combination of yarra and a configuration file I'll talk about in just a second called the P list why this is relevant is because if you want to create yourself and quickly build a Mac malware Xue you have all of the kind of latest information in here pretty readily available on finding yourself and hunting down some malware hopefully you have access to virus total or some great resources like that it's it's kind of an open source a be essentially

freely readily available to look at by the way I'll have a link to the slide deck at the end so don't feel like you've got to try and capture so a few other kind of core things that we'll see through the course of this morning dmg is an apple disk image file kxt is essentially the the equivalent of a device driver for OS X as compared to Windows now the interesting thing with both applications and device drivers equivalents on OS X they're actually directories so if you go out to the folder you'll see the whatever dot K ext that's a directory not a file go into that directory you'll see all the sub components including things like the

property list files for the configuration that make up that driver one of the I found that the OS X platform has a really interesting combination of really closed and really open in that a lot of the forensics attributes that we're interested in are widely open Microsoft Apple documents them quite well on developer.apple.com so you'll either find something there and well documented or you'll find nothing when you find nothing that's usually by design it seems property list files are a key they're basically just a JSON text config file that has configurations now there are both ASCII and binary versions so for the binary versions there's a plist buddy tool really I it's on the system by default from the command line

by the way I'll be driving almost entirely command line I will show a few graphic tools but this is designed so that I can SSH to a machine capture the artifacts that I want bring them back and do the examination on a separate system so it's just running plist buddy with the plist that will actually help you create property lists as well dot app is the application files so slash application all the dot apps again full dynamic library files which are the equivalent and packages which are really just about like Linux tar files okay so now we we're going to use a sniper forensics approach so of course there's a few key things we're likely going to want to go

for to determine on our hey is this host infected do we have malicious activity and we're just gonna fairly quickly this morning walk through all of those to give some backdrop so basic system information pretty easy really detailed so system underscore profile I like to go for the XML file option output detail level you can set to low medium and full if you export that into a file specifically called SP X then you can open that with the system information out so again let's pop over and since it is early in the morning I'm just gonna copy and paste some of my commands rather than torture you with lack of timing so I'm in a folder OS X triage

here I run the command like I said this just runs as a normal access you'll get some activity that shows as it's running here this is really comprehensive compared to a lot of the windows you're gonna pick up a real wealth of details on the host all of the drives the accounts that are on the system a real large amount of depth and then if you just flip over to the system information to all which of course you can just open with spotlight system information and then if you do a file open you can open one of those that's been gathered remotely so here's my OS X triage folder we can open that file up and you'll

notice there's a lot of details on the hardware the network software that's running just a ton of information in there yes in case you're wondering I I do have my laptop named in Mandarin that's a bad habit I picked up from former days just to give the security people stress just because I mean that way memory memory capture an analysis this is problematic right now so current version of OS X is 10.14 if you're running the latest Catalina unfortunately the two best tools that are out there and these are the ones that have the most recent support at least that I've been able to find and aware of are recall and volatility both of them only cover up through 10.12 in

particular Apple introduced a ton of memory protection mechanisms in 10.14 which really inhibits a lot of these tools working well so memory capture and analysis at the moment is a little bit problematic unfortunately good news is so far in investigating I haven't had to resort to bad to that other than the advanced stuff file system analysis so this is at the heart Linux derivative under the hood lots of roots in BSD as I mentioned earlier and a simple script I highly recommend link is out there one of the kind of two resources that's mostly current is OSX Incident Response scripting and analysis there's a pretty good script in there firewalker dot Pi that I recommend will

do a really good job of capturing and it creates two files for you we've got file info dot text and file timeline so the permissions the file info text the permissions are very similar to your standard UNIX system permissions so you can see here dot d s underscore store 644 it's a regular file that's the file types of file directory special file etc the UID that owns the file the group ID that owns the file the size of the file and then any special characteristics like you can see user bin right has a set GID flag on it so any Flags like that so it just captures that all in the file info text file timeline is

converting all of the time stamps so OS X uses a timestamp based upon zero time so it's it's time since January 1st 1904 its unique OS X for a start time but it works just like Windows and UNIX systems they just pick the difference and so you've got a capital this will convert to UTC automatically as part of its capabilities of course you can have it to others and there's these several time stamps accessible there we've got access which is of course the last access date modified date changed so the difference between changed and modified is modified will get updated when the the all sorts of characteristics happens whereas change typically just gets updated when

the the the actual contents of the file change and then birth is the creation date that's the date the file was created now HFS the file system for OS X is relatively safe I've yet to see any of the malware intentionally updating but the same caveat applies with any forensics investigation those timestamps our ultimate ly just records in in this case a b-tree index in the file system and subject to buyer beware they certainly could be updated again it's pretty easy to actually run those tools so in this case now of course you are gonna want to run this command as find the yep as as sue do a lot of the commands you're gonna run a run as root

so of course you can capture all of the activity and so I'm just running the file Walker it's in the chapter four directory I'm telling it to start at the root of the hard drive and put its output in a directory called triage is what I'm doing here and of course I'm being prompted for my password now I mentioned this one first because this of course will take a little bit I find on most of my systems it takes it not terribly long fifteen to twenty minutes it's it's pretty quick at iterating through but of course that's going to vary depending on the number of files the size of your hard drive etc so while

thats collecting let's talk about some of the key directories slash applications is where all of the user wide applications will go slash library are things like the application preferences the configs forum all of the logs should be now of course all of this is caveated that people do writing software can can do stuff however they want when I do an actual demo here in a little bit you'll see that the malware we're gonna hunt down didn't follow all of these exactly although some of this you have to slash system is the core files for OSX itself slash user is the equivalent of slash home write pretty straightforward volumes is where the mounted volumes go dot vowel is a really interesting you

see it I'll use it a few times as we're walking through the demo what happens is if you actually go out into Davao and do an LS it's empty however if you use the stack command on a file what you'll get is the inodes of the actual file if you cat a file or more a file well however you prefer to look at the contents of a file slash dot Valle slash inode slash inode you'll get the contents of file it's another path to the actual file i've actually gotten to the point where i prefer to go that way to the file just because if I've got keyloggers or other things going on that lot might be

looking for regular access to the file that I'm interested in it's an alternative path to get at the contents and then a lot of the same are a lot of the things you'll see are actually symbolic links to the slash private so things like the syslog directories etc those are actually all hanging off of the slash private directory just convention that Apple has let's pop over here a second that one's still going

so if we look there notice we've got temp we've got bar we've got Etsy those are all pointing to that private directory so that's that's where we'll end up actually looking at for potentially a lot of our things when we go for system logs and stuff like that in just a minute then of course we've got a whole bunch of files that are if you're familiar with Linux are the same we've got things like been user cores etc those are equivalent they're used the same way etc a lot of those are sub directories off of slash user capital you user for instance for individual files that are stored to an individual user rather than for the

entire operating system a few other interesting files that are unique to OS X dot d s underscore stores that's the customizations of a particular directory so that's got things like sort preferences for a directory or in there things like any particular icons all of that is in the dot deist or dot spotlight spotlight is the indexing system the search and indexing subsystem within OS X so that spotlight - should be 100 you'll find one of those in the root of each volume and that's literally the current index of the contents now the the format for these is really accessible most of these are either a b-tree file format or a sequel Lite format and so they're really

accessible if you want to pull those and extract the contents out especially as a cross-reference against the file that can that can be quite handy as you might have metadata never indexed is a skip so if that's in a volume route that I'll you move that entire Drive gets skipped over by spotlight and then of course you can have either folder name or file name dot no index which means it's it's hidden to again also spotlight notice of course all of these files start with a period that's standard Linux convention meaning they're hidden by default if you take an SL of course just do an SL - a and you'll see all of the hidden files

certainly a lot of the malware that we're seeing right now in OS X takes advantage of the period so you want to be I don't think I ever take LS anymore without adding a - a just just to see everything and then some useful files tools that are built in X attribute is just a tool that shows extenda attributes not every file has extended attributes when you do a directory listing at the end of the permissions like you would normally see for Linux permission so you're 64 for 755 etc you'll see potentially and ampere saying are I'm sorry an @ symbol if there's an @ symbol what that means is that there's extended attributes associated with that

file doing X attribute on that file name will show you those extended attributes MD LS will give you the metadata the OS X system captures metadata about almost every file it's automatic it tends to be very rich a lot of applications support the metadata so especially things associated with web utilities like your chrome your safari etc will have very rich metadata things for instance will be in there like quarantine status indicating hey what was the URL but it was downloaded from things of that nature this is again just a JSON file easily accessible and viewable stat will show you the information on the file particularly that's useful for the inode values associated with a particular file

and last but not least little file if you're familiar from Linux investigation that tells you the file type of course right and again let's take a look at some of those real quick pretty easy peasy our file listing finished we will use that here in a bit so let's do get to the right screen so here's our OS X triage folder notice here as I mentioned that at symbol so the DF store file has extended attributes I'm typing well [Music]

and I can't spell correctly alright so extended attributes just tell us that the DF store file is associated with the calmed a pelear info application often it's it's simple things like that I'm using the path for X attribute because I worked my my path up the other day infecting my laptop and I still haven't bothered to straighten it out MD LS of again let's do the yes store file notice we've got all of these different configurations associated with it if we do another file let's do my PS list for instance you'll notice again you can see the the JSON nature to the file lots of potential interesting attributes in there of course our stat on the PS list and so then I mentioned

earlier if we do a more slash fall let's say 1 6 7 7 7 to 200 and then you just put a slash between the two numbers of the inode 2 and you can see we can access the contents via that path rather than just the path same contents okay and then of course file which gives us our file type pretty straightforward standard standard stuff a lot of this is really straightforward really the the biggest hurdle I found was finding information about where all of this stuff is and what it is and specifically current because Apple makes new promises on any of this of course it is a closed-loop operating system although any of this type of

information they do a good job of updating it as they go so one of the key things that's really really valuable for us that's often a question we have is what was downloaded so the key caveat here is this is maintained by just applications that support it the good news is the vast majority of applications that run in OS X support the quarantine attribute so this is just a file it's just a sequel light file current version is dot B to the old version was big surprise be blonde that was I thirteen or dot fifteen ten dot 13 or 15 I forget off the top of my head they changed that but just look for the

quarantine events comrade Apple dot launch services and so what this is is a sequel light database of all of the files that are downloaded so in this case just for sake of ease of viewing I've got a sequel light browser here installed and I've just pulled the file up on my laptop here and you can see it's got a unique gooood that's associated with every file it's got the timestamp so the timestamp is that Mike you know why I want to keep signing Microsoft today Apple time stamp based upon the 1904 zero-day there's lots of converters for that readily available notice you'll see which application downloaded this so you can see I've got Chrome lots of different things it has

the full string that was used here you can see over on the right and then the original URL that was which is often the same as the as the strength that is comprehensive it doesn't pull over at least not that I found so the the maintenance length on there is quite long and just an incredibly useful tool to start with in terms of finding the file and I'm driving the poor poor patient video folks crazy going back and forth between my laptop sorry about that video team so running processes this is standard Linux PS aux redirected into a file run it is route of course sudu so you get all of the running processes if you're familiar

with looking on a Linux system you'll be very familiar with this connecting externally unfortunately again our best tools are the old Linux standards so netstat dayit - or list of open files - i - i is internet connections both of those are good just mainly depends on which format you prefer then persistence let's talk about how do you do persistence there are a lot of ways to do persistence so the first is the local user will all have a calm down a pallet items that plist that is a local th user so in my case my username is Swann all so it'll be slash user Swann ole etc comm Apple so that's the per user again

list and best way to view that is good old P list buddy

[Music] pop over so user Liebig Zach this plist buddy I gave earlier I'm just going for my in this case I'm using till the till the yep I'm logged in as local user so again JSON style file we can see here so this is just basic ste for configuration data property lists by the way this is an example property lists are well-documented so just go to developer.apple.com you'll get all sorts of details on how to build etc and so in this case just if you look down towards the end of each block that gives you the application name so I'm running this my personal laptop I'm running steam I've got iTunes helper automatically firing off caffeine is the tool I use to keep

the laptop from going into sleep mode VMware fusion and last but not least VIPRE VPN so those are the applications that I've got set to auto start as part of my local user configuration and those will be set for all of the users in the system a great place to start now I mentioned earlier that the applications themselves are really a folder so if you go to applications whatever application name dot app contents library login items you'll find again if now this directory doesn't exist for every application it only exists for the applications that have auto start characteristics with them then we have launch agents and launch daemons a little bit of nuance between the two but effectively they're they're

the same thing the ne and the library directories are universal so those are going to be agents or demons that fire up for all users again just take a directory of that /system /library is supposed to be the mac os/x only launch agents and launch demons as you might imagine if the malware has access they might not necessarily abide by those standards they are going to need route to put themselves some sort of root access in order to write themselves out to those directories but we shouldn't overlook at users username library launch agent I have found pretty consistently so I've analyzed about 30 pieces of Mac now we're so far as part of the last year in

pretty good detail most of them are going I'm finding for the login items plist or the local user launch agents if they've got higher than they do go for the library launch agents have seen a few they're good ol crontab works still on Mac of course Apple doesn't recommend that but it works just fine so of course a crontab - l will list out all of the cron tabs normal caveat applies right that you've got local directories that you've got to be aware of as well kernel extensions are device driver equivalents as I mentioned earlier and you can have at jobs as well so all of those are persistence items that we've got to be aware of browser

history so there's a few places that are pretty universally possible in particular the users user name library cache java cache folder so when Java executes on OSX it actually stores all the jars and idx files which there's a great parser at that github link to parse the idx file format it all of those are there of course a lot of the stuff associated especially with web-based threats certainly has Java characteristics associated with it then I've included the specific different browser so in the case of Safari they're doing at almost everything in property lists history downloads last session really straightforward similar to that quarantine sequel I'd database I pulled up earlier really detailed lots of lots

of good stuff in there and the cache DB is also a sequel Lite database so that's a cache of the folders then Google Chrome Google Chrome is using a sequel Lite database so I've included a few queries to parse some of that and interest the time will keep moving Firefox similar it's also a sequel Lite database lots of good data readily accessible of course there there are lots of other browsers out there now starting with dot thirteen 10.13 Apple changed their logging format so they have what they call Apple system logs in private bar log SL those are much more akin to event log on Windows much more detailed they're stored in a binary format what

you can do though is you can use the command line syslog or you can run console dot I app and import the ASL files and of course standard fare here applies for the log files right if you've done any investigation work on Linux it's gold you should certainly capture those because they're really really rich the audit logs in private var audit are all associated with the security tools so gate keeper which is the sub system within OS X that hey you've downloaded for this from the internet do you really wanna this is an unidentified developer all of that stuff is coming from gatekeeper things associated with X protect any of the other security tools will be in there

and those are in deck based upon the start times PR audit is the command line key things you're looking for in those logs so local terminal accesses you want to look for a login bracket PID that's going to be the events associated with the local login service you're looking for user underscore process that's the login events dead underscore processes the logout events the log off and there'll be timestamps of course with each of these entries same thing with login window that's the interactive GUI window ssh of course is sshd same convention for that one as well then there's screen sharing which isn't turned on by default but is often turns in on I found it's essentially the

equivalent of BNC for OS X those are called screen sharing D that's the event type you want to look for in the logs to find login events associated with screen sharing and then of course privilege escalation events are always important to pull su bad su after the su means it was an unsuccessful su attempt if it was successful there's just no message it just says whatever username to rude or of course you could su to another username as well on and it'll have the the device terminal information sudo will have the username and the associated information including the command so the Oh sex does a good job of logging those for us bless you and let's

show a little bit of a demonstration let's put these together quick alright so get back to my terminal that's not the direction I want to go there we go all right so in this case I'm gonna start with good old now let's just sudo su because we're gonna want to run most of this as route yes aux process list all right and just for sake of seeing open this there we go pretty standard process list now again I'm I'm I'm gonna cheat a little bit in terms of just move along pretty quickly but it's the standard types of things that you would look for with your standard Linux approach right so you know in particular this seems a little a

little iffy we've got Python running users SWA no library containers folder dot PFD which of course makes it hidden by default unless we do a dash a and then a file called KF etc that doesn't feel particularly normal so let's take a look at our files associated with that LS o F so we'll do a - I - P and our process ID 46 - 73 oh let's do that in to a file rather than to the screen

while that's going let's take a look at that file so again where was I there we go so here's our file

let's just start with that parent directory first see if there's anything else in there nope just that one file no at symbol at the end so we don't have to bother with any X attributes and that's not gonna work well is it thank you there we go that makes a little more sense so not a lot of metadata there associated with that file let's see file type

file type Python script ASCII text with very long lines that's not suspicious at all right and then last but not least let's do a stat on that so there's our inodes let's do a more slash dot ball / 1 6 7 2 o / 8 6 o 9 5 4 8 7 9 no not enough twos let's let's use copy and paste for my I did mention lack of caffeine too early in the morning right sure enough we have a Python script doing an exact of a really long base64 that it's fine right again we're running running along on time how about we of course want to potentially look at at our thing so I'm

gonna jump here copy and so if I look at the local launch agents right so I just did an LS on users Swann l library launch agents one of the persistence I gave you notice there's a cop dot Apple dot rendom sounding file name

and of course key things we also would want to do look at quarantine some of the other downloads how the heck did that get in there right we look at file and date associated with the birth date of that file that's running look in the file system this is standard IR triage type you can find lots of ways so wrapping up there are a couple other options if you're willing to run agents I highly recommend both Google Rapid Response and OS query those aren't really germane to a lot of these in my opinion because it takes you running the agent before for this to really be effective for your needs I include a bunch of reference material first and

foremost if you're at all interested in Mac forensics and you're not reading Sarah Edwards blog you you are missing something that is pure gold sir Edwards is hands down one of the best folks on the Mac forensics froak there's just all kinds of great materials out at Mac four six and six if you're really interested the evil OS X is the malware I was running I've got it running as a rat so questions let's get to questions quick yes sir yeah debugger I uh so I personally for the Mac forensics actually my favorite is good old binary ninja that's my preference of choice for the stuff otherwise I ride mostly command line but binary ninja does a

great job of course if you're familiar with Ida Pro I just Pro stuff works well for the Mac files as well URL to the talk I'll get it uploaded today I promise sir

I haven't so far I try and avoid when I've done it so far I've done DD just good old DD is what I've used today there are some commercial tools available that's really besides DD the only thing I found for the file imaging in the cases where I want to go that deep I'm usually doing a full local hard drive capture personally today sir

certainly certainly there are patterns like the ones we just saw look for the periods at the beginning look for things like the random filenames although unfortunately I have found instances where legitimate applications are also running the random filenames part of the reason why I recommend the yarra file at the very beginning is that's a really good cross sampling of the Mac malware what I started my journey is I went and researched all of those to really understand what were the approaches that those use to help me really key in because I like to part with well if real malware's doing it I should definitely understand that right that's that's really the keys there unfortunately all

right with that we've got let's see we've got a couple giveaways that I'm supposed to give away let's see I will ask an easy one what what is a dot K ext folder I said kernel driver would you like the lock picks or the blue okay your choice for answer first alright so that was KX T let's see and then I always forget to prepare hard questions in advance whose blog did I just recommend oops all this for Sara Edwards absolutely thank you so much for attending hopefully this gave you a little bit of a a little bit of an insight I haven't got the talk uploaded yet to my github but I promise it will

be today as soon as I get a few moments in the speaker lounge don't hesitate to reach out to me on LinkedIn it's my preferred way of communicating and everybody have a good conference thank you very much [Applause]