GSuite Digital Forensics and Incident Response

hello is the mic on oh cool sounds good alright so my voice is almost gone but that was as of yesterday so I decided I'd come through and still do the talk so forgive me for the crackly voice so my name is Megan Roddy I'm currently a senior security analyst at reconsent a small MSSP startup I recently graduated with my master's degree I have a few security certs I do this regularly this is like 12 times speaking at a security conference and my first time with technical difficulties I knew it'd come around eventually and that's my Twitter handle I'm almost up to 600 followers so I'm almost internet famous if you want to help me

get there so I'm not gonna spend too long talking about gee sweet how many of you use d sweet at work just a few how many know what G sweet is

everyone knows what T sweet is kind of so it's Google's like hey business version think of it of the as the oh three sixty five of Google it's gonna be Google Drive Google Mail Google Calendar all synced across the organization so as a start-up MSSP we serve small small and medium-sized businesses G Swedish cheap and so we encounter a lot of g sweet instances and so I started to learn deeper in G sweet hence the origin of this talk so first we're going to talk about a bit about defer G sweet versus traditional and when I see traditional defer I'm talking about your traditional Windows domain environment a bunch of domain connected computers emails on the

on-prem exchange server connect and your users are connected via ad everything's local machines so obviously there's a bunch of different cases of deeper but that's that's kind of what I'm speaking about when I say traditional deeper so I'm not going to go too much into the red team from Black Hills InfoSec that's this link at the bottom and I'll share out my slides later so you have it but the kill chains gave me the same reconnaissance they're gonna find out what your G suite domain is and after that try and figure out what it counts exist so that's going to be recon that you do with other email address findings so you find out the company uses the

format of first dot last name at corpnet so then you just generate a list based on the list of employees at LinkedIn some malware or breaking in through insecure firewalls and here we're going to talk more about brute forcing phishing social engineering exploitation getting those credentials using the methods access we had Keith's persistence and we'll talk a bit more about that later command and control there's there's not as much of that command and control aspect of malware calling home because the attacker can access the console from wherever they are they're not remote going into a domain environment then there's a lot of things they can do with the account they can act fill data they

can abuse the account for their own purposes so again the great talk about how to read Team G sweep I'm gonna move on and talk about the blue team side of things there's a whole range of different incidents that you can encounter you can counter malware phishing denial of service web attacks and that's just a short list starting off and some other the attacks and these can all be seen at a traditional environment but the the sole ones that we're kind of really focusing on and G suite are really related to the mail and the file storage which are fishing potential for leak of information and their potential for account abuse so those are do fall under

traditional d4q but that's the the focus of what's gonna happen what your guess C&G suite for the attack the G suite is hosted on Google servers and you're accessing it through Google in the cloud so you're not gonna see the attack vectors like SQL injection web attacks because Google's got it pretty locked down and if you do experience that compromised then there's a whole bigger issue it's Google's fault the platform is owned by them so in a traditional environment if they may have gone through your firewalls they may have exploited a web app they could have fish or social engineered or brute force and that's what the only thing we're really gonna see in G suite deeper is that

method of getting login credentials because that's the only way you're going to access so in terms of the environment again traditional deeper we're talking about big networks tons of computers servers workstations network devices firewalls and in terms of the configuration I mean you might have a de tu Active Directory or something like that to sexually configure like the workstations but really you're gonna have a bunch of widespread configurations if an attacker gets in they can reconfigure the local machine settings or if they access a network device they can reconfigure those settings so you've got say spread all over the place where the G suite we have a single platform that Cloud G suite platform and you have all the core

configurations set settings centralized to your admin console and so when you're trying to find out what the current settings in the environment are or if somebody might have changed settings you've kind of got that single pane of glass as opposed to logging in to 40 different types of devices trying to see where settings may have changed so kind of the summary of that is that traditional v4 when we're talking about regular d4 we talk about a large attack services a ton of different types of they may be responding to and looking for evidence or artifacts there's tons of sources tons of configuration tons of data stored in different places where Z sweet you've got you have to get a log

in to log in you you need specific there's maybe a few specific incident types that you encounter as opposed to a bunch of different methods and all your data and configuration settings are contained within the platform so I didn't spend too much time kind of going on the whole overview because I think that's more useful to be able to show you guys what what I'm putting into practice so this is the scenario details are going to be pretty generic NDA but this is something an actual incident I worked so it's it's coming from it real life it's not made up basically we got an email saying oh so it was a company they had a bunch of clients rich private

people and all their clients specific to their client list it seemed like maybe their client listed leaks like those people are getting spam and phishing emails related to this company and so it seemed like someone found the client list but they don't publish the client was publicly there's no no way they could have done publicly and I thought well the only place that we really have all that information would be in the emails or files or contacts in our D suite instance but we have no idea like we weren't tact as far as we know we weren't fished so so how did they get in so they contracted recon to come in and find out where we compromised so

starting this incident we know there might be a compromise nothing has been done because they don't know if there's a compromise and we need to find out everything we need to find out who compromised them when where they compromised how where they compromised why were they compromised so like I said performing SQL injection bypassing the only real entrances by logging in whether that was because they brute force the credentials are fished or whatever so you can export the law on it logs in G's suite of all the logins and this is the less detailed version it just says IPD event you can also see whether they were logging in using single sign-on on another application or

whether they were using Outlook or something to log on but for our purposes we just want to see all the IPS well here we have three unique IPS pretty pretty easy to parse through you do some who is see if anything strange going on and if you did you'd find out there's two u.s. IP addresses and then there's a Chinese IP address and if my calculations are right Bob cannot log in from the US at 6:00 p.m. oven be in China at midnight so something's were wrong but that's all fine for this but so a lot of it for me at the time was manual review I uploaded a lot of them into like a geo IP lookup and then kind

of scrolled through cross correlated which IPS were which users so it took a good amount of time but sure enough we have actually found huh strange Alice this and then five minutes later she logged on from India and so there was kind of definitely sub abnormal activity and you may think so like maybe she was using a VPN or something that was messing with the IP address to be a different location but typically our clients we're dealing with their outsourcing their IC in cyber security for a reason so they're not using fancy tools that would mess with their IP and sure enough so then we have we have this one IP so we have the IP sorry Allah so

once once I did this work so I did all the work and he was like wow that was really manuals so I'm gonna make it not manual now that the incidents over but now you guys don't have to do what I did so he wrote a python script you you upload the CSV that I showed you and it plots all the logins and it's actually color coded so like red his failed login yellow is one thing boom is successful long and it's plotted on a map this is pretty usable activity especially the company we knew they occasionally went into the Bahamas and stuff like that so you know nothing strange about the Mexico Cuba type of logins but then when

you plot this at the same time that starts getting a little suspicious that you have a cluster of lions in the US and then you have these blue successful logins in Africa a few minutes later so this is really cool tool now that's open source you go to that blog post it has a link it's actually an interactive map you can click on the little points and see what exactly it means and so like if I click on that point it says Alice logged in at whatever time so now it's all automated but before we manually reviewed the logs so at this point we know that there's definitely something weird going on and we know what user was

affected we also have the time stamps for the logins so we know now that this attacker has been in the system for four months and they hadn't noticed until now and he's been logging in from Africa on and off over the course of the past several months so yeah winner is Alice and four months ago so we start our containment phase of incident response we disabled her account we reset the password on her account we leave it disabled until the entire scenario is complete and also Google the cool thing is you can reset the lock in sessions from the admin panel for a user so that attached credentials also get cleared so that that attacker was using cached

credentials it would force those cookies to be deleted and that attacker would have to real aughh in and with the password change that didn't work so we come back to our where are we now we know whose account was compromised we know when it was compromised at this point we know that the login activity we're seeing with the compromised account is not happening anywhere else those strange abnormal overseas logins are only happening with this one account so we're not 100% positive but there's a good chance this is the only account that's compromised at this point what have we done we've disabled the no compromised account done a password reset in preparation for it being renamed and we've reset every existing

session so now we don't need to know all the things we have a few things look we need to know how did the attack happen how did they get in what was the account used for once they were in and was there any persistence in place after they got in so first we we think about the methods that they could have gotten those credentials failed fancy event a success unless they happen to guess the correct password on the first try but what we we didn't see that we didn't see a bunch of attempts of failures of n/a success so it was kind of indicative that they they had the password at the time they were entering it so I thought

well it's phishing that's when we hit a brick wall because at their lowest tier you get all the things you get Google Drive you get Gmail all their applications but email retention is 30 days and the attacker has been in the system for 4 months and so our best guess is that she received a phishing email she of course was like no I didn't I swear but one it's been four months two users always say they didn't receive the phishing email so we would have liked to search the logs from the time period that the login got like that would have been our ideal thing see what email she received at that time and looked for the email that said like

fishing in the subject line but we did not get to do that because it was a premium feature so our the best we could get due to the retention periods and the amount of time the attacker had been in the system is most likely based on the methods that the has and based on the logs we do have it was likely fishing so then we want to see what did they do when they were and Google they log everything there's no like turning on blogging it's not like Windows where you go to do an investigation and you're like cookie show me those logs and I well we didn't turn on logging so logging is all turned

on so we export all the logs the screenshot I put there because it's uh we use the hives if you haven't looked at the hive case management system it's an open source case management system and it's a great platform you should look it up but we created tasks to review everyone logs analysts picked them up put eyes manually reviewed the activities specifically we kind of looked at it overall but we're really keying in on that one compromised account trying to see what did they do what did they walk into what did they change any settings did they do anything strange with email etc etc again we ended up not really getting anything from it it didn't seem like they did a

thing they just came for the comments they from what it seems they logged in and they looked around and that turn up aligns with what we were told like heat off the company - the thing is that somebody was taking proprietary information customer data client lists and using it for other purposes spamming and fishing and stuff so in some sense it's not surprising it also meant that there wasn't much more evidence to find because they got in they pulled the client list and they kept logging in over the four months on occasion so it looks like they were just sitting there reading the data seeing what they could use from that information so it was a

breach of confidentiality rather than integrity so going back to persistence this is theirs we basically had to manually check all of the settings of G suite to see if any of these kind of methods exist it'd be nice if I had all the elite programming skills and could automate this but I just had the ideas like because G suite has an API where you can call the settings so like ideally you'd be able to call all these settings and check them but I'm a plebe and don't program so I just mainly check all these settings but it said the ways you can think about how they would persist because obviously you don't have scheduled tasks or startup registry keys

to make your malware ever you want run so there's at passwords so of course I get to the point where we say if 2fa would have been in place new have been good well adding to FA after if they're at passwords may not help at all because at passwords are free when you have like a mail client and you have be sweet you have to FA enabled both the mail client doesn't know how to handle to FA so instead you have to give it basically a password only account that's specialized for that use but if I'm an attacker I have the password I figure they're gonna finally get smart and put to FA on I can

basically sync to my mail client with that specialized password that doesn't require second factor token you need to check to make sure there's no app passwords really that the attackers added and remove them if so there's a lot you can do with api's in Google and you have to authorize the API but if while I'm in the system I create api's that will send me information and datoria approved it's just gonna keep going so that's another aspect you need to look at going back to the two fa thing if I'm afraid to a phase I go on the account like I'm pretty positive I could get back in social whatever but now they're gonna have to

FA I can add my own to sa device that I now use once you enable to a thing like I turn it on for you but I set up on my phone too I could if I notice you've enabled it while I'm in the account I can download those backup codes so that I can retrieve he's the backup codes logon later in case of this person who was really just interested and it seemed interested in getting proprietary information if they were smarter they would have set up like email forwarding and forwarded all emails to their own personal account so that even when you're kicked out all her emails are still going to you so you don't even

have to log in because she's basically forwarding you all her emails and they get email filters similar things setting up redirects or something so that emails go to you that's also a method of staying hidden once you get in is making sure that you have email filters like if there's any google suspicious login alerts that those emails aren't being delivered so the key moral of the story is use two-factor authentication if you aren't logging in via the web use two-factor authentication it's it's your password it's like the only thing that is really truly without a doubt gonna keep that account secure and it was one thing when everything was on prem and you couldn't access email easily for outside but when

you're using gmail to log in you should have to affect a enabled for everyone that's that's kind of the if only they would have moral the thing is people think 2fa is hard and executives don't like to do 2fa so they just say it's not a requirement so when you inevitably come across a company that was breached and they didn't have 2fa on the lesson is that like Ron Swanson and bacon you should be prepared as an incident responder you are going to come across environments that you may not be familiar with tools that you may not be familiar with I was lucky because working with an MSP it wasn't an internal incident where there's an

incident and it was go we had the whole contract signing an NDA and all that stuff so I had a little while a few hours where I knew what was coming so I read every single D Suite documentation I could actually the kind of the reason I started writing this presentation I've wrote it written a blog post on it was there wasn't much on D Ferengi suite specifically but I read all the documentation I my company uses D suite so I walked into our admin panel and I clicked on every single setting so that when I got into the clients environment I would feel comfortable and know what I'm doing so that idea of if

you could encounter it be prepared for it and the the other flip side of it though the environment can change the core fundamentals the court procedures related to Incident Response are gonna stay the same yes you may click on a different button here there or you may check for different things but overall you're still following the NIST Incident Response framework of preparation detection and analysis containment eradication and remediation and post incident activity that's not going to change it's just the mindset that you need to bring with it and you need to adjust that mindset especially as we're moving into non-traditional environments we're moving into cloud environments and other situations where kind of the things are going to work differently so

I have to know that there's less limited ways to get in but there's also more ways they could in different ways they could establish persistence I knew I wasn't going to get a hard drive and run memory forensics but I knew I was going to have a bunch of logs regarding I'd have to look at so it's that flexibility that we need and I are especially and defer especially as environments are changing there's moving to the cloud there's things like containerization like is this like kind of a specialty like I'm even forensic people in this room who feel comfortable getting a hard drive and doing hard drive forensics would you feel comfortable if I gave you a docker

container that you had to perform forensic analysis on it's those kinds of things where you may not come across it now but there's a good chance that one day he will come across it so even if you can't prepare for every environment in terms of the exact controls you need to use you should be preparing to alter your mindset based on the environment you're going into my talk we ran a little short but if you guys have any questions I'd be happy to answer them yeah

well in my situation the clients we work with our SMBs and they don't have anyone that's where you ask them who their IT team is and they're like well Jerry took a computer class once and really kind of knows what he's doing so we don't get in contact with their team we did I mean we did talk to the client I say like did you click a phishing email four months ago but like I said like one even if they were gonna tell me it's unlikely they didn't remember and sure enough they said no we haven't um there is the logs that they do have our pretty detailed and what what we would do in this situation

obviously this was like a one-time they called us for IR but for our clients who we perform regular monitoring services on if they have d suite we'd use the you can forward those logs to a sim with their API so we'd get those in the sim and even though Google's not keeping them for more than 30 days we would keep them for more than 30 days so that's kind of the preparation phase of IR making sure that you have what you need when the time comes that you need it yep they were using Google Drive for company documents so um yes so it's kind of like so it's kind of cool because it's not the exact file mimic metadata and the

says that you see on a Windows machine but you can't look at the files and see the tanks date and revision history we did review the files to see if any had been altered there'd been no altering there'd been no sharing that's another method of persistence kind of that it touch on you could set the share the Google Drive files to be publicly accessible so even if you kick me out I can still go look at all those files I shared but there's no evidence of messing with the settings related it seemed like they didn't have any persistence they were just looking and at the time we we did not have we're not able to look into whether

there was methods of seeing who view defile that'd be interesting for further research but at that point we did not I don't believe they did it was all up there so yeah I mean it is possible again the the date oh it wasn't clearly there it's not in the metadata whether it's um a downloaded it so it is possible that they downloaded the files locally but it's getting more research figuring out where the evidence is for that yep

um the you're talking about Google's archiving functionality yes so that's one of the paid features where it's not on the lowest here is like tiers two and three yeah it I think they have it the one below two which is like Google business or Google teams or something but at the lowest tier you get the products and you get 30 days email retention logs seem to stay on there more than 30 days it's only emails that roll over but yeah they there is a built-in backup functionality if you pay the additional amount yep any other questions

[Applause]