The Silent Breach: Security Threats in Google Workspace

So welcome everybody. Our final speaker of the day is Kong Nuion. Uh the session is the silent breach security threats in Google Workspace. Uh Kong would love your questions so please ask them in the Slido app. Uh or you can go to bsidessf.org. Um as this as this is our last session, I want to thank you all uh for spending the day or the weekend here with us at Bides. We really appreciate uh you coming and also to see a full house for the last session of the day is also very impressive. So kudos to all of you. Uh and K the stage is yours. All right. Awesome. Thank you. Uh thanks everybody for coming to the last talk of

the day. I know everybody's really tired after two days of conferences. Um so uh today I'll be presenting uh the silent silent breach uh navigating security threats in uh Google workspace logging shadows. Uh this research was uh done in collaboration between uh Shu Jang Wang and Cominate. Uh Shu Jang is the head of security research at Obsidian Security. And uh on the comminate side we have Rex. He's the co-founder and CEO uh of Cominate and and myself. Um a little bit about myself. I started in the industry doing vaugh research uh on mobile space uh mostly focusing on um kernel space and then I moved over to uh red teaming side to focus more on exploit and post

exploit dev. I loved it. Stayed in that space for a while with different employers and then I came back to doing full-time research briefly before joining combinate earlier this year as a founding security researcher. Uh so at comminate uh briefly uh our mission is to provide the best uh AI workforce for every security team and my job there is basically be a sparring partner to the AI sock analyst to improve its capabilities. Uh we actually uh score number one uh in human efficiencies uh for uh 2024 Defvcon sock competitions. So uh that's enough about us. Uh so for the agenda of today uh we'll first look at why are we even interested in audit log in the first

place. Um and then we'll go over some of uh Google Workspace attacks uh like uh Chrome extension back door uh domainwide delegation attacks uh and business email compromise attack. And in each case uh we'll look at what the locks uh we'll examine what the locks look like uh and what they're lacking and what could be improved and augmented in in each uh scenario. And then lastly we'll give some uh final thoughts on the subjects and some takeaway lessons. Um so why audit logs? Um so other than compliance reason uh logs are really useful in incident response as well as uh threat detections and hunting. Uh without logs it would be really challenging to trace the steps

that uh an attacker takes within the network. Right? And even if you're an offensive person, uh knowing what the locks look like would would be very beneficial to you so that you know you can craft more stealthier attacks there. So and in the case of of Google audit logs, uh there are two main uh sources. Uh one is the Google workspace and uh the second is the GCP log. Uh however the quality of of Google log has a lot to be uh desired and certainly can be improved as as we'll show in the next sections um of of the talk. Uh so well first let's dive into uh Chrome extension backd dooror. Um the most well-known uh Chrome extension

backdoor incident in uh recent years was probably the Cyber Haven incident. Uh Cyber Haven is actually a a data security organization. uh their Chrome extensions was discovered to be uh compromised at the end of last year. Uh this incident uh appeared to be uh part of a broader uh campaign targeting multiple Chrome uh browser extensions. Uh this incident started by a fishing attack uh compromised one of their employees credentials. Uh this allowed the attacker to publish a malicious extensions um version of the of the Cyber Haven Chrome extensions which uh excfiltrated cookies and authenticated sessions uh from targeted websites. Uh the malicious extension was I believe was active from December 25th uh to 26 when most people were actually out of

office. Uh so the threat actor went on to compromise more than 30 additional uh Chrome extensions. uh they appear to use uh sophisticated fishing techniques to um authorize malicious ooth applications and exploited vulnerabilities in in OOTH authorizations. Uh the estimated the total estimated impact of this campaign was around uh 2.6 million users. Uh so to understand this kind of attack better uh let's first talk about a bit about the the Chrome web store access management model. Uh so each organization that publishes Chrome extensions would have uh one dev account that they pay the initial $5 for. Uh and this dev account uh would be in charge of updating uh publishing the extensions. Uh Google doesn't actually

officially support team roles or access delegations in the web stores. Uh so if you have a dev team, you either have to share credentials internally for this dev account which is not ideal and not recommended or there's another way that you can do it um which is you can create a GCP project uh and then request the Chrome web store scope from the authorized dev account and then upload update the extensions and and publish a new versions. Um this is actually uh a way that that some companies use to automate their C uh CI/CD pipeline for the extension development. Um and and also uh typically uh on the Google admin uh admin console uh third party uh app

access can be configured restricted uh via uh security access and data control uh API controls. But as you can see in the screenshot there, web store is actually not listed as a as a Google services uh to to restrict access there. So now that we understand the the context around the Google web store, uh we can go back to to what happened in the in the Cyber Haven case. So the fishing email uh disguised as the official communications uh from uh Chrome web store uh developer support uh claimed that the cyber havens extension violated policies uh and was at risk of removal. Um this email was directed at a a privileged extension developer uh to

um directed the the developer to a a malicious ooth app uh named uh privacy policy extensions. You can see in the screenshot there uh which mimics Google authentication process. Uh the dev uh eventually granted permissions to the app. Uh the threat actor was able to back door the extensions and uploaded a malicious versions of the cyber havens browser extensions to the web store. Uh this version included uh code to excfiltrate uh cookies uh session tokens and other sensitive datas uh from users. Uh and and also this this malicious extensions was able to pass web store security review. Uh so the the security gap that was exploited here was that the threat actor leveraged the the oath

authorization flows uh which bypass MFA. So even if the developer has MFA enabled uh no MFA prompts was triggered during this ooth process. So to uh to reproduce these attacks uh we uh we created a Google cloud project uh named my extension publisher in this case and then we enabled Chrome web store API uh configure OOTH uh consent screen created OOTH client ID and use this information to craft a fishing URL send it to the extension developers. uh the ext the extension developer approved it uh as you can see in the in the screenshot there and we get back an authorization code uh with this authorization code we can issue request to get access uh get

access tokens and refresh tokens. Uh with the refresh token you can keep requesting access tokens because the access token is only live for about an hour and with this access token we can then upload and publish uh ARM version of the extensions there. So after the attack we look at uh at the logging uh see what's being logged right. So typically uh an oath app will leave an audit trail as an activity event but not for activity in web stores as you can see in the in the screen in the bottom there uh is uh only an authorized event was was locked. Um and and if you look further into that log, the only details

you get is that the uh privilege user uh privileged developer authorized uh the Chrome web store scope to uh my extension publisher. Uh other than that, uh there's not much else that that's being logged here. Um so for better logging, security teams will kind of have to script the the public Chrome web store page and compare that version that you get to an internal version periodically to see if there's any change. All right. So, uh, next we'll we'll go into the domainwide delegation enumeration attack. Uh, this technique was discovered by the Axon team. Uh, I believe was a year and a half ago. They reported it to Google as a vulnerability. Uh, but it's less of a

vulnerability. It's more of a feature that's not really a bug. Uh, so things haven't really changed. It's it's still very much a technique, still very much alive, and you can use it tomorrow in your next op. Uh, something like that. So basically in short uh when a Google service account configured with domainwide delegation is allowed to impersonate any user in Google workspace with predefined scopes. So uh a compromised domain wide delegation service account can lead to lateral movement to Google Workspace after successful enumerations. So to understand this uh attack better uh let's first talk about the relationship between uh GCP and Google workspace uh with regards to identity management. So when you use Google cloud

uh for managing uh users um to managing users groups uh the hub for managing users groups and settings can be either Google Workspace or a Google cloud identity account. Right? So as a result uh Google workspace also has a crucial role in managing user identities across uh Google cloud services. uh and even if an organiz organization uses a thirdparty identity providers like Octa or Azure AD uh Google still requires authentication through their identity as a service mechanism which means that organizations still need to sync their third party identity providers with either Google Workspace or cloud identity and and Google Workspace um domain wide delegations allowed identity object uh object which can either be an external app from Google Workspace

Marketplace or internal GCP service account to access data across workspace on behalf of any users. So to enable a domain wide delegation, a super admin has to grant access to the service account by uh going to Google admin console security access and data control API controls and then go to manage domainwide delegations and click on add new um as you can see on the screenshot there. Uh and then they have to uh take the the unique ID or the OOTH to client ID of the service account and granted permissions to some scopes. So as as you can see here when when you set up domainwide delegations for a service account uh the service account is linked

to a Google workspace app uh with the service accounts uh client ID so that you can set up which oat scopes this service account has access to. uh in this case the surface account is linked to an example app on Google webs uh webs uh I'm sorry on on Google workspace that's uh authorized some a lot scope such as uh email uh drive calendar there uh so and under the hood right in in the case of of a service account as the identity object uh the flow goes along the line of a JWT token is created and signed by the service account key uh and then this JWT token includes scopes and target user and the token uh is used to

exchange for access token. And this exchange is only successful if the scope and the impersonated users are both uh legit. So to reproduce this attack, uh we created a scenario where the attackers have gotten access to the service account keys. Um service account keys can be compromised via different means. uh like in the case of a developer's endpoints compromise and the key is stored locally uh the attackers can enumerate the endpoint and grab the keys there uh or it could be leaked on GitHub or if the the thread actor exploited uh an RC to get inside a GCP VM that has an attached service account they can curl the meta metadata server to get it. Uh and then this service

account uh has domainwide delegations uh in Google workspace. So we leverage the service account key to impersonate a user and enumerate the emails with a with a a snippet of code that you can see there. So uh after the attack uh we look at the logging and and and again there's only there only like two useful events that you can see in the log there. one is the the the authorized event is locked there. But uh but what we're not seeing is that no failed attempt was locked or any token exchange attempts were locked and even if you go further into the log, the only details you get is that example app uh call message.get

on behalf of a legitimate user. Uh so so it really does doesn't unveil whether it was domainwide delegation access or it was a user delegations and and so like to mitigate things or augment the logging cap like lack of logging in this case uh we we recommend strict access control on GCP service accounts with domainwide delegations and also apply least privilege principles to the o scopes being granted to them and also monitor for API calls done by Google works. space apps on behalf of users. And then lastly, we'll uh go into uh BEC, a business email compromise attack. Um and uh typically uh the in in the case of business email account attack, uh threat actors would take over

a victim's account. Um usually through uh adversary in the middle fishings. Um once they get into the mailbox, uh they can browse emails, they can hijack um uh payment email conversations. uh reroute transactions uh and also uh create mailbox rules to hide correspondence to avoid being detected. Um and to reproduce this we we also created a scenario where we compromise the the user's credentials. We log into the account. We create an u a park um email rules to delete any emails that that that has the subject uh security alert in there. And as you can see uh some of the things are being logged is is view uh open send reply link click etc etc. But what we're not

seeing is the the rule creation was not being locked. Uh and also when you go into further into the locks to find more details for forensics you wouldn't find either user agents detail uh sessions ID uh or device ID being locked there. So to mitigate the the lack of logs in this scenario um you can write some custom code to monitor changes in email inbox. Uh in this particular case we wrote some code to pull email filters periodically using a service account to impersonate a a user and pull Gmail filters. Uh you can also use the Google Gmail API at the bottom of the screen there. uh user uh the users um settings filters list to to pull filters for a

particular user ID as well. And then what you get back is that JSON there uh which uh include the ID which is the the unique filter ID. Um then the crit criteria field is the condition that that triggers this filter um which in this case is the the filter looks for incoming emails with uh subject contains a security alert and the action is is what to do when this criteria matches which is uh move to trash. Right? So uh so takeaways um to uh to to recap all this uh so audit logs are critical to cyber defense teams um and and some of the you know mitigation for this is hopefully Google will improve loggings

to add more details that useful for forensics there uh but you know in the meantime uh you know for each of the attack like uh Chrome extensions back doors we can scrape the uh the the the web stores uh page to to get the the newest like periodically to get the newest versions and com compare that versus uh internal commit uh to see if there's any change there and uh for things like domainwide delegation enumerations uh apply uh least privilege uh principle to GCP accounts with uh domainwide delegations and ath scopes uh being granted to them and also uh monitor for API calls done by Google Workspace apps on behalf of users uh and and for things like BEC or business

email compromised type of attacks. Uh leverage uh Gmail APIs to create uh custom monitoring as as we as we showed in in um previously. Um but but really invest in in intelligence oper uh automations, right? So the complexity as you can see uh the the complexity and and the volume of cloud audit data make it impractical for for man manual investigation to catch every back door ooth abuse or business email compromise attempt uh in real time. Uh so you can also build a system to autonomously monitor uh correlate or investigate activities uh across audit logs APIs and use your behaviors to to augment human defenders uh detect stealthy attacks earlier and and ensure that no critical

signal is is missed. Uh if you want to bounce ideas on this, I'd be happy to discuss more offline. Uh you can uh reach out to either uh Rex or myself. Uh these are our our emails and uh the QR codes for for our LinkedIn. Um that's uh that is uh it from for me. Thank you for attending my talk. Yeah. Thanks Kong. Um one question if anybody else wants to add a question feel free to.

Oh, question is, and I love how this question is phrased. So, thank you for to whoever asked this. On a scale from awful to so so bad, how how would you rate uh GWS audit logging? Um, well, I mean, I I'm sorry. I'm a vendor, so I I can't really rate it on this scale, but but as you can see, as I mentioned in the beginning, it has a leaves a lot to be desired. I'm I'm going to leave it at that. But but I I show you everything you can get from the locks already, so you be the judge. Okay, so good answer. Yeah. Uh what do you think about compliance frameworks for Google

Workspace like the Scuba program? um for compliance I I mean like you know like it it decent but in this talk um other than compliance reason I I I really I really wanted what I really wanted to get across was that you know the details that were useful for us in in the threat hunt or a forensic scenario which you can see is uh there's not much there so so yeah oh another one came Yeah.

Do you know of any good thirdparty products to ingest and monitor logs? Any other suggestions for GWS log/alerts to set up? Um I mean in terms of products I'm I'm not aware of of much. uh but as you can see uh Google doesn't provide a lot of details in the locks but what they do is that they provide really good API so you can write your custom rules as we did in two examples there so you you can go back to our examples to to see how we we write uh custom alerts and and monitoring and and you can leverage that uh like I mentioned in the case of business email compromise uh the the

Gmail API is actually pretty comprehensive uh and then also um you know I think I think the the the The only place that really lacking is the web store which which what we found is there's no API to monitor there. So we kind of have to scrape the web store page like the public web store page uh manually to to get the to get the version number. So so yeah I mean Google has actually has pretty good APIs uh to to write uh to write uh custom alerting and monitoring. Yeah, I think this is our last one. Have you dug into workspace user takeout or domainwide data export in Workspace? if so thoughts. Uh we did it a little bit but

uh but uh that's actually part of of our our next part of our research. So so so hopefully you you'll follow us and and and we'll publish that very soon. That's in the next couple months. That's a great segue. Sounds like we're going to need a a part two hopefully at Bites next year. Absolutely. Yeah. Thanks, Gong. Appreciate it. Thank you everybody. Yeah.