Inside Google's Discovery & Remediation of a Critical CPU Vulnerability

all right thank you okay uh just the quick intro I'm Yousef hin uh I'm a security engineer at uh Google uh I very recently switched teams withing Google and currently I'm in a team called in uh infrastructure Enterprise infrastructure protection uh and it's basically the team that is responsible for building security controls for Google's infrastructure and that was very recent and up to the point of changing teams I was actually working in another team called vulnerability coordination Center and what I'm going to be talking about here is more relevant to my past role and I'm going to go through some story of an engagement that I led of a response to an identification of a

vulnerability which uh I did when I was in the my previous team so Yousef husin uh the team that I was in is the vulnerability coordination Center and that team the core responsibility of this team is to lead the remediation to Super critical big big large scale vulnerabilities that affect Google products as you probably can imagine sometimes addressing vulnerabilities is not as simple as just push a button and update sometimes it might involve a lot of people a lot of teams a lot of systems and getting them to fix things sometimes a little bit more chall challenging than you would hope and sometimes it's not really just a technical problem sometimes it's it's it's got like communication legal and

documentation artifact creation it's really it can get really really messy very easy imagine if this happens at the scale of Google it could even be more complex than you can imagine so this is one of the core roles the team plays the other team that I'm also part of is the team that responds to the bug Bounty uh researcher reports so I three these reports when they come in when a researcher finds a bug in a Google product and then I would three it learn what it is reproduce the the exploitation of that particular vulnerability and then work with the developers on how to fix it now the bug Bounty part isn't necessarily involving critical uh

complex vulnerabilities that need somebody from the vulnerability coordination Center to lead but sometimes there is an escalation path that happens from the SP Bounty program to vulnerability coordination Center when we find something super interesting because I'm part of both themes uh I'm the only one at Google that actually does both sometimes I would be sitting on the uh in the B Bounty side and I would find something that I rearch which seemed a little bit maybe small and then I find out that oh well this is actually pretty big and then I would escalate it to the vulnerability coordination Center to this other uh person really good person named Yousef that would take the response to that vulnerability and Lead

it so I would escalate it to myself and I would basically do here and

there how do we do that the the the vulnerability coordination center team is an interesting team because you really have to be pretty good at a lot of things and that includes like M leadership working with other teams in terms of like the technical capabilities and how you can roll out things but at the same time taking into account all of the different complexities that surround fixing vulnerabilities now that I introduced myself uh I wanted to introduce this person I I tend to be a little bit nervous when I do when I do talks like this especially when I'm surrounded by super smart people like yourselves and so I thought it might help me to have somebody kind of help me

with the presentation in a way and this is him his name is Leo I don't know him I just got to know him this morning he might come in throughout the slides two or three times and uh I've heard that he's a good some people told me he's a good security engineer some other people told me he has no idea what he's talking about maybe we'll find out throughout the presentation there's going to be two or three times I'm going to say something that is so so bad and it's just so wrong feel free to scream and let me know when I do that let's see if somebody can cash it and feel free to do the same if he

says something like that disclaimer this talk is not technical

I guess I li it's a little bit technical little little bit not much how many of you can actually look at something like this and be like oh I know what this is I can read binary just like that and uh and I can tell you like for example oh this is a machine instruction for running a hello world program or something like that probably not many that's fine I can do it it's very difficult and uh the reason I mentioned this is because I'm going to go through some story about a response and I just wanted to kind of build a context and just cover a few Basics just to make sure that we're on

track and we're all on the same page so as you know when you write a program and you compile it and you run on a machine it's just a series of bits that's how the machine runs things now for humans to be able to read these machine instructions there's like a way to step back from binary a little bit and you can read something that makes sense human readable more human readable more human friendly to read right and uh this is probably very familiar to people that work with like malware so I've been doing incident response focusing on incident response for the past like nine years and I did a little bit of reverse engineering for

malware and sometimes we get to deal with these things where we actually have to read a form of C to be able to tell what the program is

doing so the first time I was looking at binary and I was like oh this is difficult I won't be able to just look at binary data and be be able to tell what that is and then somebody would tell me well don't worry we can use a tool called the disassembler I'll basically tell you this from the binary I looked at this the first time and I was like wow that didn't help

I was a little embarrassed I had to wow surprised right like okay but then I just couldn't lie about it it's like what the heck is this so move maybe that means move move e ax ebx what this does is that it turns out that for the machine to perform a calculations and a lot of the the the the operations that it does would have to move certain things around bits that we moved around between boxes they called they're called registers uh and to do that uh it would have to copy data from one register to the other and then it will do something with that data and basically what this particular instruction is is basically just a

representation of a machine instruction that does a copy from a source to a destination and in this particular case there two registers from ebx registered to eax okay simply I think and then there is another instruction that looks a little bit more complex because it's shorter CU in this case we have the instruction but we don't have the operand we don't have the where is the source and the destination what move but what is it and the reason for this this is because this one uh the operand of it are implicit so when you run an instruction like this there's a bit of setup that happens in the background that you don't have to worry

about but it's already implied where data is moving from and to and then there is this rep in front of it which is really short for repeat do it more than once Okay so we've got rep move what happens if I write a machine instruction like this and give it to the machine rep rep move it's kind of like telling someone I'm going to do something more than once I'm going to do something more more than once kind of doesn't make sense right like so turns out what happens is that it would be ignored it wouldn't cause a problem if you give this to the machine to run it you do this two times yeah

same thing things will go just fine so that's one concept they just wanted to cover and there's another when I was talking about the operand the registers ebx and eax registers this particular operant has a size back in the days 32-bit registers 32-bit machines had eight registers how many bits do we need to address the eight registers three bits

so that was enough but then I had 64 bit how do we address 16 registers three bits in the machine instruction or brand well what they did was basically hack it a little bit to make it work and they were like well remember there was this prefix that gets ignored if you add it more than once well that's not going to cause a problem if we add a prefix but then we can basically just use it to lend us a bit to the three bit that we've used be previously and that will end up with like four bits and now we can address 16 bit 16 registers in the machine okay so that was covered one more thing I want to cover

ER M fsrm these two are CPU optimization features that were introduced in some of the Intel uh CPU families the names are not very friendly erms fsrm erms I think is short for enhanced repeat move store fsrm short for fast short repeat move I have no idea why they come up with these names quite honestly I mean I'm pretty sure these are brilliant people but I I can bet like they could have come up with something a little easier uh so what they do what these features do is they an optimization feature that will B basically make the system run a little bit more efficiently when it does certain operations so I was talking about these repeat operations

earlier these repeat operations need to run in a way that is more efficient and that's that's what the CPU will do with you with erms but erms had a problem if the data that you're moving from one source to the destination is under 128 by then it's not really worth the investment to do the optimization because the optimization itself has a little bit of an overhead preset up that it will do to make it work if the DAT is under that limit under the 28 by then it's really not worth it but more than that then it will make the system a little more efficient and that's where fsrm comes in fsrm is an enhancement to that feature that will

basically make it work with data that uh that is under 128 by the last concept that I just want to cover or it's a bit that I want to cover is this if anybody knows what this logo is tell me okay all right that's enough I'll get back to this thank

you all right a response to story so now let's get to the story that I promised I was actually going to tell you now that we know what this is

so researchers at Google do a lot of interesting stuff of course and one of the thing that they do to identify vulnerabilities is some various ways of fuzzing one of the thing that they do sometimes is they get basically get one program have two identical copies of it with certain characteristics that are different and then they would run it and if the system behaves differently as a result of one of the programs from the other then something isn't right so basically they would introduce a lot of different conditions and then they would just randomly fuzz it then see what the system is going to do so what happened was when they were doing the fuzzing they introduced certain

conditions they added a prefix in a system that has the feature of fsrm and in tell uh CPU with certain conditions using the prefix on a system that supports ifrm sometimes it leads to this by this I mean a crash now generally when we talk about a crash it's bad news right but in in exoit development and hacking especially it's it's near pretty good news especially if you're the person developing the exploits right like it's a lot of times it's actually quite a milestone you just want to get to a crash and that's basically what repar was there a vulnerability that was identified as a result of This research that causes a denial of service

or causes a crash causes a machine check exception on a system so if you give a certain set of machine instructions to the machine it would crash so how do we respond to things like this at Google uh so first of all we start with the discovery phase which is something that we already went through it was discovered and this was escalated to the vulnerability coordination center team and I was assigned to lead the response to this how are we going to fix this and let's get it fixed initially you would have something called an embargo period where nobody talks about this outside and we just have to go through the process of identifying effect for this vulnerability and fix

things that are critical before we just publish this and we would have to establish that with the vendor in this case was Intel so we're going to go through certain phases and I'm going to go through from the start to the end explaining what we've done and just to make this a little bit clear sorry I I just couldn't help it uh but I had to add this just to illustrate okay we're there so what do we start doing initially something called an impact assessment and we have a standard way of doing that at Google oh we've got a vulnerability we want to assess the impact and that basically means what we have to do

is how bad answers questions like how bad is it how can we reproduce it is the reprodu is the reproduction of it pretty easy can anybody just exploit it if they know the vulnerability is it does it require some setup across Network or not a lot of things like this we have to understand just to understand the exact risk of this vulnerability but then we also have to couple that with the products that are affected so that we can tell for example we've got like s machine running in a Google environment that is affected for example so that's different from just five machines right so we've got the impact assessment the first thing we want to

answer is well this is causing a crash is that really a bad thing like okay you're just causing a crash right like so I'm just going to say something causing a crash isn't really a problem right because you've got your rack you've got your servers and you're running things in isolation and you if you just crash you're crashing your own systems

right did somebody say yes okay well I mean in today's world I mean we all know right you share more than you know right like with others when it comes to like Computer Resources right uh can somebody tell me what comes to mind as a product that could be affected by something like this what was that Crow strike

okay huh that's a very good example in fact I would say that's the number one example but thanks for sharing so Cloud environments right now okay so this guy is saying it's no big deal I guess we kind of disagree with that so like I said Cloud we share things in Cloud and turns out that if you run this uh the set of machine instructions on a guest machine you would cause an MC machine uh check the exception on the host Machine level which basically means you can just open an account cloud provider that has a lot of machines that using Intel CPUs how difficult is that pretty easy right just openm and then run the

instructions and then you cause a crash on the machine you don't know who you're sharing it with it could be with someone very very important another C customer and then you cause the machine to crash so Google Cloud was the number one we were pretty concerned because we was like okay well Google cloud is a big product and we've got a lot of stuff that we have to do this is how it looks like basically it's pretty simple you've got the repar X deoy you run it on an attacker VM you cause the physical machine to crash and you was gring uh crashing other virtual machines that run on the same host machine I mentioned earlier this

Borg I asked what it is and somebody said Borg so a little bit of history back in 2000 around 2003 Google thought okay we didn't need to use these machines a little bit more efficiently in general so we wanted to have an orchestration system that's going to orchestrate how we run software across Google in our data centers globally and that's where work uh came in it's basically the system that uh orchestrates running of software workloads across our data centers across all all of our clusters globally so you would do that in containerized uh uh systems and then later on something became public and Borg was the origin of it something public that does pretty like

something very similar to this I think became public that does something very similar and I all know about it can can somebody tell me what that product is kubernetes exactly so at Google we use Borg until today pretty much everything at Google runs on work and like I said because of the nature of it containerized environments machine that are shared that means are they affected by a vulnerability like this I think it is right in a similar fashion to Cloud now when it comes to systems typically that are affected do you think that's all can somebody tell me a name of something that might be affected that I didn't

mention okay we've been talking about servers you think potentially client devices could be affected so of course because you know they Al some of them also run uh Intel CPUs so one of the things that was effective was also Chrome OS devices and but these are client devices right so you could crash client devices in the it's it's it's it's it's very difficult to do we didn't have a pro of concept for exploiting this in client devices back in the day when this vulnerability was discovered but in theory it's possible to do that through a browsing session but you would end up with crashing the client device but my friend Leo is saying no it's not

a big deal I kind of see his point here right because was like yeah I mean you're not really sharing your laptop right with someone else most of the times and if you crash it yeah I kind of agree with him like well it's just an annoyance right you're just going to reboot it I mean like there is an operating system that we all use on love if it does an update it will just slap you across the face and it will do the update right like and you just have to restart the machine and so it's really not that big a deal so but here is something I want to mention which was important when the researchers at Google

discovered this vulnerability they were testing and fuzzing on a black box which is the CPU that we don't really know how things work internally as much but when we reported this to inel they'd be able to do a little bit more so they looked into this more and they were like oh interesting turns out that with this you can do privilege escalation

so so I think the little pration is a little bit worse than denial of service or crashing machines now in that frame of uh thinking I think you you might be able to say Well for now this applies a lot more to client devices because then if you exploit a client device you can basically privil uh privileg eslate and then you can own the system so one of the things we did was do this impact assessment we have all these products that we know that they're affected uh we have this document called an impact assessment these s of products what are the number of the Machin that are affected which basically translates to just a bunch of I'm sure you all SQL

uh and then we would look at these machines get these numbers write them down so now we know how big this is and then we come to the question of how do we fix it now of course there's the solution of we can wait for the vendor to fix it and that could work that maybe it won't work but even if it works there is a risk with this problem because this vulnerability could get leaked if the if the vendor takes like let's say two months there's like a time frame during which this could get leaked it could be used by someone and I'm sure you'd agree that a lot of people would be interested

in getting something like this in their hands so we thought well how about we wait for the vendor Intel and at the same time try to develop something of our own we keeping our back pocket and see what how things go somebody came up with an idea and they said well there's a solution can anybody tell me what she's trying to say

okay so he's saying chicken bit so the chicken bit it's actually true there is something called the chicken bit sometimes in the CPU when they introduce a new feature it would introduce a bit that you can flip and it will basically disable that feature I think they name it the chicken bit because if you own a system and you enable that feature you have the chance to Chick out of the feature so we could disable that feature that we talked about earlier the optimization feature which was the requirement for the exploitation to work and now you're protected right so it works can somebody tell me why that would be a problem given what I've talked about so

far what was that well yeah that's one way but I was actually open for something else I was talking about it being a performance feature right yeah so it affects performance it uh because it is an optimization feature so when you disable it performance goes down by about like 10% do you think this would be something acceptable like in an environment like Google be like oh gcp let's just go down 10% Like no it's not going to be that simple but it is a solution like if uh you know uh things go wrong like really just go go south and this get saked and we really have to do something it could be an option we're going to have to pay

the tax of going down 10% and then we're protected against exploiting vulnerability right it's probably better to go down 10% in performance than being hacked right so we did actually obtain the fixed from the vendor which is a micro code update to this vulnerability and then there was a period of of testing it and this testing happens between a few large companies Google and other similar companies where we would do the testing on this fix for about months or something like that about a month and then we can basically say well that works very well which means it's likely going to work very well for everyone else so for platform for Server platforms we've got defs and for line

platforms also got the fix now for server is a little bit more complicated because we've got chicken bit and the micro code update and we have to talk about performance for clients it was really easy we had the update published later and then it was just applied on client devices so we can roll out that update to the server platforms when we tested this it got to the point where we can say it's PV validated P validated production validated PV which basically means all these large companies tested it and now we tell the Intel about it and then we're ready to publish this at the end of the Embargo share it to everyone so the server micro code roll out is when

we wanted to basically estimate how much time is it going to take to update all the machines at Google that are impacted by this but then there are a couple of strategies on how we can do that so we could reboot all the machines and apply the update like a bios update but I was just talking about performance rebooting machines could cause an issue especially if you're running things on gcp you might be impacted I don't think you're going to be happy if we just tell you hey we rep puted the machine because there was a security update and you're running you know a machine critical application on it then there's the other approach called Hot Load which is a mechanism

with which you can apply for for more update without rebooting the machine and this was tested it was successful on a bunch of machines a big sample at Google and we were confident enough to go with this so we went with that uh updated strategy on gcp and Borg pretty much all the affected infrastructure it was successful and it was transparent now when I say it doesn't have an impact does it really not have an impact like zero I mean if you're really going to be looking closely at the CPU Cycles you know or or if you have a cockroach into the CPU and like looking yeah you're going to notice that there is definitely

some impact but it's not the kind of impact that affects an application that runs on that machine and who does that like who actually looks that much into this like the performance counters and things like in the CPU be like hey there's a problem

after that the Chrome OS fix update was ready now at this point they talked about a lot of stuff that we've done and as you might imagine a lot of these decisions and a lot of this work can be a little bit complex and stressful and sometimes you might panic and doing things while you're panicking is a little bit like difficult and it can get a little bit crazy uh and you might not actually make the right decisions when you're in panic mode right so we have something called imag which is short for Incident Management at Google and is inspired by uh by protocols that are followed by Medics and firefighters basically because if you think about it we do

share pretty much a lot of things when it comes to managing incidents it's a set of protocols on how you can manage incidents at Google including things like building a team for that incident creating communication channels and a process on how you can lead the incident end to end so you build a team and that was the team that's me the best team ever right like I can imagine like a lot of you would relate like if you just do everything by yourself would be just easy right just simple you don't actually have to tell anybody else to do anything but in reality that's not actually the way it works so I would be The Incident

Commander following the iMac protocols I remember when I joined Google there was a mandatory training for iMac this is how you do iMac I remember uh so the way it works is that when you create a team with iMac you become kind of like the CEO within that incident and then you would be basically responsible for it it's basically the only time they say you you'd be able to boss your own bus because your boss might be part of the response and it' be their boss so I'd be the inent commander and then of course the first person that comes to mind would be someone called the operations lead cuz I need people to start fixing

things do the operations of stuff of course under operations operations lead is not going to do everything by himself so he's going to have to get somebody to lead Data Center mitigation and everybody that I'm mentioning here is a lead under each lead there is another team and a lot of individuals under that lead so in this case we've had data center response lead and then Cloud response lead and the person that's actually going to work on mitigating Cloud environments and on the other side we've got somebody who's going to be leading mitigation for chromos responses corporate systems response lead and platform security lead these are responsible for the corporate side like internal infrastructure there is the tech research lead and this

person is the lead of the team that actually does the research and the ones that identifi the vulnerability we have a tech lead who also does the engagement with Intel with the vendor communication lead of course because we have to create communication artifacts when we're communicating with other googlers other people at at Google or people external about this we have to do this in a certain way so we need somebody to lead that so internal external and we needed somebody to lead how we're going to be communicating with gcp customers and of course there's legal lead that actually takes care of things around legal approvals of how we're communicating things so this is how the

team looks like in general and then this orange one is a little interesting so this is alphabet companies that are not Google because you also have to fix things there we had a lead that was going to be leading mitigation in that side so we got to this point we've addressed everything we had the team and everything was successful of course the last bit would be to publish this so uh everything is fixed we're fine now we can tell everybody about it because everybody else needs to also fix their stuff so there's disclosure day and it's the day basically we wait for it's been like 3 months being stressed out about this response and we we done it right okay we're

happy so we published it and when we published this we published it in a publication called the security bulletin I'm sure a lot of you know about and then we also published The Tool called ice break so that anybody would be able to just run that tool in their environment and be a and be able to tell if their environment is vulnerable to this or their machine is vulnerable to this vulnerability and then later on somebody sent us this

message if you were the answer then Commander would you be concerned I was just about to go on holiday so we did in investigate this a little bit we started running the Ice Break tool ourselves in our environments and stuff and we went to the people that developed the Ice Break tool to look into this a little bit more and turns out there was a bug in the ice break tool so the tool that will tell you whether you're vulnerable or not had a little bug in it so it was lying so that was good news it was a scare but things were fine what did you learn from this response as you've already seen when we had this vulnerability we

dealt with it like a traditional response just like you've got hacked even though we haven't and that's how we do it at Google when we have an vulnerability of this scale be like well we already have a problem and we call we can follow the same set of protocols and how we can manage an incident in general what did we learn well one of the things that we learned was that following iMac protocols was really really necessary it really did help a lot what can we do better it's always better to automate things so you don't actually have to do things by hand or on the flight during an incident and it's very difficult to do

this we still did a little bit of let's just do this now for the first time kind of stuff during the incident so maybe some of this could have been better if we could get to a point where we don't actually have to do things on the Fly and you already know what you're doing and you just follow a Playbook it's very difficult to get to that point where we can say everything is just on a Playbook but that's one thing we could improve can we automate definitely we have a lot of great automations at Google but there's always a way you can make things a little bit better are we PR are we prepared for the next one I did change

team so it's their problem now but uh no I'm just kidding I mean I I do believe that generally I would say Google is prepared for this now one of the things I worked on uh when I was introducing myself I didn't mention one of the things I've been doing also is developing security tools for security teams so I love building automations and one of the things that have that I built at Google for example was ingesting a lot of intelligence data from different sources and then dynamically adjust bugs that we have internal at Google about vulnerabilities if there is an exploit to it that became public then automatically you basically just raise the alarm uh and basically just uh uh

modify the severity of certain vulnerabilities that we have identified internally uh and work with it in a prioritized fashion and sometimes automatically respond to these vulnerabilities so I did develop that automation one of the things that I thought I could also develop is something that helps us when we respond to large vulnerabilities like CPU vulnerabilities and one of the things that was challenging was doing things that you feel like well why am I doing this by hand and one of them is creating artifacts like uh the security bulletin communication uh documentations preparing a reactive statement just in case of something goes on a leak gets leaked or or or something like that in case something gets leaked uh so I

thought we could automate some of this work also including things like preparing the fixes and and linking that to the automation systems that basically push to fix this a lot of this so I thought we could automate this so I worked on a project where actually did start working on an automation platform that would automate a big portion of this and this would basically create a process what I did was I created a process that is very well streamlined for responding to CPU vulnerability so that when The Incident Commander is leading a response to vulnerability like this you don't have to uh worry about a lot a lot of it or thinking how we can do things you don't

have to think about that just focus on the response and how you can do a lot of the details and how you can follow the process and how you do the response you don't have to worry much about that just do what you're prescribed and it's driven by this Automation and the artifact that are created by this Automation and uh last thing I want to mention if you want to connect with me I'm more than happy to be connected with in LinkedIn and that's my email which I think is pretty easy blackhat google.com and uh I hope you enjoyed this uh presentation and it kind of gives you an Insight of how uh we do

responses to large scale vulnerabilities at Google thank [Applause]

you so any questions

[Music] uh saw one hand up there okay I think you got it was first so give me just a second just a disclaimer I might be able to answer the question directly if not I might ask you to chat over launch hi Jo uh so I'm just curious um if it was considered or not before receiving a new micro code from from from Intel especially for the gcp case which is let's say th risk one um to develop some mitigation on the on the VM level so far the hypervisor

so the problem with this is that it's a bit of a unique kind of vulnerability and this is similar to the question that sometimes I get about how do we detect this can you use like some logs that you can get at the operating system level or something on the network level and the problem is it's very very difficult with these kind of vulnerabilities because it's very very low level at the CPU level you can try to do things at a higher level but chances are they're not going to work very well because it's a very low level vulnerability so it's one of those things that I probably just say it has to be fixed with a micro code update that You'

get from the vendor most of the times or disable a feature like uh the chicken bit bit that I just mentioned uh yeah even for detection by the way like one of the challenges we had was how do we detect this if somebody's exploiting this right right now one of the ways that we thought about was performance counters of the CPU now if you want to do this at scale like Google it's it's it's difficult but you could come up with things like this but but it's it's it's it's really challenging and then the other thing is let's say that you're sshing into a machine across the internet and you're running these instructions on a network

level how are you going to detect something like this so you see this these are some of the things that we get which is kind of similar to what you just mentioned about mitigations it's a very lowle kind of problem it's difficult to do anything about at a higher level than that I hope that answers it anyone

else hey thanks for the for great talk I I have a question in your playbook like you spoke about chicken bits and how activating chicken bit would Grant a performance hit do you have like a an escape valve because for example the Spectre and meltdown and other mitigations have a really big impact on CPU performance is there an escape valve in the sense that rather than push for a micro code update where the the fix is going to impact performance even more that you just go oh we're just going to go disable this feature and take the performance hit yeah so are you asking whether we checked how the performance hit looks like yeah if in your playbook you have a

a path where the incident lead can just go oh we're going to skip a micro code update because the impact is just going to be too big across the board yeah that's kind of what we did because we did test it we test the chicken bit it wasn't just a theory we tested it and the performance hit was about 10% and we were like well that's too big but let's just keep it in case we don't have any other solution we might get to a position where we be like you know what this is leaked or something and we didn't need to use that position uh use that solution but for now we're fine let's just wait for the

vendor all right awesome any more questions just there one in the front just last one we need to then move forward sorry okay thank you uh great talk uh I I'm kind of C curious about something that's not uh commonly talked about as at least I don't hear uh it talked about a lot um and uh if I probably understand why so if you can't answer this it's okay uh but for example you mentioned uh that during the for example Discovery and the treatment of the vulnerability you had an embargo period during which you could not talk about anything and there was also for example the risk of somebody leaking um for example uh did

you ever uh identify or at least suspect there could be a risk of for example of an Insider threat uh leaking details about this vulnerability and the exploit yeah that's a very good question I didn't talk a lot about or maybe at all about uh I don't think I talked a lot about the logistics that go internally when how we actually manage this now Google has a very open culture in general but when we're talking about things like this we have to be careful right so when I was The Incident Commander for this response there are certain protocols we have to follow and whenever we we only follow a need to know set of protocols

if you don't need to know this because you don't have a significant role in this response you shouldn't know about this at all what we do is we use code names we have a code name for this response and we refer to it by that code name and that code name wouldn't wouldn't help you infer anything about the vulnerability and then we would only invite the small set of people that are going to be involved in this mitigation and sometimes some other people would be involved but they don't know about the vulnerability because they are only asked to do something like for example if we give you a binary and we say push it to that system you don't need to know

what that is just push it and then you push that system so yes so we have to follow this set of protocol ensuring that only small group of people know about the details and only the ones that really need to know about it how we establish that there are a lot of mechanisms as you know like for example uh typically what we would do would do things like creating like a an alias or a group that only contains these people so communication would only use that and certain meetings that are only going to be involved in these these people communication channels are going to be unique for this response a different Communication channel only for

this response and we're careful about who gets added to that so these are some of the things we do if that makes sense all right thank you hey guys you can catch youf outside if you want further questions we moving on to the next session thank you very much J thank you everyone [Applause]