BG - JIT Happens: How Instacart Uses AI to Keep Doors Open and Risks Closed

good morning everybody I think it's still morning um want to introduce myself my name is Dominic Zanardi um I'm on the sharing the stage with uh my friend and colleague Matthew Sullivan who uh I join on the infrast infrastructure security team at instacart um so I want to thank um want to thank the audience for spending some time with us but I also want to thank uh bides for uh giving us the opportunity to sell you a yacht not an 80ft cat of Moran like we'd all love uh but this is yet another AI talk um we're here to um most most at most security conferences uh AI is viewed in a negative light we want to provide a

positive light we're not saying llms are a magic wand but they're used to greatly enhance uh security tooling uh this isn't a paradigm shift but it gives small teams the breathing room we've we need uh because the cavalary isn't coming but the controls are we have a lot to do security team workloads are growing um Am I Wrong is anyone bored at work today um we need to be faster and more agile every day we hear do more with less there are always more controls who's going to write all this

junk Ron he knows what's up so we have a couple options uh with all of these incoming requirements uh we can level up be the best team we can be be the best in the world or we can turn to our robot overlords um and reach out to Skynet because we also believe in work life balance so automating the gray area um when I think about security automations uh I think about binary decisions most of the time when we're setting up automations we deal with true false we deal with static data sets llms can help us in the gray area where context is everything um where humans might have to spend hours pouring over logs pouring

over giant data sets uh just to make decisions let's say your audit team asks you we need users to only request appropriate access every single time before AI this might be a look you give your auditor I don't know but with AI let's let's vacuum up the previous audit logs let's vacuum up the history that's been provided um and recommend the appropriate role to a user based on what they've actually used another one we need you to review and update your Ro description when there's a change in privilege audit will ask us to regularly update our role descriptions whenever something changes before AI go through every I am role look at the policy statements update the descriptions for them put

them in a spreadsheet provide that as evidence with AI let's instead take those steps pull the policy statements to a role send those to an llm and have the llm write the description for you it's not difficult it's just tedious why not have ai do it so uh before we demo this uh just quick crash course in impr prompting an llm uh they have two query components uh the template or system prompt is the first one uh we want to make sure we provide accurate instructions to the llm uh so that it knows how to operate the user prompt following it typically provides the content uh that it needs to follow those instructions um quick reminder um it's a

good uh thing to note that when you're co-mingling instructions and data you want to make sure that um you lock it down if the data you feed to the llm is provided by the user so roll descriptions from AWS apis um let's look at how we'd use a tool for an AI driven script uh to generate Ro descriptions from AWS apis um you're going to see a side-by-side display of data on the left are the prompts that we sent the llm on the right is the response from the llm so let's generate a roll description we'll switch that perfect so we're going to kick off this python script as I mentioned we're using a library that splits data down on the

left is what we are sending the llm I am an internal auditor provide your persona right away this helps build the output you get back if you want to appease Auditors say you were an auditor um we give more information we're basically saying please provide a three sentence summary of what can be done with this role so we can use it standby for list of actions in this case this is a live pull from a demo AWS environment we're literally pulling the um I am policy statements from the role that we've sent and with the assistant prompt and the user prompt together we're going to get a response from the llm based on those policy statements

saying this role allows users to manage and interact with the S3 sqs you want it to be um you want it to be descriptive enough to approve be approved for uh Auditors on the outside from an Evidence perspective you want requesters to understand what they're asking for without showing them a policy statement um and you want the approvers to know what they're authorizing So This is highly visible in three spaces um at instacart we use a platform for Access requests known as conductor one um that platform is completely based or at least in our configuration we base all of our config in terraform and identity as code so this piece can easily plug into our terraform files so

the next step here that one is to take this information and literally pass it into a poll request so not only are we generating it we're now sending it to GitHub said my windows disappeared you're good yeah perfect so as we open this up as I mentioned all of our configuration for conductor one the access request platform is based on uh terraform files this Ro description us to be an example rule that does a little this a little that we now have upto-date information on exactly what this is you can change the configuration to be as short or as long as you want um but right here we can review the changes approve it and move on all

audiences uh are happy Auditors requesters and the authorizers so so's here to walk us through how instacart's uh using these Concepts in production to solve more real world challenges solving the pains thank you thank you thank you I'll have you hold my speaker NS I'll trade so that's a really simple example right we're pulling in data we're throwing into an llm it gives us a description you can probably start thinking about some really interesting use cases you might have though are there times in your business when there is some complicated alert I know the cloud um tooling that we utilize the alerting is typically very hard to understand and if that triggers some sort of event that comes to your phone

what's helpful the Json blob or hey it looks like a new IM IM user is doing some weird things in your AWS account right even using these simple Concepts it can actually really help be a quality of life change and so these are the types of things that we're working on at instacart is even taking our alerting running it in through for us we use tin a 's workflow that just makes it easier to digest the other thing too is then my en call rotation just doesn't have to be so scary to a new hire you know I remember when I started on call it was a lot to take in and if it's more simple

English then it allows somebody to feel a little bit more relief as as these pieces of data come in and out so again it's really important to us that you look at these things not as some scary new technologies that are going to ruin our lives but like well what if we did embrace them and what if we actually made it so that these were a quality of life life Improvement for us so we didn't feel so burned out at this point I'm going to talk about how we solved this real world problem we wanted to go public uh so I started instacart I accepted my offer on the day we filed our S1 which is our intent to go public

with the SEC uh what I did not realize is that basically from that exact moment until we went public in September of 2023 that is all I would do was just that work uh and in case you're wondering no they didn't tell me that is what I would be working on but that's okay I really enjoyed it um so a significant amount of time was really spent trying to figure out how to build an access control program that could meet our objectives and we could be proud of you see when a company goes public in the United States your stock offering becomes subject to sbin Oxley of 2020 or 2002 uh and what we could do is bore you to death or we

could not do that and simply say that basically this means the you don't cook the books and I as instacart shareholder have come to the conclusion that's generally a good idea not cooking books so socks dictates that we will have the right people uh that have access to financial data only authorized persons this makes sense we don't want people one y we don't want people seeing materially impactful financial information or compromising the Integrity of that financial information makees sense you see there was this company called Enron Enron lied about their financials investors and even some of their own employees lost everything jail time you get the picture the entirety of socks can be distilled down

to saying boy that sucked let's not do that anymore sorry so socks takeaway access matters it's the only thing I really want you to take away about socks doing socks means that you generally care about access all right so we've got my mandate and my deadline IPO is coming uh what can I do to make this Access program great because I have this thing I don't know if you guys are like this where I don't like working and I would rather do literally anything else all the time but because I have to work in order to survive then I do it too hard right and so if I'm going to go in and make an

access program it's going to be awesome right uh so one of the things that we settled on early is okay we'll use Justin Time access jit um and basically in jit you grant temporary permissions as Dom mentioned a second ago we did a evaluation ended up purchasing a commercial solution in this space uh who we felt was aligned with our values of kind of like disrupting the status quo a little bit um so we fast forward and uh now we've got our roll out of that done we've got one roll onto this just in time flow it's kind of like an administrative role our infrastructure team uses um but we hit a snag though as

we quickly realize that manager approval sucks that's really hard you see the problem is your manager is busy they lack context they lack expertise sorry managers you do some managers just turned 50 and are out of the office trying to find themselves by backpacking across Europe and most people would just buy a sports car and get a dog but what do I know so we need to do some real talk about how we're going to fix this spice level 100 here we go let's talk about doing everything everywhere all at once with access requests only instead of that is everywhere all at once is on fire everything is on fire and everything is terrible in the status quo in our

industry and I think it's time we acknowledge it security and audit continue to double down on the existing way of doing access requests whether that's meeting socks or if you're a Fed Ram shop whatever it is you're doing the same thing we've been doing for the last 15 years possibly maybe you're maybe you've improved and that's great the problem is the model is fundamentally broken we're made to do but anyway though we have to do this thing we submit the ticket and we wait a week for a manager approval and we wait a week for the owner to actually action on the request and then you defend your access until they pry it from your cold dead

hands because it took so darn long to get it in the first place you will never let it lapse somebody walks up do you still use this oh every day yeah I keep prod alive uh-huh mm being able to hit that S3 bucket it's keeping prod alive I gotcha right the fact of the matter is your access requests look like this graph when it comes to manager approval denied 0.1% approved 99.9 and it's because they click the wrong button tell me I'm wrong our industry has a bad case of the not my problems when it comes to access approval with ask ask your manager for approval and ask system owner for approval we've solved access

requirements as a checkbox compliance exercise and nothing more we haven't added security so I thought to myself since I'm spicy let's not do that how are we going to fix it and I figured it out we simply had to kill all the humans we could do that though by pre-approving access our identity governance tool allows us to provision temporary access right so what we did is we set a maximum time of 90 days or less for really important things and we allow users to renew that immediately as long as their user is part of a pre-approved set of attributes that we keep in the terraform that we were just talking about so Dom actually spearheaded

building a uh a terraform provider and we want to share that with the world it is specific to conductor one the tool that we use at instacart um but this actually is uh a ter provider that kind of puts conductor one into the instacart way and so we've built a kind of an abstraction layer it's opinionated um but we think it works well and most importantly maybe it can help somebody else and if so fantastic so have a look at that if uh you happen to be a conductor one shop or you just want to see how we're thinking about this problem uh for whatever tool you might be using so we've done it everybody we

solved access hooray okay it's not actually that easy we solved a good portion of it but the problem is that some things aren't cut and dry this is not cut and dry we have developer power user developer power user can basically write to uh high impact S3 resources and it's not clear how we would build rules for that so we had to set it up for manager approval uh you see the condition for needing this role as far as we could tell in our role engineering was have you worked here a long time are you important do you get involved when things break that's a hard set of criteria to add to your workday profile how are we going to solve that

we need another robot so we need to build a robot we wanted to develop a process where we could trust uh the automation of just in time approvals instantly in Risk appropriate situations we wanted to leverage an llm to do what it does best to take in a huge amount of data and help us sift through it I must be clear I know you're already thinking it no we do not ask an llm should we approve this don't do that that will go poorly for you what we can ask the llm is look at these people who already have access to this entitlement look at the person who's requesting access are there similarities between those two things and llm is really good

at that taking a huge amount of data and sifting through that I can do that as a human but for an entitlement that might have a 100 200 people in it that's going to take me a long time with an llm it takes seconds we also are optimizing the enduser experience we can get rid of a bunch of significant complexity from their side of things it just looks like they put in a request and it gets approved and they get access they don't realize that this entire crazy thing has happened behind the scenes to make that occur so let's talk about automated access approvals we built Gadget and in case you're wondering yes we open

sourced it we'll talk about that in a second um Gadget is our answer to this problem and most importantly it's our answer in a generic way um what we didn't want to do was solve this just for ourselves and then just kind of vanish Into The Ether so we've built a pluggable interface for interfacing identity information llms and IG tools um we're pretty excited about it and we would like to show you a demo now about an hour ago I submitted an access request uh and everything that you're about to see is actually real in production which means that it will fail and the demo will suck when that happens I have a recording um so my access

request is for the AWS role that I use for my daily job uh I'm going to go ahead and run this utility it's going to take a look at that access request and what you're going to see is that left and right Viewpoint again of interfacing with the llm so we're going to see how we prompt the llm uh with this gadget utility because it's going fairly quickly I'll just scroll up system prompt here is that we have a list of employee IDs uh and we're creating examples for the llm so this is a very verbose example I'm sorry it takes a lot to prompt an llm to do these things but we basically say a new applicant would

want to join your group and for example if they were an analyst in online grocery which of these two job titles is Rel relevant and it would answer well the staff engineer and online grocery is the most relevant versus a compliance auditor if there's no overlapping information just return nothing we take that system prompt and send it and now we send our user prompt so this is my real data I am a staff security engineer whoa boy I'm a staff security engineer on the INE hyen security team take a look at all of these titles of people who also already have this role and compare it and we get a fantastic match right away that match happens to be

Matthew laurore senior or staff engineer on my team so we're doing good we've got an exact match great next thing we're going to do is Rerun that same logic and we're going to ask about organizational units many companies have different organizational units that do different things let's take a look at that relationship between the OU I'm a part of and the role that we're requesting same thing we prompt the llm and then we start feeding it real world data out of that is going to come a lot of really strong matches effec L this is going to say hey there are a lot of people in that role that match your organizational unit very well what are

we doing with this data each time I get an answer uh literally in the codebase it's going into uh it's a score a numerical score that's going into an array and just being stored we're going to run a computation function in a little bit so we're not making decisions yet we're just getting some results uh in the real world this is actually something that we fire off all three questions at the same time in a multi-threaded pool and then get all the answers back at once and check for the result the final thing we're going to do is to actually take a look at my uh entitlements name and description a second ago Dom just told you that we can

do this crazy thing where we look at all the entitlements access we can generate a description a human reviews that make sure that it seems correct we load it into our terraform we pull it right back out in this tool and then compare my job function to the description on the entitlement the AI will take a look and see that I in security am requesting the security role and it's a role used by the security team what's the relationship score 1.5 this is the one place where you could be like what if the AI hallucinates because I've asked it to generate me a score so we have to be careful with this data this could be bad data I will admit so

how did we check that first we ran this through 3 months of data back and replayed it all from real access requests and we manually looked at every single score and I said does that look logical and it was within the realm of being good enough in 100% of cases and it was perfectly accurate in about 95% of cases so we just determined that was plenty good enough for our cases the other thing I need to remind you of is our goal is to just not approve everything if we've done that we're already better than the status quo right so we can have a little tolerance for you know failure here all right uh so

now we've taken those three scores we add them together I get a 2.0 anything above one means this is probably a good match Gadget has a configuration option which allows you to say which entitlements uh allow automatic approval and addition to those roles if you're going to do the manager thing because maybe you have a high impact role and you don't want to do auto approvals Gadget still comments and that can really help your managers so of course you have that rubber stamp problem right but we add a comment that says we really recommend you don't do this take a close look understand what you're being requested understand what you're approving understand whether or not the person who's requested it really

needs this as part of their job function and we found an incredible amount of success with that so even if we don't do auto approval just the guidance piece we've seen immediate value at instacart which has been pretty awesome head back where is my display H I've lost my slide deck I'll just reopen it all right again one of the things that was really important to us as we started this journey is I hate when companies take more than they give from the community and this was something that I thought we could really try to change the way we do things in our industry and so we're really excited to be able to open source this I beg you we would love

to hear from you I want to see PRS I want to see you add more tools um and and this is a very uh like I said very pluggable interface if you want to run a different llm we use open AI you want to use anthropic that's supported write a plugin for it if you want to use your own IGA tool that's a plugin supported if you want to write your own scoring mechanism and not rely on ours fantastic you can write it at as a plugin this is also able to support running as a Damon or as a Lambda or as a web server so if you have a web hooking capability in your existing IG tool it supports that

out of the box as well at this point we're going to go ahead and stop and ask what questions the audience has we really appreciate your time thank you so much for [Applause] coming and I'm expecting good questions because this is kind of a spicy topic y microphone's in the middle seems like the mic is

off now the mic is on perect that's good that's a great use for llm thanks for the talk and the and the tool especially open source Tool uh I'm imagining you didn't start with zero access you started with a company that has a bunch of stuff that you can use to go tell whether the current request matches past requests um have you thought about how to do this in like a bootstrap situation right like we're going into a new cloud or first time we've interfaced with this tool or those sort of situations you know I'll be honest uh I did not think about how you would start from zero to one just because I was starting at like

99 and going to 100 but uh I think that there is a really interesting opportunity to do some incredible things with let's say you're really doing zero to one no compan is actually at zero right so we can come back to what Dom was talking about if you can write some basic things that say take in some Cloud tril log see what people actually doing and then maybe that can help you with your role engineering that's great I mean there are companies out there trust me I know we're customers of them that want to charge you $200,000 to tell you what your people are using and then provide you a good role template to use

we do that in 40 lines of python right so there's an opportunity there to just be Scrappy and um and I think for me that returns me to my roots I don't know about everybody in this room but like I started as the one security engineer at a startup and it was horrible and I had a ton of fun and did it for 10 years right and like I remember those days we didn't have budget for products you just build some shell scripts and prey and it was enjoyable to kind of get to look at this situation kind of like that be Scrappy be efficient and fast and how much progress can we make with as little

time as possible and I think we achieved that goal anything you want to add yeah I would just say just to Echo that when you have cloud trail logs feed them in and tell it hey generate me a new role we've done that out of in a couple of instances so it works out yeah can't stress up human in the loop is still needed I mean you need to be sanity checking these things it does not always spit out valid terraform yeah so human in the loop but it's it's again it's a tool yes man I've never seen before from instacart um hi uh what do you do if you have Auditors who even though they're

not supposed to ask for absolute Assurance they are asking for absolute assurance and they're going to ask you and I know they're going to ask you this because I work with you uh how do you know for sure that it's not hallucinating like what additional checks can you add Beyond reverting back to a horrible manual user access review well assume that you're going to do a garbage like you know normally you'd say garbage in garbage out we're going to start garbage out and then check garbage in right so take everything all those IDs that you just saw spit out back to us double check those go back to your same list of users make sure that the

data that you fed in stayed the same the IDS are still the same the job titles you fed in came back out an llm isn't going to start swapping those it's going to hallucinate them and so you have good signal right there to know if what it's generating is real or not real um and then the other thing too is you have to ask just super basic questions you saw the structure of the Json that we request back it is dead simple simple structure simple key names we're not doing like nested Json you know responses you can't the llm will freak simple questions it's like talking to a four-year-old I would know I have one

so thats

good thanks again appreciate your time today [Applause]