← All talks

Non Human Identity Attack Surface: A Live Hacking Demo And Defense Strategies by Michael Silva

BSides Toronto32:51453 viewsPublished 2024-10Watch on YouTube ↗
About this talk
Presented on Oct 19 2024 at BSides Toronto 2024 The proliferation of non-human identities (NHIs) – APIs, service accounts, tokens, and keys – has introduced a vast and often underestimated attack surface. This talk unveils the hidden dangers of NHIs and equips defenders with the knowledge to combat them.
Show transcript [en]

awesome hey everyone how are you uh came from the US that hurt a little bit when we came in second um won't lie the uh only thing keep me a float is the Yankees are winning in the world series or getting there so we'll see uh yeah so hey I'm trying to go to the Lea game tonight okay I'm trying to be local right so Caines um so what we were talking about here right so non-human identity right this this is a uh a space m first let's I guess let's talk about who I am right uh I am from the US uh Marine veteran us uh I've done a lot of different practices I've done data

Fabrics networking security um kind of everywhere I don't like to be one thing uh a lot of it's self-taught uh so the events uh the CFT uh you know uh events the you know networking discords like get engaged that's how you're going to learn um you know the you know the other part behind it you know that's more relevant to this is you know building out socks and playbooks and run books for uh Department of Defense um operations centers for Cisco um and then more talking about you know kind of the attack side so I've been on the attack side and defense side um you know you'll see throughout this talk you know one of

my models I I say whenever I go somewhere is if you don't know how to attack something you don't know how to defend it uh so that is a big piece of why I try to teach the attack side along with the defense uh to go there so uh first we're going to talk about what non-human identities are it's a big buzzword term uh that came out about 18 months ago two years ago um little disclaimer I do come from a company that does non-human identity security uh we're a startup that's been around for about three years uh we are Israeli based we are um you know penetrating the market in the US we're also here in you

know Europe and also in Canada uh and other places but we're going to talk about um you know what are non-human identities you know get rid of the the marketing term and kind of talk about what the actual problem is uh we're also going to talk about you know what is the attack surface and why should you care uh then we're going to walk through I have a recorded attack uh you know it's easier than you know with this you know copy and pasting or typing um and then go through some defense techniques besides by my product right that's not what I want to tell you I want you guys to be able to do this on your own um if

it leads you to buying product great right but that's not why we're here today uh so what are non-human identities I always think Webster is pretentious for having his own dictionary so I like to make my own uh and I like to call it uh you know what is a non-human identity it's programmatic access to processes and data where a human does not have to be involved right or required to be involved so you think you know what does that actually mean right you we're talking about API Keys service accounts uh some of the Lesser known right you know the the extensions application extensions web hooks um you know ooth apps machine identities right um you

know these are the items we're talking about uh these are items that kind of fall through the cracks uh because they're in different verticals they touch different verticals they go across Cloud cross SAS across different environments so they kind of fall into those gaps um so how do we get here right so there's a couple ways that nhis make it into our system uh one you know everyone's seeing the one the left everyone's probably done it three times in the past month right you know allowing access when you start a new job onboarding a new app giving it access right that creates a token uh that's going to go from there you know we have

other humans creating nhis right which is creating an API key manually creating a service account in ad uh going from that point of view and then you where we started seeing the boom and we'll talk about that in a second is the nhi creating nhi programmatic access of creating new programmatic access right so you can see how this kind kind of has an exponent Factor uh some of the early research uh is you know we're safe to say that for every one employee there's typically about 20 uh in a company right so you think about 1,000 employees 20,000 nhis uh this does balloon up to up into the 70s to one uh depending if there's heavy development

heavy pieces um so how do we get here right you know what it what caused this to actually happen um you we used to start with Legacy on premise we we were ad on Prem we had our IDP we had something that was our source of Truth and that was the only way you could access our systems uh from there we moved to the cloud era right everything was Role based we were giving machines identities to Due items we were also being very granular giving them multiple ID you know identity to do items uh then became the the magical digital transformation period where everything went to SAS pass you know anything as a service uh that created a lot of

external connections into your system Market Marketplace uh you know making sure all these apps can talk to each other uh and we're only seeing this balloon even further up with you know the amazing amount of gen developers that are out there now that are creating all these different cool apps that we want to put into the piece right so it's it's creating a this balloon effect so there's also besides technology business processes that are leading to why this is happening uh you know there's mergers and Acquisitions right you're bringing in junk uh that you don't know about you don't have the backstory on you know why the surface account exists we don't want to play the screen test game and try to

rip it out and see what it breaks uh offboarding gaps right you know the traditionally humans are not tied to the non-human identities we don't know who actually created the service account unless it's a record we don't know who last update who still needs it API Keys uh you know SSH you know Keys all these other pieces uh you know we're trying to create all these connections between different systems right it's creating more noise creating more connections creating more lateral movement process uh and then obviously you automating of processes right so having the ability to actually not sit there and do that right so what is the attack surface right there's really two or or four uh really

two categories external internal right external is the ones that you always hear about right you know someone you know got a token leaked someone got a vendor breach someone got uh something of that nature right that's the external the internal is more you know Insider threat uh using a service account to go elsewhere uh lateral movement uh priv escalation right so what is an external direct attack look like something as simple as this right we have broken web hooks with keys in them and passwords in them we have uh a leak token uh we'll see that here a little bit later in the attack we also can get stuff back through API calls and headers right

there's secrets and and other items we can get back uh really why business is starting to care is because the attacks are starting to be used right uh and this is where we start seeing all the news articles I call this unfortunate free marketing for myself right because it happens and it's bad uh but it brings attention to the area right we're seeing about at least one major one a month uh between GitHub circleci hugging face AWS right and these are all environments that touch many customers right and we'll see how that makes an impact here uh in a little bit but uh the you know the cloud flare one actually came from the OCTA breach right OCTA got breached

which turned into a cloud flare breach right uh and then Cloud flare all of their stuff got uh in a little bit of trouble right uh and what actually happened uh to kind of break down with this game from it was a someone in the support center for OCTA uh had a human based attack ranw attack they had a service account that had access to the har files uh which are session uh session tokens for their customers uh that was able to be accessed and they were accessing customer systems they got to Cloud flare uh Cloud flare was notified by OCTA to rotate all tokens they had 5,000 connections from OCTA into Cloud flare missed four those four

got compromised right so having the ability to see what you know how small it can be of a mistake to make a huge problem uh kind of goes from there so let's see what an attack actually looks like right uh so this is the attack you're about to see it's about 8 minutes I do it at double time because it's usually an hour talk uh but this is uh you know what we're actually going to try to do right the real three pieces I want you to keep in mind is while we're going to gain access uh in a way that's pretty easy uh we're going to do lateral movements so we're going to try to see

where else we can go uh and then we're going to steal conceal and persist right we want to get some sort of financial gain to get some sort of physical gain from this attack we don't want to just do it just to do it uh maybe we do but not today uh we also want to conceal right we don't want to just give away the breadcrumbs of what we did and we want to persist we'd like to come back right we don't want to be a one-time visitor we want to be able to come back into this environment maybe a different way right so before we you know walk into the attack there's a couple

things to know one in this scenario we're targeting a company called Square CD not a real company uh and we're targeting them from just looking at all their developers all their developers

githubschool a key so what right if I drop a physical key on the ground if I pick it up I'm going to know either goes to a mailbox a car it goes to a house right I may not know what car uh I may get some sort of indicator based on an emblem on on the key right but I can still tell there's somewhat of a physical difference between the keys same thing for The Logical keys right they've fixed prefixes they've fixed key lengths uh there's a lot of characteristics that people who find these keys can easily say I know where this goes right uh that's why we have secret scanning uh type tools because

that's what they're looking for right so I'll talk through this uh as we go right so what we're going to do here is we're going to first clone the repo uh that we have uh for this public uh for this public repo for this user at uh Square city is developer uh we're going to go ahead and CD into it and we're going to run a open source tool called G leaks uh what this is going to find is two Secrets one's a web hook one is an adbs access key right so we can see the file where that key is is a shell file so we're going to go ahead and try to cat it out and we're actually going to

see that we don't find it right so something's puzzling here right what we're then going to see is that we're going to look at the branches and actually see that the leak is an unmerged branch which happens a lot orphan branches unmerged branches and if you have a SAS Das tool that typically only looks at the main branch right there's a lot of data out there we don't care about the rest so what we're going to do is we're actually going to look at the commit and we're going to find the access key and the secret key right so now we've from a public GitHub repo that they probably download a project to do something uh you know themselves but

they didn't wipe out the unmerge branch we're going to create a new AWS profile with this key to see if it works right so what we're going to do is you know get caller identity this is going to be the who am I of AWS to understand that yes we're a user and we're Square CD cicd service account right great we have something so we're going to stay on the non-human side and we're going to start listing secrets and key man so we're going to list secrets to understand are there any secrets I have access to in this case we have two right we have cicd slackbot we also have a personal access token in prod uh we're

going to use the slackbot because it's also named cicd there's a good chance we have access to it and we do right we have two keys inside here two values one as we talked about the prefixes we can tell one is a uh xox is slack B is a bot p is a user right so I can see that I'm going to put in Postman uh these two values because I'm going to start using the slack public API endpoint because conveniently they gave us enough public API endpoints to figure out where it goes right so now we're going to use uh this public API endpoint to say you know like who am I right the who am I of the

slack right so we're going to do first do the bot token because it typically has less privilege uh we're going to see that it is valid right we have a slack workspace that we can see but the more interesting Parts can to be in the headers in the headers I can see what the actual scopes of what this key can do against that workspace right so this is all public information right now I can see okay let's go ahead and look at the user they have more Scopes one of the most dangerous ones we've see in this is channels history which is actually a very misleading one which actually means I can read all history of

public and private DMS right so that's a more of a user token we're going to look at which channels are in here we see a couple called Dev we call cicd right so okay makes sense so now we're going to do a search against the history of all these channels looking for a prefix of an AWS access key we've already known that we did it before so chances are we may find it again and we get lucky right who's seen this message about 100 times if they've done right please delete this key after you're done copying it to your password manager right we see this all the time in customer environments so we're going to go ahead and create a new

profile using this new ads access key I've already copied it in to save us a little bit of time here and we're going to see that you know we're going to do the same thing caller identity is this even a valid key right it's great that we found it it is right and now it's their bad day now I have an admin key right it could be an admin could be maintenance could be something I've now escalated my privilege to something else now I can see the permissions I have secrets read right I can make my own I have S3 I have RDS right I have a lot of cool things inside there so I'm going to list those Secrets

again see if there's any more that I have access to uh and I can see that you know we're going to try that other secret which is that prod personal access token personal access token is API key for GitHub for anyone who didn't know uh and I'm going to see that they have that key there again I don't need them to label it for me I can see with GHP that it's going to be a personal access token for GitHub I'm going to go ahead and do the same thing I'm going to list all repos uh that this private that this key has access to and I'm going to see one right here called Square CD app

prod which is a private repo inside their organization that they now have access to I'm going to put that repo inside another environment variable and I'm going to read the contents of this to go ahead and see if I can actually you know describe what's going on inside here and I'm looking for a few things right I can see the read me file that's fun right I can tell what it does uh but I'm going to start hitting some gold here in a second scripts you know maybe there's another password here we go I have source code right this is what I came for and this is what I want I want the source code and I'm going to take it

right so what I'm going to do from here is I'm going actually clone this repo right so I'm going to take this repo but I also don't want them to know what I did right so I'm going to start poking around an S3 right everyone's favorite Cloud protection right everyone's going after my S3 bucket so let's pretend like we did right so we're going to conceal our actions by going after a data breach right so now we're going to probably set up a whole bunch of bells and whistles uh so we're going to go ahead as fast as we can we're going to actually see that we can see three buckets uh one's called customer connector keys right customer

connector Keys has a whole bunch of Json files in it right customer connector Keys also has one called important customer right so we're going to see okay let's copy down this key and see what this key actually is or see what this Json file actually is is it just a document is an actual key what do we have so we downloaded it we can see that we have important customer right there we're going to cat it out and we're going to actually see it's an access key to Google Cloud right so this is now another environment that we're getting access to the most important part is that it's not to their environment we've now moved

from being against Square CD to Turning Square CD into a supply chain attack now we're going after the customers so what we're doing now is we're going to list all the projects we have access to we're also going to see you know what can we actually do we see we have an editor role right that editor role is all we need to start creating our own fun stuff inside this account so we're going to create our own secret right we're going to create our own secret which is going to is it done okay it's done I don't know why it's done but uh what we would do next is we're going to actually create wait did I missed I was

talking NOP okay so what we did next we actually create a service account with that key to actually create a back door in a customer environment that they have no idea we were in in the first place right so we' actually gone from doing this right if we you know recap this in a nutshell right we started with a an Oculus you know API that we found out in the public repo that someone had not merged to their main maybe they were cleaning it up before they used it they did something cool at work they want to do it for their public side but they never cleaned it up before they actually uh got around to it we found it right

we're going to go ahead and jump between all these different SASS and Cloud environments right and then we're going to go ahead steal that source code which was what we came there for that was our money maker right because now we can use that for many different things and then we you know by concealing our crime we stumbled upon other items and Es buckets that gave us the ability to actually create uh you know a supply chain attack where we got into TR you know where we got into other customer environments and turn them into from a Target to a supply chain attack and now we're targeting their customers now they have to send out that terrible email from legal that

says rotate all keys rotate all tokens but it doesn't matter if they do we've already created more access in those environments anyway I don't care what they do from that point of view now I'm going to keep going down that chain right so how prevalent this issue right so know question all you get is well who's targeting me right they don't have to be right and this is a this is a cool piece of information that uh a company called cybernary uh did some research on this they're a consulting firm uh what they did was they put Canary tokens which are just tokens that are real tokens but they don't go anywhere think of Honey

Poots right that are out there in the wild and all they're trying to see is do people find them right if people find them what happens to them what do they try to do notify me all these pieces right so what's actually happening here is that when he put the keys inside of all these different types of environments the infographic on the right is how fast they were robotically picked up so this is robotic pickups this is not someone targeting them they just threw it out into these different environments and saw how fast do I get a hit that someone's trying to use this key because it's now publicly leaked right and this is how fast they were

it's a really interesting article I at the end I have a QR code that actually has this article in there uh it's a really interesting read right because you can actually see that where he thought he was going to be breached or where he thought he was going to be attacked was not where he was attacked and that's one of the other parts when I talk to just businesses and customers is no one's sitting in your your business meetings no one knows what your crown jewels are right they're just finding trying to find a way in they will find your crown jewels later but they don't they don't know what you're trying to protect heavily they're just going to

and see what they can find right uh the other part is you'll see the next action taken right they tried to get caller identity figure out who they are they try to invoke models using AI models against your environment right it's the scary one right they're trying to list Secrets list vaults right they're not trying to get any R they're trying to keep doing the same method they're trying to continue the non-human side because it's persistent access I don't need Biometrics I don't need MFA token I don't need all these other pieces right I'm just going to keep going down this path uh so how do we protect it right uh there's some basic ones that even if you

put it into GPT or or or Gemini whatever they give you these ones right but these are kind of Basics that we need to go back to right which is proper active inventory right having the ability to see at all times what is my inventory across Cloud SAS on Prem which ones talk to each other uh which ones still need to be active right um you know tying those back to who is the human owner ship business ownership Financial ownership technical ownership right um you know proper offboarding processes right um you know if if this gentleman in the front row if you left my company right I need to know that when I remove you from the company you don't have SSH

Keys you don't have tokens you don't have anything on your laptop that you could then come back and get to my system right so understanding you know what are those proper offboarding processes uh expiry notices right you know for tokens keys right nothing needs a forever lifetime even if you think the project can be around for 5 years set it for 180 days you know every year whatever it may be make a process where it's reviewed right make a process where it's actually looked at more than once you know if you know if you ever see not never expire on a token that's a problem right that that should never exist there's no persistent access forever uh

that should exist uh lease privilege right you'll you'll get from you know I came from the dab side to the security side right I call it going to the dark side right so uh what you can also see inside there is there's conflicting right you know Dev wants to say hey just give me access so I can make something really cool right Security's like well hold on you're creating a gap right uh you know continue to say hold on you're creating a gap don't worry about holding up business right do not create that Gap do not let them say hey I I'm going to you know just give me you know high level privileges contributor access once

I'm running it I'll come back to you and tell you what I need they're never coming back I can tell you they're never coming back they're not losing that access right um so don't buy that story uh setting rotation right uh another part educating end users Right End users are creating this access as well when they allow connections you know they come from HR they come from sales they come from all these other places uh very similar to how we started doing fishing uh you know attempts right we can use nhi for fishing it's another one I do where I use slack message and teens messages to send messages and get them to do stuff right uh you know but

understanding that these act you know these cool new widgets cool new apps that don't have to make them work as hard right they have a level of risk that come with them educate them right the ones we've seen companies that are educated that they respond the same way that those who are educating their and users on fishing respond right no one's clicking on that package label that's waiting downstairs for them no one's doing that well hopefully no one's doing that anymore but they they know now right they know not to click same thing with all these different extensions uh you know a little bit deeper um if we wanted to say like how do we go a little bit above just Basics

right how do we start looking at these non-human from a active threat monitoring uh perspective one of the toughest Parts about uh nhis is the fact that they come with a level of nuisance from the money side right and when I say that is uh anyone who's run a Sim or run a project for a Sim knows that collecting dog collecting logs cost a lot of money right inest costs a lot of money so what do we do we do risk reward analysis right we say hey I don't care that octa's authenticated 5,000 times correctly today I'm good I don't need this log anymore right it only takes that one time for it to go bad right so

again not telling you to turn it on right that's that's a cost analysis piece um that's actually where we come into play but the the other part is that you know setting up the ones that you if you do throw out Canary tokens you throw out some stuff like that right see what people are doing at your environment you know set up monitoring alerts set up you know different events that say Hey how do I go ahead and take action based on items I never want happening inside my environment uh some elevated ones right these are always the fun ones uh Canary tokens you'll read the article you'll see it I guarantee you'll try it right

it's actually a really cool way to see what's going on inside your environment in a very easy way they're free uh there's there's sites that give you free Canary tokens to use use them carefully right but also use them uh to make sure you know where they're going uh it's a great way to just say hey what is my real Attack surface from the outside right not what I perceive it as but what are the other people doing um you know engage in pent tests you know everyone in this room probably doesn't need to be convinced about that uh and I won't dare to use the term purple team uh but you know a lot of those things come into

play right you know when you talk about learn to attack it learn to defend it uh it's a it's a good practice to know both sides to know how they interact with each other to know how they're going to complement each other if you keep losing to the red team find out why right I guarantee I know how they're doing it because I did it right and you know we did a red team event where uh we were the first time that that actual event had ever been caught uh it was the easiest POV we ever did in my life right so uh but there's a lot of cool things you can do by making a you know a a cool

piece from that point of view uh security chaos engineering it's uh it's something I do for you know you'll see a lot in the dev side with just chaos engineering but understanding where are my redundancies where are my lateral movements where are I you know can I actually kill something if I tried right if I said hey I'm going to kill your access after you do something how do they still have access right you know understanding where those kill chains are understanding you know where my redundancies are understanding you know do they have more access than they possibly need right there's a lot of cool things you can do uh from that point of view but uh so that's it uh

from that point of view um this QR code uh I do a lot of you creating content as well you can follow me linked if you want uh but there's other we did a really cool I think it's like a 40 page document with the cloud security Alliance just talk about the state of the cloud of non-human identities what are people doing about today how do they care about it you know what what are they using uh from a purely just you know industry standard uh the link to that uh Canary token pieces in there uh I do update it pretty weekly depending on where I am uh what's going on in the news so yeah that's it I

I'll be here for uh questions yeah we have a few minutes for question so if anybody wants to ask thank you the uh thank you for the monitoring page I was going ask the question but you covered it and I'd only say the other trick we've encountered and used is not just looking for usage but looking for where they are successfully used yes if they're suddenly used SU some place knew why what happened and the last one on the customer jum would that have been blocked if there was a successfully implemented Anderly managed to bring your own key system for the customers yeah it's a good question right so it it really depends on the level of usage

monitoring and what's the reaction to that usage monitoring is it is it something that says hey it's being used from a different IP and a different location I want to notify you is there an actual piece go ahead and kill it uh as our place to you know you know there's a lot of components from that point of view um you'll see I didn't use any special tricks or any it's just literally describing and go right uh so if I have access right if they have a uh you know maybe a white list or they have a blacklist things of that nature right there could be ways that I I can't actually use the key right now I'd find

another one right from that point of view I'm just saying like this is this this is a normal environment we encounter all the time uh from that point of view but yeah there there's many ways to go about right there's you know by all means it's not a you know One-Stop shop to fix this uh it's more about you know bringing the awareness of how easy this is uh if there if the path allows so you're heard first good going back to the graph you had was it cyber with cybernary yeah cyber just stats on how long that's available on their site or yeah so it's an article uh yeah if you click that c it's on it's on my little

link tree there um yeah from that point of view so it's a if you look up cybernary um I think the article is called like worst place to leave your tokens or something like that um but it's it's a really really really cool art it's very in-depth as well uh they go through the different sections uh of going from there thank you how do you combat the thought of I need to get this working and I'll come back to you and talk to you when I've got it working we'll figure out the rights it sounds like you don't have a good plan so we're not put an operation right so that's that's the the easy way right if there's

if there's a financial gain from pressure from the business right um that's one thing right but you need as long as you're raising a flag right you raise the flag that says hey this is not something that security team is comfortable with put the caveats around it say why you know try to put white list around it try to put something some sort of protective bubble around what's going on from that point of view um but again they're not going to like the hearing it but you just tell them like if you can't tell me what access you need you don't know what you're doing right so that's a big piece to push back on them on um you're

not it's a I've been on both sides I was going to get really on a tie raate there for a second but um you're you're not their subjects to push around right so you know be make sure it's a it's a conductive uh conversation have your backing have your your research right say hey here's the you know what you want maybe you want contributor role editor role because you don't want to be you know or delineated or user delineated when we start getting down like really granular uh tell them what the impact of that could possibly be and are they willing to accept that risk for the business right um and and they'll leave it up to you know that

business unit that you know manager director whoever to make that decision of hey I'm willing to throw my hand up to say that yes we need to provide that access because it's critical it's you know money for the business kind of go from there you've done your job so to speak right uh maybe we want to turn on non-interactive Vlogs or something like that for that you know piece for that a little bit uh to see what's actually going on but uh yeah try to Basel on it you know make them go through the proper channels good how big an issue do you think it is like so sometimes with SAS products right you have a human identity

tied to a nonhuman identity right I'm a developer I set up an apiq data so then I leave the company right you talk about offboarding you can't kill that API that's associated with that human idti um how big a problem do you think that is in terms ofusing it's it's a big problem right and that's why we have a product right we're not here just to you know throw stuff out there well we we have a product that does that human ownership we actually have uh Behavior Analysis as well tell you what is it touching where's it touching from how active is it right you know to give you the understanding of you know if Barry

leaves the company does Mary need to now own this token right from that point of view or can it leave with Barry right so there's a lot of different heris you look at and a lot of different usage uh you can look at from that point of view but that that is a major challenge uh creating for that it's even worse when you have an MFA and you take someone else's junk into your environment and they have no idea because we got rid of their team they left what it may be uh so it's one of the biggest issues okay maybe one more question and you can answer while we get the next perfect sounds good that would be great

so obviously the next best fromi give probably move them to like Environ varable system variabl so protected by that contain technology does your product do anything to work with J to monitor for are you your Environ varable system variables is that next step yeah so so we're watching vaults we're watching V variables we're also watching for leaks right so did someone you know not use the variable and use the pl Tex instead right you know tie it back to to commit tie it back to the the developer did the commit to tie it back to the owner of the token right let them know that they need to rotate right let them know uh what the appropriate process is but yeah

that's it's not a one uh One-Stop shop for everything to fix this but we're we're trying to get there right so uh yeah we we can handle lot of that problem awesome thank you everyone