PW - The attackers guide to exploiting secrets in the universe

good afternoon everyone welcome to oh is that good that too loud sorry for the surprise there uh welcome to the passwords contract um this talk we're going to have the attacker's guide to exploiting Secrets by McKenzie Jackson uh before we begin just want to quickly thank our sponsors that is our Diamond sponsor Doby and our gold sponsors pris Cloud semrep blue cat Plex track Toyota and conductor 1 they make this all possible so thanks to them uh this is going to be filmed live streamed and uploaded to YouTube afterwards this means that we'd prefer if you just keep your phones silent or as silent as possible uh no need to film it or anything we'll have all that taken

care of uh and that also means if you have a question uh please wait for someone to bring a mic around uh raise your hand we'll bring one to you that way your question can also be recorded for posterity sake so the entire internet can uh can know what you are asking um um without further Ado then Kenzie Jackson take it [Applause] away awesome uh thank you so much it's great to be here uh this is my my second talk here uh so really really happy to be presenting at Beast sides for all you lovely folks I was kind I wasn't sure if anyone was going to come I thought most people might be at the bar but I'm

really I'm really pleased to see a few faces here so thanks for thanks for making it out quickly a little bit about me so I'm from moo New Zealand uh I live in the Netherlands now I'm a Advocate I'm a security and developer advocate for a French company called GG Guardian uh you can find me anywhere on social media at the handle advocat I'm also the host of a podcast called the security repo my mom it's my mom's favorite podcast she recommends it to everyone so uh it would be great to have more than one person listen to it uh if you if you want you can find it on the QR code scan at your own risk I promise I won't do

anything malicious but you never know [Laughter] um okay so this is talk is going to be a little bit different for me it's about 50% slides and then about 50% live demos I have no idea why I decided that was going to be a good idea when I submitted this but I'm committed I'm up here now so let's do it so with that in mind if we can all take a moment to pray to the demo gods that everything's going to work uh well for me here uh so first of all uh was anyone at my previous talk uh a couple of people all right there's going to be a tiny bit of overlap at the

end uh but I always like to start with just what we what we're talking about when we're talking about secrets so a secret is a digital authentication credential so these are typically things like API key security certificates uh that type and the idea is that these are made to be used programmatically not by humans so that's why they often end up in places they shouldn't because humans touch them uh they put them inside their source code and that ends up in our applications in our code repositories so this presentation's going to look at various ways that we can exploit secrets in different Technologies each one of the the exploits I'm going to talk about is

accompanied by research and then a real life breach and uh a demo of how it works in practice so that's how the situation is going to kind of unfold I'll talk through some research of it I'll show how it actually works in the wild and then we'll have a look at how we can actually do it uh on the tools ourselves uh all the demonstrations I think I kind of put at the level of script Kitty uh so anyone here that's script Kitty and above can definitely perform these attacks uh but you know again don't do anything malicious uh with the information here so Secrets they are everywhere uh we can find them in all kinds of wacky and

weird places the more that you start looking for them the more that you realize you know just how frequent of a problem it is and this is really known by attackers at the moment and when we look at a lot of the attacks that we have secrets are probably used in just about all of them in varying ways now that might be the attacker's initial access they might have found this secret or purchased it somewhere and that's how they broke into the systems or it might be something that they've used later on as a way to elevate their privileges after they've already found their way in so they're used in in lots of different ways and why they're so valuable is

because you know when an attacker when when we break into a system we have there's a saying that a hate called a attack only has to be right once it's wrong because uh anyone that's kind of done some penetration testings know that you have to be right multiple times to be able to get anywhere um and secrets are great because they allow us to persist our access without being detected we can correctly authenticate ourselves we can Elevate our privileges uh so these are kind of the gold standard the first thing that I look for whenever going into a system uh when I have my black hat on is to try and find these secrets to be able to maintain my

control so the first one here I want to talk about is uh is kind of abusing the the GitHub API to be able to find these secrets uh and and how we do that so we're probably all familiar with this website GitHub it's the largest uh collection of source code in the world and if we look at some stats of how big it is this is last year all of these stats is just I'm just talking about public uh repositories I'm not talking about private repositories so about a billion commits were made to GitHub last year so a billion contributions of code 94 million developers are using GitHub according to GitHub themselves and last year there was about 85 million uh

public code repositories that were created so this is a huge amount of source code that we can sift through and a lot of this is intentionally public there's a lot of Open Source stuff on here but a lot of it is actually code that wasn't meant to be public um and you can find a whole bunch of really juicy information on here in your recon phase if you're looking at public activity on GitHub so some research that we did at gitguardian we scanned all of GitHub uh every all public activity for entire year and we discovered 10 million Secrets uh on there uh so these are huge so we can actually have a look at some

of the secrets that we have you know data storage about 25% so 2.5 million uh credentials that give us access to databases and data Cloud providers 20% that equates to about 2 million cloud provider keys that were leaked publicly on GitHub repositories last year now cloud provider keys are great too because you know we can validate these automatically so we're not talking about 2 million keys that look like cloud provider keys we're talking about 2 million valid cloud provider keys that we found last year so if you're paying for cloud service you're stupid you should just look on GitHub um now don't do that I'm kidding I'm kidding other things we are do you know

lots of messaging systems these are really interesting as well I can talk about that a little more a lots of great stuff that we actually uh well great depending on who you looking at there lots of interesting stuff this is and again this is all just public uh and if we look at kind of the types of keys that we found you know Google API keys are really the number one but you can get down you get lots of there's a very long list of credential thousands of different types of credentials you know but when we look at these we find huge amounts of them um that can actually do stuff you know three three almost 4% of

the secrets that we found were GitHub access tokens so someone's put the access token for their private repositories in their public repository um so you know that's very interesting way to do it uh so it definitely can happen in there so let's talk about exactly how we can start to abuse this public information so the first the first way I I don't really like this way but it's the most obvious and easiest so I'll just talk about it briefly and this is using the search feature in Google to be able to find credentials so here we're looking for a file name called credentials good place to start an AWS access key um and there's lots of

different types of these dos GitHub Dorking that you can do now amazingly you will actually find credentials this way but it's very labor intensive and it's not that great because uh most of the secrets that you will actually find in history on development branches uh so this way isn't isn't that great there's a much better way to do this uh if you like me and lazy and that's using the GitHub uh API the public API uh so I'll show you this in a minute this is this is an address that anyone can go to anyone can go to this uh this thing you don't need authentication and this is a ledger of everything that happens on

GitHub publicly so it's very easy to monitor so when when I when I say to people that things are public everyone understands that if I know the address if I know the URL or your username or repository name I can find that on GitHub that's what public means but it's also broadcast on this API so it's very easy for an attacker to monitor this now there's lots and lots of non-malicious reasons why this is good and there's lots of services that use this um in legitimate ways but it's great for an attacker as well so there's a couple of events that we really want to look out for the public event this is the most

interesting this is when a private repository is made public so this is when you make a repository public you make all your history public with it so a year ago if someone committed an API key on a developer branch and now you've decided to make that uh reposit public that's still there hidden away and the other one is a push event so let's take a quick look uh oh wrong way at this here so this is the the the the event itself um now GitHub is a fire hose of data so as an attacker this is kind of almost too much to digest especially if you're trying to look for specific targets so we actually get lots of information in

here as well including uh the email of the committers so you can look for specific company domain names if you want to if you're taring that so you might have employees that work for those those companies that are committing uh information publicly and then once you kind of have someone that's committed with a company's domain you can keep track of that uh of that user and then if they commit on any other email address you can also monitor to that so there's lots of ways of you can sift through the information of this to try and make it a little bit more more relevant but what I actually want to do now is I kind of want to do the the the

worst thing that anyone can do and I today uh are going to leak some keys on the GitHub uh uh public API because I want to show you what's happening so I have here some honey tokens so uh honey tokens are fake credentials uh that essentially are a trap for an attacker so here we have uh some AWS access credentials this is really the the top level that an attacker is going to want these are very very juicy and very interesting uh so cloud provider keys and what I'm going to do is I have this GitHub user here called leaky McGee he's a really bad developer has all kinds of bad uh practices I'm going to create a

configuration file I'm going to paste my keys in here and we're going to commit these uh to the public git repository so the the reason why I'm doing this is I want to show you how quickly these keys are going to be exploited so if I go back to my dashboard uh what I can actually see is I can see uh who's going to try and use these Keys every time an attacker tries to use this every time someone tries to use this I'm going to get their IP address where they're calling from what what API calls they're using on my keys uh so we can actually see how often and how frequent an attacker is going to be used

these so I just created this it might be a little bit too soon all right there's no there's no things here but we're going to come back to this uh a little bit later on and I want to show you how many people have tried to exploit it exploit these Keys uh just when I'm in talking so what's actually happening who are the attackers that are looking for these and how can you use it it's quite interesting because there's different groups of attackers so firstly uh there's so so much information on GitHub finding an AWS credential for a specific Target is quite challenging so what often happens is you have attackers that are very good at harvesting so looking

through these huge vast amounts of data like on GitHub harvesting all the credentials and then they sell them to another group of attackers who is really good at specializing and exploiting them so I'll get a lot of traffic on my honey token today and over the next week then it will calm down and then I'll get traffic again because that credential will undoubtedly be packaged up in a group of valid credentials that will be sold to attackers to exploit so that's typically how this situation uh really really works so is there any examples of where leaked Keys publicly have actually uh made an impact or attackers have found them so there's there's actually quite a

lot of examples of this uh there's one here that's quite interesting it's uh Toyota who leak these Keys uh themselves so Toyota obviously a car manufacturer they have a mobile application called tconnect now they didn't manufacture this mobile application by themselves they used the contractor that contractor accidentally pushed some of that source code to a public repository uh in 2017 and they remained there for 5 years before someone actually found them now these hard-coded credentials gave access to the database of all the users using tconnect so as an attacker this is this is gold for me because not only do I have these email addresses and personal information from people I also know that own a Toyota I also know that they're

using this mobile application so if I wanted to conduct a fishing campaign I have a lot of great information that I can use to Target that so this is an example of when leak keys were actually used by an attacker let's go back to an example here and we can already see that two people in India have already tried to use my AWS key uh so this is what uh so this is we can see there or one person actually it's the same IP address uh so we can see that theyve using travel hog this is an open source cred itial finder uh this is the call that they've made get call identity this is

the lowest level call that you can you can make so this is the reason why they're using this call is because they want to just check if these keys are valid but they don't want to create any suspicion around uh around someone actually having these what will happen is you'll start seeing later on different types of calls we made such as you know trying to apply policies trying to create other users because then they're going to try and find out and very quickly what typee what CPE of access this key has so it doesn't take very long we'll take a look at the end of the session and see how many other people have tried to exploit that we can

all take uh bets on how many how many we think all right so I want to talk about uh something else now so we've just talked about public uh source code so this is very interesting there's lots and lots of great stuff on here uh we can find really interesting credentials in public uh source code but if we're specifically trying to Target a certain person a certain group group this may not be the easiest approach to take it may not we may not be able to gather any credentials you for that what's much better is trying to get access to private code repositories so private code repositories code is very very leaky this is just some of the examples

of some Source codes that have leaked uh recently uh there's been massive ones like twitch's entire source code 6,000 repositories were were were leaked uh and some massive companies that I would consider have a great security posture like Microsoft Samsung uh so this is a much more interesting way because source code whilst it's actually really sensitive um and it is a treasure chove for Secrets especially in the history I'll talk a little bit more about that at the end um but it's it's actually very leaky you think about all the developers that have access to the to it you think about all the places it's backed up it's shared in wikis it's in JY tickets it's in uh

slack messages so we can really leverage the fact that source code is going to end up in lots of places to try and gain access to it and uh this here is a is is some interesting so in our research GG guardian we work with lots of companies to secure their secrets and typically what we'll find is in an average company that we work with they have about 400 developers and with that if they're if they if they're on the upper end they'll have about four appsec Engineers the developer average is about 100 developers per one appsec engineer I think it's a little bit less um but you know that's what that's what it tells us

so if we do some maps when we scan for that we're going to find uh 3,400 credentials typically that's what we typically find when we do our initial scans of a company this F in their private source code repositories now we actually averages averages out is it's actually 13,000 total secrets that we find and 3,000 of them are unique so if we take that number 13,000 we have four appc Engineers that that means that they have to sift through all of that information and even a just the 3,000 they have to sift through all that information and what they have to do they have to investigate with the developer if this is a real credential

they have to rotate it without creating any downtime and they have to redeploy again without creating any downtime so this is this is a massive problem and that's why they're there because it's actually very very hard to solve so how do we get a oh first I'll talk about uh some twitch so let's have a a real life example of a breach so twitch had their source code exposed it was due to a misconfiguration very briefly twitch's source code was publicly accessible uh bad guys are very uh very trustworthy when it comes to being able to find this stuff very quickly so someone found it and then they leaked it in a torrent so there was 6,000 repositories that were

leaked we scanned these and we found over 6,000 Secrets now that seems like a lot but actually this is really good twitch was actually doing a good job to only have 6,000 secrets in their source code repositories is probably better than what we'd expect there was 194 AWS keys in there and and lots of other things including stripe keys and GitHub oor credentials so it's really interesting what we can find in there so that leads us to the question all right if we can make our way into the private source code that's great but we have to first do that so how do we actually get into our private source code so there's a couple of ways I'm only going to go

through one of them in the demo the first one the first two are not very exciting but they are pretty standard so first one is buying access this happens a lot a lot of those Source codes that were leaked were leaked from a group called lapsis it was a bunch of teenagers um and they managed to get access into Microsoft Nvidia Samsung source code so how did a group of teenagers break into these companies with such fantastic security posture they posted in Telegram and they said hey we'll give you money if you if you give us access so not very hackery but it works it still happens and it's a it's a viable way of actually getting in

the other way is fishing the recent one is the Chrome extension they really were targeting uh developers in this for Chrome extension developers uh to try and gain their credentials uh to to try and be able to move into to different areas exploiting misconfigurations and git this one's quite exciting and the one that I prefer and the last one which is very difficult is supply chain attack so I'm going to talk about the code COV supply chain attack but basically code COV is a tool that was compromised the attacker's goal was only to get into the the private source code repositories of certain companies so when we talk about misconfigurations what's a misconfiguration and g that we can

exploit well there's one that's very very common and that's having exposed doget directories so when you go g in nit if you're used to coding it's going to create a folder called dogit a directory called dogit in this dogit folder is all your history all your metadata of everything that happens it's actually really really sensitive um and what can happen is this often just this folder finds its way out onto publicly publicly accessible places if we can find this folder we can go back and we can re structure it to get back to the original G repository this is really really useful for us because we've got all that history in there and it's really easy to

do cyber news did some research they did lots of scanning they actually found 2 million of these dotg directories that were publicly exposed so this is actually happens everywhere so let's have a look at exactly uh how we can go about finding this so there's a couple of steps that we need to take uh to be able to do this so the first one is that we need to be able to understand if we have a Target we need to understand hey what are all the subdomains that they have what are all the domains that they have where this folder might actually be located so there's a really cool tool now I've been told that I'm not allowed

to expose any real vulnerabilities in this talk so I'm not going to I'm going to behave um I'm just going to use an example but there's a tool I like to use two tools to do this chaos and SubFinder I'm just going to demo SubFinder this is a very easy what I do is now I'm looking for all the subdomains that relate to hacker one I'm using hacker one cuz I know that there's no vulnerabilities that I'm going to unintentionally uh discover and in a minute hopefully do I still have my internet yeah and in a minute so this is now looking for it and in a minute it's going to spit out a list of all the

subdomains that it's been able to find that relate to a hacker one and what I'm going to do is I'm then going to use that list to be able to um here we are and now going to use this list and I'm going to look in all of these to try and find if there's any exposed doget directories now this is a very small list usually it would be a very very large list in companies uh that we're looking for uh but it doesn't matter so I'm going to use a tool here called git scanner uh so I'm going to use git scanner and what this does is this scans this list for me and it tries to find

these exposed. git directories and then it will dump them into my folder if it finds any so I have this hacker one list uh so I'm just going to scan them

oh

sorry so what this is doing is now it's going look and it's going to check to see if there's any git uh exposed git directories now a lot of these are saying non nonvulnerable um again I didn't want to show any that we vulnerable it comes up green when it finds one and it's very very frequent that it does find one and you'll see here it says may be vulnerable what this means is that there is actually a dog directory there but it's not uh publicly accessible but if you want to you can go further on that and you can actually try and find because quite often the folder is not accessible but you can find

individual files from that so it's worth doing further exploration even if comes back that you don't have permission for it often that very frequently they're misconfigured and you can and you can see it so then once we have our um once we have our git directory what we need to do then is we need to extract that information so I here have a folder where I have a dot g directory here that's been that I've I've discovered but when you click on this there's no source code it just comes up with areas like GI hooks so we need to be able to convert this back to a point where we can access the code I can use the same

tool for that with the extractor functionality and I just put in the area that my my folder is at and then I want to put where I want to Output it to this is going to take just a small minute this is going into that dogit directory and it's converting it back into a proper git folder with all of its source code that I can then use and find so now we have this extracted folder and you'll see in here we've gone back to the original source code so this is how we can often find access to it now once we have access to the source code it's very easy to discover secrets in there

we can use lots of different credential scanners I could almost guarantee that they will be Secrets buried in the history here so this is a great way of being able to access that private source code that's going to contain a trove of sensitive information for us so if you want to know this the tools that I used chaos and SubFinder are great ways to find subdomains extract them uh I use git scanner to find the dogit directories and extract them dump them in extract them uh and then I will scan them with a tool called GG Shield to find uh secrets so that's one way that we can kind of get into the the private

code repository if you want to explore more ways of how you can abuse misconfigurations in git there's a great tool called GGO and this is basically uh uh a git repository that has lots of different Mis configurations with instructions on how to do it and what you'll actually find is a lot of these Mis configurations that are in get-go are out there in the wild that you can exploit so there's lots of different exploit exploitation methods and using misconfigurations to get into different areas of source code so why why on Earth are secrets in source code so much I'm going to move on to other areas now finding them inside applications and running applications but why is source

code such a problem so the the number one way is that we find source code in the history uh that we find secrets in the history of source code so this is an example if you're used to coding you're probably familiar with this this is a git uh just a git branch and what we have is we have our main branch this is kind of what we would deploy and then when people are working on different features we often create feature branches for them that are kind of separate what a developer will often do is get a credential hardcoded into their feature Branch thinking no one's going to find it they will then remove that because they know that that's not

what's meant to happen but they were just quickly doing it and then that gets merged and no one actually knows that there's a secret in there that secret will exist for the entire life of that g repository unless you rewrite the history which is a total pain I wouldn't recommend it but basically that's why secrets so frequently in there when we're writing source code everything we do is tracked everything we do we keep a record of so we make a mistake one of our employees one our developers make a mistake even just for a minute and that can come back and bite Us in the day when people like me are trying to get into their private source code there's

lots of other ways that they end up and code in autogenerated files and in logs if you have a debug log uh you might have a print out of your environment variables these very frequently contain Secrets as well uh not having any get files having secrets. txt I we see this a lot you know uh sometimes that we create them in templates like if you have a Jango project it will create a key for you you have to actively go in and remove that key uh and put it somewhere safe A lot of people don't um and then we also see that people that just sharing Secrets uh in source code so lots of ways why Secrets end up in

source code and there's lots of ways to be able to exploit this as an attacker both publicly and privately all right so I want to move on to the next part which is Docker IM I couldn't decide if I want to do Docker images or packages uh you know exploding package manages but I thought Docker was told a more interesting story so I'm going down on this path so uh if you're not familiar with what a Docker image is a Docker image is like a mini virtual machine if we want to run an application that application needs lots of other things to work so Docker packages that that all up so that your application should run in any environment Docker Hub

is where we store these Docker images so lots of companies St all their Docker images on Docker Hub you can download it and that's how you use that application uh there's millions of millions there's tens of millions of of public Docker images uh on here uh and yeah there's there's there's a huge amount of information now these should be public because if you have a public application you want people to be able to use it so there's no problem of having a Docker image public but we started looking into discovering how many of these Docker images will actually have secrets inside of them so we found out nearly 5% of Docker images contain at least one hardcoded

secret so 5% so if you download it doesn't take you very long to be able to find something interesting in here you download you know 10 10 Docker images you're going to find a couple of uh 100 Docker images you're going to find a couple of secrets so and that doesn't take very long to be able to exploit this so why why do we have secrets inside Docker images there's a couple of reasons of what it is but essentially a Docker image is an application what is an application it's just your source code now it's been transformed into a a a version that's nonhuman readable but that doesn't mean that I can't extract the source code from it so what I want

to do is I just want to talk about an attack that uses that used the docker image and this one's interesting because it relates back to private source code as well so there's a a tool called code COV it's a code coverage tool um it sits in your cicd pipeline doesn't matter if you're not understanding what all that is but basically what this tool does is it tested how much of your application was being tested it did a very specific thing and you'd put it in your environment and it you wouldn't be too worried about it how people used this product is that they downloaded their publicly accessible Docker image to be able to to use it inside that Docker

image there was a hard there was a hard-coded credential uh in in plain text attackers actually found this hardcoded credential what did this credential do it allowed them to be able to tamper with the source code that doc Ed there was specifically A bash uploader script they turned this they turned code COV the tool malicious and at the time it affected 20,000 of code CV's customers so what did their malicious tampering actually do what code kov uh what the attackers did was they injected one line of source code that said every time you run code COV I want you to take a dump of all the environment variables and I want you to send them to me at my remote address as

the attacker so any time one of these 20,000 targets use code COV the attacker got a bunch of Secrets From it now a lot of those Secrets hope were test credentials or for test environments so didn't affect production but the attackers weren't after those credentials they were actually after one specific credential they were after the credentials for private source code repositories so the attackers then moved from code COV into the private source code of lots of different companies and this included uh Hashi Corp monday.com rapid 7 twio they all uh had breaches because the attackers made it into their source code and there was Secrets inside their source code so this is just an example of really you know how this can

yeah how this can happen a secret in a darker image means that you can get into the private source code so this was the example of a supply chain attack this is quite sophisticated um but finding the secret and Docker really isn't now I don't like to pick on companies normally ever uh about preachers but I'm going to pick on one and that's hasik Corp because Hashi Corp had a had a breach because of this and if you don't know what Hashi Corp is hash creates a very cool product called Vault is probably the best Secrets manager on the market I actually really like Hashi cor I don't think they'll ever give me a job now but um but I'm a

big fan but uh what actually happened was hashy cop created created the term Secrets rule hashy Corp said created Vault to say with Vault you no longer have to give developers access to your secrets you never have to worry about Secrets R all if you're effectively using Vault hasy Corp had to announce that they had a private key in their source code because of this breach so if Hashi Corp has secrets in their repositories there's absolutely no hope for anyone else um all right so let's have a look into into uh Docker so the first thing I want to actually do is take a look at what is a Docker image so one of the

problems of why this this all happens is we don't understand the Technologies often uh intimately that we're using so when we create a Docker image all right we can't open that with anything else we can't access the source code from that easily so we just assume that it's all secure and you can't break it down this is what a Docker image looks like on the inside this is a tool called Dive I'll explain what what we're kind of looking at so on the on the left where my where my keys going up and down these are layers when you create a docket image you do it in layers so you can add a file and remove it throughout the

process and if we go down we'll see a bunch of green so the green is what has been added so if we have a look at this uh in this application

sorry you can actually see that there's a bunch of a bunch of source code in here that we that we've added in so we can see a lot of python projects so this is the source code of our application this source code is inside our docket image so if there's source code in our Docker image and there's secrets in our source code we can extract it easily from this Docker image and we don't even need to extract it we can just use tools to be able to scan it so here I'm just going to use a tool uh called GG shield now again I'm not allowed to to show real life vulnerability so I'm scanning a Docker

image that contains fake Secrets but what I'm doing is I'm using this tool called GG Shield I'm just saying Hey I want you to scan this Docker image it's going to download that Docker image for me it's going to scan it and it's going to let me know if it finds any secrets inside here so when I said that 5% of Docker images contain uh contain a secret so if you do this 100 times you're going to find about five Secrets five real secrets and this is how quickly you can easily do this 100 times by the time I finish my presentation and here we can see we have some Secrets again these aren't real um these are

just example Secrets but we have some API Keys we have some beerer images we have some username and passwords and these are typical things that we find inside Docker images really easy to to extract this information uh if we need to so I really like to Harmon the point that non-human readable does not mean secure I just want to talk about one other thing so why why does this happen so again it's kind of not understanding the technologies that we have uh if you're not familiar with Docker this will look like mumble jumble but I'll explain what's happening is when I said that Docker creates is built up in layers we have here an example that I

found quite regularly which is concerning so we have well basically what's happening is they're adding a file into this Docker image while they're building it the file is called net RC that net RC contains credentials that's used to connect to a package manager and then what they're doing at the very end they're removing that net RC and they're doing this because they think hey this is a sens file I'm going to use it and then going to get rid of it and move on with my life not realizing that docket is built up in layers I can still get that file from your original Docker image so there another reason why kind of why this

happens so again really easy way to be able to find and exploit credentials not very difficult and there's lots and lots of them out there right I'm going to talk about the last one here which is mobile applications now I did a whole talk with this before if you were at that talk I apologize the next couple of slides are going to be uh the same but uh for the purposes of other people I'll I'll still go through it so here this is something you might see on the Play Store on the App Store depending on what you use this is a mobile application so what is a mobile application uh again they're non-human readable so that means that they're

secure right again no it does not so there's two types of files that are going to be on depending on what operating system you're using if we build an a mobile application it's going to be an IPA if it's on Apple and it's going to be an APK if it's on Android what are the files these are glorified zip folders especially the apple one and I'll explain why um so let's go into a quick a quick demo of of what we have here so here I have two here I have two applications this is my iOS app. IA my Android app.apk and what I'm going to do the first thing I'm going to do is

I'm going to am I on the right yeah is I'm going to decompile the Android applic app so I'm using a tool called JX uh and basically this is going to convert that APK back into a version where I can read now it's very easy to download these apks you can do it through a mirror um you so there's there's no way you can download any files that you want and in a minute this is going to give me some source code uh that I'm going to be able to look through now for the iOS application much more difficult I made this joke at the last time too so I apologize you're hearing all my jokes twice

but all we need to do to extract source code from IOS application is change the extension to zip so then we go use dotzip we extract it we give it a minute now we have our payload now this has an app inside of it and all we need to do is remove this and now we have our source code for our iOS application if there's Secrets inside inside here inside that they're very easy to find and also our Android application has finished so now we have uh our source code for our Android application as well now we can look through this uh if we want to try and find these secrets manually that's a very very uh long process so instead

what we can do is we can just be lazy because that's what attackers often do and we're going to just scan this so right now I'm using GG Shield again to scan uh the path where exploited my Android application too we're going to go ahead and do that it's going to take a little bit of time um because we're scanning 21,000 files um and we'll come back in just a minute uh we'll come back and we can have a look at the secrets that we found in here so right the moment it's getting some errors because there's a lot of media files in here that it's getting stuck on but we'll come back in just a

second so if we think that our mobile applications are secure and that people can't reverse them we can so this leads you to the questions uh how worthwhile is it as an attacker to try and find Secrets inside mobile applications spoiler very worthwhile so if you're interested in the tools and the workflow that I'm that I use to do this here we have a tool called G gplay downloader this is how I downloaded the tools I decompiled it with JX again this is just for Android and I scanned it with GG Shield so this is the workflow that I used this is at the level of a script Kitty right anyone can really do this um

and then for Apple much simpler I Ed a tool called IPA tool to download it you just change the extension and then you can scan it with GG Shield uh so I quickly go back still still detecting uh so now let's just go and do an actual uh example of a breach this example is from Jason heck he's a legend uh he was very kind enough to come on my podcast my mom really enjoyed that episode um and uh he talked us through an exploit that he did when he was hacking into doing some pen penetration test of a bank where he found secrets so this bank which was a tier One Bank in the

states uh was uh was being was being obviously it's being used by the bank's customers so Jason was taking a look at that he decompiled it he noticed that some of the functionality of the bank was that you could take pictures of checks um and cash those checks he found out that those images were not being stored as encrypted on the device and then here's curious as to where these images were being stored they were being sent to an Amazon S3 bucket and inside the application was the key for that Amazon S3 bucket that was hardcoded so then he could access that Amazon S3 bucket and found 10,000 images of unencrypted images of checks so you know

this is from this is this is this is a financial institution that's hardcoded secrets in their mobile applications um just because people don't think that we can extract them all right now we're finished scanning and as you can see we've got a lot of secrets in here now this one I've hidden the secrets but this is a real example this is a real app that I got from the Play Store I've hidden its name um you don't have to look too hard but we can have a look at what we've found uh we've found some Google API keys that are valid so we so this is quite serious there's some slack web hooks that are valid I love these

because it means I can do internal fishing campaigns and post in there and there's also uh some Google oor keys as well so and Facebook Keys these are all hardcoded into the application this isn't that un abnormal um so you want to wonder how many applications actually contain hardcoded Secrets how many well we don't need to wonder because our friends at cyber news did some great research on this and they found that about 56% of applications contained contain hard-coded Secrets now I like to stress here that that doesn't mean that 56% of applications have hardcoded secrets that you could easily use as an attacker there are keys in there like Firebase Keys which wouldn't allow you

to do anything malicious unless they had misconfigured their Firebase account which regularly happens but there is lots of secrets on there that are very interesting as well and as an attacker I'm still interested in the keys that I that I can't use it's a piece of the puzzle that means that if I get another level of access or I can do something else I might be able to use that key in a way so there still sensitive even if I can't use them immediately the keys that we found the number one Keys was uh Google storage buckets so this kind of relates back to the example that Jason gave us of having storage keys in there

the reason why I assume this is so problem is it seems to make more sense to send an image or something directly to data from the app and not go through your back end but that's a terrible idea because that means your keys are in your app so lots of really interesting stuff that we find in here and the total amount of secrets that they found was 124,000 um and that was from 50,000 mobile applications so you know if you're wondering what to do on this weekend that's a great way all right so I'm almost at the end I I have three minutes to go I'm very happy none of my demos failed so I can relax now I'm

definitely going to the bar after this um but the last part I'm not going to talk too much about this I'm just going to talk about it for three minutes my colleague Dwayne did a talk that was much that was in detail about you know how to properly manage Secrets if you didn't go to that you might want to have a look but we'll have a look the number one thing don't hardcode your API keys don't hardcode your secrets even to test if they're anywhere to find I'm going to find them if they're in they might end up publicly if they're in private source code repositories we talked about lots of ways I can get access to those so

just don't hardcode your secrets just because I can get into your uh private source code shouldn't mean that I can paone your organization or your applications um and because source code is something that's very very hard to secure based on how leaky it is so we need to adopt the mindset of this should be considered open source even if it's private uh user correct Secrets manager so I'm trying to get back in the good books of hasi cor Vault the best one use it um but there different levels to this right so uh if you're using something a tool like Vault this is a very very heavy tool that's going to requir people to manage it it's kind of require

training of your staff to use it but it is the best but the problem that I always see is that sometimes people have enthusiastic uh people in the security team that will want to implement something like this but not have the correct ability to roll it out so then it doesn't get used and then it becomes pointless so then if you go down there's dedicated secret managers like run on SAS tools a keist do pass one password has some great tools out there as well for Developers for managing Secrets uh short of that you can use the secrets managers on your Cloud infrastructure and short of that you can encrypt them and put them in

your get don't do the last one it's a bad idea um but the other ones you can all do and what I like to stress is that you know um people always say that you should aim for the top and you should aim for the top but that doesn't me necessarily mean you should go there straight away you got to make sure that people are actually using them because uh my gripe is a is that before we have the argument of what Secrets manager is the best let's stop committing our secrets and get let's start there and if whatever whatever tool is going to help us do that it's going to be easy to adopt we can work it towards going up

later um use automated Secrets detection this is my quick plug so I work for GE Guardian so we have automated secret detection and tools we have some open source tools I use GG Shield a lot and if you don't trust me and you don't uh there's some other tools the other open source tools that you can look basically there's no excuse not to have secret detection it's somewhere in your in your process there's plenty of tools that are open source that can help you do that because that's what I'm going to be using to try and find your secrets so you should be doing the same thing that I'm going to be doing as an attacker get

hooks are a great way to actually stop the bleeding um and rotate regularly limit Privileges and whiteless services so the one I'll talk about a lot just rotate regularly you should absolutely be rotating your keys regularly there's two reasons for this one obvious one is that if I find them hopefully they're invalid two it means that you know how to rotate them so if you have a breach you actually know what to do the amount of times that people have a breach and you they'll uncover that actually Keys have leaked the attackers have access to them and no one knows what they do and if you revote it is that going to crash production so if you're rotating

regularly then you have to know what the keys do because you're rotating them limit with privileges if if if the amount of times I see admin Keys being being used for read only um you should never have admin keys for anything other than admin level stuff and whitelist Services make sure that only certain machines can connect I'm overtime by 1 minute uh here's some uh some research if you're interested if you want to learn more about how to securely manage Secrets there's the maturity model and if you want to learn more about the research that I've presented there's the statea secret sprawl uh there so thank you so much for your time I hope it was an enjoyable

session for you and I'll see you in the bar in a minute yeah we have for questions y perfect um first one comment I know that you work for uh G Guardian but also GitHub has built in secret scanning as part of get up get up Advanced security yeah built into their platform the other question um it cost a lot it cost a lot of money it does cost a lot of money no you're not wrong yeah it's a very expensive product um thankfully I'm open well all the stuff I work on is open store so I get it for free for all that stuff but yeah um given that these keys are just falling out of

these APK files I know this is maybe not a question for you because you might not have the answer but supposedly both Apple and Google are doing auditing of these apps before they end up in the App Store do you have any insights as to why as a part of the auditing that they're doing they're not running these off the shelf tools to make sure that these credentials are not ending up in these apps before they get published on the App Stores yeah for sure so Secrets detection is actually uh quite difficult to always get right it's very hard to have like a high level of of accuracy in there so if you were going to block an

app from going onto the App Store you know for false positives that will create a lot of friction uh to be able to do that um so there there that that is absolutely why what the auditing looks like from the app is that you know is this doing anything malicious uh is is data being sent into to weird places are you folling our procedures um it's interesting because this brings up the shared responsibility model so uh the shared security responsibility model is that vendors and security people and organizations should all bear some responsibility so what you're kind of saying is that apple and and and and a play store should have some responsibility for for for secrets and I

agree but that's a very hard conversation to convince them to do it there are some companies that I want to give a shout out to that are adopting this shared responsibility model really well one of them is AWS if you leak an AWS key on GitHub uh like I did today uh AWS will actually quarantine that key for you and let you know that you've leaked it that is is a great that is a great step that AWS took to be able to do that properly twio also does that as well so um there are some companies that are bearing responsibility um GitHub has obviously SC like adds free secret scanning for for open source but they

also have an invested interest in selling that like I do so you know no we'll we we'll battle about this later uh but yeah hopefully that answers the question speaking of which can we get a look at your uh AWS honey secrets and see how oh yeah sure sorry I forgot and by the way thank you great talk uh okay so we have uh uh we have uh nine requests coming from here uh at the moment so quite quite a lot of activity uh from here they're all using the G caller ID uh and there's a few different IP addresses but some of them so here I see too so I think there's probably a

couple of different actors uh in here one from India and then some other ones that we unknown of where they're coming from so nine nine is what we've got which is not bad for for 20 minutes you just some inspired some really terrible ideas scan the App Store take all the secrets out check them into a git repositor on GitHub and then GitHub would then tell get that would then have Ed us rev revoke the secrets on them yep yep for a it's like you know revocation is a service via GitHub yeah yeah yeah yeah yeah thank you um have I guess have you guys or do you know of any projects to um kind of

implement a GG Shield type uh scanning within um repo uh or Docker container like like a Snick type scanning like doing it automatically like yeah no we don't know anything else so at G Guardian we've been scanning public gift repositories for like seven years and we're we're interested in doing um the same for Docker but the problem is is that what we're scanning in GitHub is uh is a is a is a diff it's a it's a small segment of code what we're scanning each time so we can do it now if we wanted to scan darker images we're going to have to pull those darker images which can be very large um so it's a lot harder if

there was some way that we could just see like because and that's every time you update a Docker image if there was some way that we could see what's you know the diff of the docker image that would be much easier so it's it's a challenging problem because the docker images are so large oh that's yeah I mean Snickers open source project right so it could be kind of a pluging yeah it could be yeah yeah yeah yeah yeah there's there's there's definitely ways to do it um uh but but it's just kind of about having the resources to do that you know and that's and that's probably similar with the App Store uh comment as

well I mean these are very big files you got tens of millions of them it takes resources to be able to scan them so it's it's difficult um but you know the bad guys don't care so we have the time they have the time any other questions well thank you all so much thank you McKenzie thanks [Applause] everyone