BSides LV 2023 - PasswordsCon - Tuesday

foreign [Music]

[Music]

[Music] thank you [Music] foreign [Music]

[Music] Hallelujah [Music] oh yeah [Music] thank you [Music] thank you thank you [Music] foreign foreign

[Music] thank you

[Music]

thank you

[Music] foreign [Music] thank you

[Music] foreign [Music] foreign [Music] foreign [Music] foreign [Music] foreign [Music]

[Music]

[Music]

[Music] foreign [Music] laughs [Music] [Music] foreign [Music]

[Music] foreign

[Music] foreign [Music] thank you [Music]

[Music] foreign [Music] foreign [Music] [Applause] [Music]

[Music] foreign [Music] thank you [Applause]

[Music] foreign [Music] foreign

[Music]

[Music] myself [Music]

[Music]

giving me Wind and Rain some kind of butterfly baby [Music] don't leave me [Music] but I don't wanna jinx it baby [Music]

[Music]

[Music] foreign [Music] don't leave me alone baby you'll get me in the rain [Music] baby

[Music]

[Music]

oh oh [Music] [Music]

[Music] I don't know [Music]

[Music]

foreign

[Music]

[Music]

move it up

[Music] foreign [Music] foreign [Music]

[Music]

[Music]

[Music] foreign [Music]

[Music]

[Music] thank you [Music]

[Music] foreign [Music] thank you [Music]

[Music]

thank you [Music] thank you [Music] thank you [Music] foreign [Music] foreign [Music] foreign [Music]

[Music]

thank you [Music]

[Music] foreign [Music] thank you

[Music] thank you foreign

[Music] thank you [Music] foreign [Music] foreign [Music] foreign [Music]

[Music]

[Music] thank you

[Music] foreign [Music] thank you [Music] foreign [Music]

[Music]

[Music] thank you [Music] foreign [Music]

[Music] foreign [Music] thank you [Music] foreign [Music] [Applause]

[Music] what's up [Music] [Applause] thank you [Music]

[Music]

baby you'll kill me [Music] even some kind of butterfly baby

[Music] dream between myself

[Music] don't wanna overthink it baby [Music]

some kind of butterfly baby [Music] appetite

[Music] but I don't wanna jinx it baby [Music]

[Music]

[Music] thank you foreign [Music] don't leave me alone baby [Music]

[Music]

[Music]

oh oh oh [Music] [Music]

[Music] thank you [Music] foreign [Music]

[Music]

[Music]

[Music] foreign [Music]

[Music]

[Music]

[Music] foreign [Music]

[Music] [Music]

[Music]

move it up

moving up

[Music]

[Music]

thank you [Music]

[Music] thank you foreign [Music]

[Music]

[Music] thank you

[Music] foreign [Music] foreign [Music] oh yeah [Music] thank you [Music] foreign [Music] wow [Music] foreign

[Music] foreign

[Music] thank you [Music] factions

foreign [Music] all right

[Music] thank you [Music] [Music] foreign [Music] foreign

[Music] foreign [Music]

[Music]

[Music]

[Music] thank you foreign [Music]

all right [Music] foreign [Music]

[Music]

[Music]

thank you [Music] foreign [Music] foreign [Music]

[Music] thank you [Music] foreign [Music] [Applause]

[Music] foreign [Music] [Applause]

foreign

[Music]

[Music] you're giving me wind away some kind of butterfly baby

[Music]

[Music] I don't wanna overthink it baby [Music]

[Music] don't leave me [Music] but I don't wanna jinx it baby again

[Music] but I don't wanna miss you baby [Music]

oh [Music] foreign [Music] don't leave me alone baby [Music]

foreign [Music]

[Music]

[Music] oh oh [Music]

[Music]

let's go

[Music] foreign [Music]

[Music]

foreign

[Music]

[Music]

move it up

[Music] come on [Music] foreign

[Music] [Music]

[Music]

[Music]

[Music]

[Music]

thank you [Music]

[Music] thank you [Music] thank you

[Music] thank you [Music] foreign [Music]

[Music] thank you foreign [Music] foreign [Music] foreign [Music] foreign [Music] welcome [Music]

thank you [Music]

good morning everyone um I want to get started a little bit early because I want to make sure that I don't cut into any of the time uh for Aldo here because I'm really looking forward to this talk so I just want to go ahead and at the behalf of the password con staff and b-sides welcome everyone here to Vegas you know this is a really cool thing to be doing this again here okay um so um just to take the the energy down a notch here it's got a few announcements just to start off things here uh so first of all um I guess we're contractually uh you know I really want to thank our sponsors because we

wouldn't have this here you know uh if it wasn't for them so thank you Adobe you know our gold our Diamond sponsor here as well as Toyota uh semgrap and blue cat so thank you we really appreciate this I love doing this here you make this possible so I really appreciate this a lot also for cell phones uh these talks are being streamed on live here so please you know uh don't have any side conversations on your cell phones or something like that there you know this is a passwords contract so I'm sure we can go ahead and figure some way to be able to deal with that if it happens here uh so also kind of really real quick here you

know we've been doing passwords con for a long time you know haven't we solved this problem or something like that there um and the answer is obviously no but uh one thing I kind of want to you know get a little bit of an audience q a uh starting to start it off here but who here has had their password stolen I certainly have you know I've almost getting bored of this now you know it's like oh my password got stolen again I guess it's a Tuesday um so let me go ahead and tell you probably the worst data breach I've ever been a part of so when I first graduated college I loaded up literally everything I own

into my car and I was going you know job searching and stuff like that and one day I woke up you know came down from my hotel room and the back window of my car was broken and I found out I owned significantly less stuff so like some of it was really kind of annoying like I had disassembled all my furniture so they stoled a bag of all the bolts which is just mean um but you know they also stole all my important documents because I left them in my car so they got my birth certificate they got my social security card they got my passport they got blank checks like literally if you can think

of something that's physical uh that you don't want to get stolen that got stolen from me so you know I might not actually be you know Matt weird I might not be you know I might be the person who broke into the car as far as you know um but you know I was able to recover from that I was able to go ahead and get new credit cards I was able to get new checks I was able to get new IDs and I think that's really speaks to kind of the resiliency of this authentication of the fact that I can suffer the worst possible breach that you can even imagine and I was able to still be able

to recover from that and that's where I think you know passwords are really a big role of this too because we have so many different promises passwords you know we get them stolen all the time yet we can still book plane tickets we can still show up at Vegas we can still you know have the hotel you know except for credit cards and I think that really speaks to you know the value of all the hard work that everyone here has been putting into this type of field so you know an obvious question though is you know why are we still using passwords they get stolen all the time that you know can't we move to more of a

passwordless type of an option here and that's why I'm really excited to go ahead and introduce Aldo here uh because he's going to tell us about some of the challenges that occur when we go ahead and move to this is here as well so can we all go ahead and give a hands here to Aldo all right thank you uh thanks for that introduction uh well my name is Aldo and I'm here today to talk about passwordless and passwordless security uh so it's actually an honor to be here in passwordcon and you know uh I was a little bit nervous to be the first one to be speaking today uh but actually uh quite the opposite because that actually

takes the pressure off because if this stock sucks uh then the bar is gonna be so low for everybody else so uh if I if anybody here is a speaker uh you're welcome so let's let's try to make this talk uh suck a lot for you for all of you all right so so uh yeah who am I right so I've been doing abstract for about 15 years a little bit more uh you know from doing pen test so calls review to everything in between I am also an ovas chapter leader in my city and right now I am running the application security program for hyper which it is a password let's start it's kind

of ironic that I'm talking about passwordless vulnerabilities but it's gonna make sense trust me so before I begin uh can you just let me know how many people here are Developers all right uh what about pen testers nice and is anybody here already using passwordless either at work or in your personal devices awesome cool thank you all right so let's get started so uh let's begin by answering this question uh does anybody here think that you've seen a password solution is actually worse or is actually less secure than having a password can anybody think that right I feel you all right well the answer to that question uh actually no is not the less secure uh yeah sorry

about that uh yeah well and actually that is that is my whole talk uh thanks for coming here today uh thank you no no no uh sorry that was a terrible joke I'm sorry about that now this is the agenda that we're gonna be talking about today uh first of all some background on why am I giving this talk uh what is passwordless for those of you who are not familiar with this Paradigm and then we're gonna deep uh dive into what are the actual issues with passwords implementations and at the end we're gonna have some recommendations for all of you uh developers pen testers and Enterprises uh so to set up the expectations uh this

talk is about uh you know this is a brief introduction into passwordless uh this is not a full token passwordless uh we're gonna be talking about vulnerabilities in a passwordless implementation uh specifically a web application uh you know passwords can be used in other types of applications but these vulnerabilities are related to passwordless in web applications uh and this Pro these issues were identified in a particular passwordless product uh that shall remain nameless but no it's actually my employer so uh so but this vulnerabilities applied to any application that is using passwordless so this is not specific to a product it applies to any application that is used in passwordless so what this talk is not about uh we're

not going to be disclosing any new attacks I'm not dropping any old days today uh I'm not doing a new a full token password Here uh we're not going to be talking about how to break cryptography uh you know passwordless one of the implementations uses uh public key cryptography so we're now going to be talking about that today and uh yeah why am I doing this today right so basically uh I think that nobody's talking about passwordless vulnerabilities uh you know the vendors right now and the whole industry is moving away from passwords slowly but it's moving away from passwords to the uh passwordless Paradigm but uh nobody's actually talking about it and you know

uh a lot of big players such as Microsoft Google Apple they all are pushing for passwordless but they are not talking about their issues right I actually went to balldb.com and searched for passwordless and I get several results none none of it uh some of the stuff that I'm gonna be talking about today is already public and they are not there uh out of curiosity not to throw shade to any of the competitors but I went ahead and I went to their websites looking for some advisory page some something related to vulnerabilities and I didn't find anything so in in short uh nobody's talking about this uh just a couple of weeks ago when I was

getting ready for this talk I went ahead and looked for uh baskets on Google and as you can see here we have many examples of companies already implementing passwords we have well they are implementing pastkis which is a way of doing passwordless and we're gonna be talking about that so uh we have one password we have Tick Tock that is now using pass Keys we have GitHub we have PayPal we have Apple ID and this was just a couple weeks ago and I view those right now and search for passkeys you're probably gonna find more examples like this so a lot of companies are going passwordless and uh but how secure are they that is the

question so what is passwordless for those of you who are not familiar with this Paradigm and Paradigm or are not really sure what I'm talking about password list essentially means uh not using a password in short that's that's that's that's it basically uh and no that's really not a job that is actually what it means one of the most traditional implementations uh is using public key cryptography but there are more uh and essentially well if you are not familiar with public key cryptography essentially it creates a keeper one public key and one private key the public key station the server the private key essays with you or your device and then the application sends a

challenge that you have to sign with your private key and only you can sign the challenge it's very secure a lot of cryptography works that way nowadays so uh this is how uh the most common ways of implementing passwordless nowadays a couple of quick mentions uh when you are using a password manager that is not passwordless and I mentioned this because I saw a post the other day about somebody telling oh go passwordless use a password manager and uh that is obviously not passwordless they're just well you're restoring your password somewhere else so that is not passwordless when you have two-factor authentication that is also not passwordless because you are providing your username and password and then you

are providing an additional Factor uh and lastly when you have one time password or obviously the name suggests you have a password that is valid only once but it's still a password so in short passwordless means using no passwords and uh these are some of the ways that you can actually Implement passwordless uh let's let's talk about them real quick uh one of the simplest ways to implement passwordless is using a magic link uh for those of you who are not familiar with magic links essentially a website sends you a link to your email you click it and you authenticate to that website that's it and uh I think that's why it's called Magic link because it looks like magic

you don't have to provide a username you don't have to write a password you simply click on a link and you're in uh that is actually very practical but uh it has some disadvantages such as if the attacker knows you're uh well if the attacker has access to your email well they own your accounts right uh these magic links can be delivered using email SMS and basically uh any way that you can reach your user the second part of the uh way to implement uh password is using security Keys uh such as these ones so these are UB Keys these are physical security Keys uh anybody here already uses duplicates or security Keys oh perfect that's amazing so the uh this

is actually my preferred way to implement passwordless uh but as you know this is this could be quite expensive especially if you are a big company uh for instance let's say that you are Facebook and you want to deploy passwordless to all of your users uh I don't know how many users they have probably a billion I don't know so can you imagine just trying to purchase UV keys for your million users uh that's not gonna scale and lastly we have Biometrics which work in the same way uh well very similar to security keys they create a public key pair and you have your own private key and that is unlock using your Biometrics so this is a very good alternative to

security Keys uh everybody has a smartphone nowadays and they can use that smartphone as a way to provide passwordless access using your Biometrics this is just an example of how is lack those passwordless essentially you go to the login to provide your email and they are going to send you a limit an email with a code and you provide that code and you don't have to provide any passwords at all that is another way to implement passwordless all right but now we have Fido I mean uh we we see that we have several ways to implement passwordless but we really didn't have a standard until recently so uh everybody was doing it on the wrong

way so this is why the fight or light and Fido Alliance success the fighter Alliance basically is an organization that uh is in charge of maintaining a specification for the fighter protocol uh Fido means fast identity online so essentially it has two main components one of them is web Autumn which is uh short for web authentication it's a set of apis that browsers can use to communicate between uh these authenticators and web applications and the other component is the client Authentication Protocol which we're not going to be talking about today we're going to be talking heavily about web button today so again this is a set of apis that the browser used to communicate between uh

the authenticators which can be uh security keys or fingerprint and web applications also I wanted to mention passkeys uh when I say baskets when I say security Keys when I say uh passwordless and when I say biometrics I am referring to the same thing they are not the same thing I am simplifying things but for the purposes of this talk uh it's all the same and uh all these vulnerabilities are gonna apply to pass Keys security Keys uh Biometrics and any other Authenticator and just to double down on web button again uh this is a set of apis JavaScript apis that allow your browser to communicate with authenticators and um web application or relying party

authentic errors can be anything such as physical security key uh Touch ID on your Mac Windows hello and your Windows computer and Biometrics on your mobile devices so this is an example of what using web Orton looks like on the left you have a UV key in the middle you have fingerprinting Chrome and lastly a fingerprint in uh on an iPhone so if you previously have seen something like this you are already using well not passwordless but web attempt all right so why why going passwordless uh I found this fine gentleman online which uh probably hates passwords as much as we do uh so well we are in a password right uh probably a lot of you

know that everything about password suck so uh you know storing them protecting them hashing them uh yeah you have to use an algorithm that is slow enough to stop attackers but not slow enough to bother users uh well it is a it is a problem right when you've got passwordless you don't have any more passwords to remember you don't have any more passwords to protect uh you don't have to worry about ineffective complexity factors and you don't have to worry about people don't know in their own passwords uh you know the last time that I got my wife a phone I was helping her trying to set up her phone right and I asked her hey can you

log into your Gmail account so we can set up your phone and she was like uh I don't know my password like what do you mean you don't know your password and she doesn't uh you know she'll log into the Gmail account once and she forgot about it and the same thing happened with my parents when I got them new phones they didn't know their passwords so uh well they were all able to reset them but I mean this is an issue nobody well a lot of people don't know their own passwords uh passwordless is traditional then uh is faster than traditional MFA it protects against phishing for those of you who are not familiar with this essentially when you

register a security key that security key is registered to a specific website so if somebody was trying to fish me and they sent me a fake link for my Microsoft account even if I were able to uh fall for that trick and I plug in my jubikey it's not going to work because that security key or that fighter credential is only linked to a specific website so that is not going to work and lastly we don't have any more password resets we don't have any dictionary attacks etc etc so all good things when not using passwords so thanks thanks for going through that I needed to provide some background for those of you who were not familiar with

passwordless now let's talk about this so passwordless has a few misconceptions uh like we just saw a lot of people may think that passwordless is less secure basically you don't have a password right so what is protecting my account if I don't have a password no password means no security that is not the case we're gonna see it and uh for instance we also I have heard from people who think that somebody else can unlock their phone using a photo or maybe their twin for people who have twins uh and also people thinking that fingerprint can be clone uh those are General misconceptions that I have heard from people uh really not working intact and also we have stolen or lost device

issues so basically uh if you lose your device an attacker is going to be able to authenticate to a website using your device I think that if an attacker steals your your phone and they are able to bypass your biometric authentication uh you probably have bigger things to worry about than they being able to log into your Facebook account for instance and lastly uh this one is actually true if you lose your device you're gonna lose access to your account so that this is why you must have uh either a backup account a backup device or a way to recover that account so that's actually a valid concern all right so now let's talk about real

vulnerabilities in passwordless implementations finally all right so uh this is the first issue and I on purpose didn't provide the title because I didn't want to give away the all the fun in the beginning so uh this is uh directly from the web attempt documentation so if you go to that website you're gonna see that uh basically you need uh you need to provide a user object uh essentially that user object has a name a display name and an ID this is required for any uh whenever you're trying to authenticate or whenever you are trying to add a new authenticator to an account uh just give me one second oh sorry oops excuse me excuse me one second

there we go sorry about that

all right so yeah so I was saying that you need to provide a user object when you are adding a new device this is expected because you need to map that specific device to a user right so this is all in the documentation for web attempt and then I went ahead and looked for some examples of those specific implementations uh the first one is from uh or friends at jubico essentially no I mean dual security and you can find the same they are saying that you need to provide a user object and that user object contains an ID a name and a display name uh this is directly from the documentation again this is expected uh

this is the way the web button works uh one more example uh this we can find the same thing uh you need to provide a user object it has an ID a name and a display name uh this is Javascript code by the way this is the web authent API uh lastly I went for uh the ubico documentation in order to add a new uh jupy key you have to do the same because jubikis are also a 502 credential and are also using web auton so you need to provide a name a display name and an ID all right so that's part of the EXP of the expected data that web AutoNation is expecting right

uh also for passkeys as I mentioned pass keys are also a 502 credential fast keys are also uh using web authent so you also have to provide an ID a name and a display name with all that being said this is a real implementation of passwordless solution [Music] um for you uh for those of you who are doing pentas you know that this is burp this is basically uh an HTTP intercepting proxy it's allowing us to tamper with all of the data that is the browser is sent into the server uh if you take a closer look uh you can see that the application is sending a username and a display name as part of

the specification a quick question here for my princess or friends what do you think it could happen if we try to Tamper uh that username to another email let's say the CEO's email any guesses no well uh the specification doesn't actually say it but uh and this is particular specification if I tamper with that value instead of uh adding a new device to my account I was able to add a device to a different account so essentially by doing this uh you could essentially take over any account in the company that you wanted just by adding a new device you could impersonate any user in the system and remember that we are doing passwords here so we don't have any other way to

authenticate uh simply by sending this request you can actually take over any account that you like this actually has a CBE uh you can go ahead and look for more details let's talk about it all right so this application was you know was following the specification to the letter it was doing everything that the documentation says uh there wasn't any anything wrong with it uh one thing I want to mention is that the documentation doesn't say that you don't have to trust this data uh it may sound obvious to US security people but it's this is not obvious to the people who are implementing web attempt uh or at least if it's documented I couldn't

find it so uh that actually makes me think I mean how many other applications are there that are just following the web button documentation and they are not providing that additional check um essentially uh if you want it actually if you like to remove that data from the request it's not going to work because web authen is expecting that data so uh yeah so it's look it's leaving the developers to know I mean they it's they are on their own to implement this additional check because nobody's telling them that they have to do this right so uh this actually is for me I I should reach out to the Fido Alliance and ask them hey why are you not uh pointing this out

why are you not documenting this or if you are where is it because it's not obvious and nobody I mean at least this implementation was not doing it all right uh moving on to magic links if you remember magic links are simply just a way to users to authenticate right uh they receive a link they click it they are in this is a way to implement passwordless uh but one thing I like to mention is that web applications are usually just as vulnerable as they are flexible and what I mean by that is that you can do pretty much anything that you want with web applications if you want to read a username from the headers you can do

that if you want to read a username from the cookies you can do it I'm not saying that you should but you could uh so this is a problem with uh because you are letting your developers do whatever they want right so this is a simplified example this application had a way to authenticate users uh using a link something like this essentially they the application was provided in a token the user click it and they are in however this particular application had two different user roles one for users one for admins they were both able to authenticate using a magic link however uh we can do some Force browsing and for instance if we try this is an

example of course but if we added the word admin the token for the user was good for the admin and uh let's say that you got a token as a user you simply change the link to an app that were admin and you'll become an admin just like that this also has a CPE you can go ahead and look for it and view one more details so let's talk about this the token that the application was using was actually quite secure I mean it was using uh good cryptography it was really long I mean uh it was being created using secure functions uh the talking was expiring as soon as you click it and even if you didn't click it the token uh

was being expired after some time so the magical implementation was didn't have any flaws apparently but as you know developers need to be sure to uh protect every single endpoint they need to be able to uh verify every single input field and attackers only need to find one uh endpoint that you missed and this is what happened here all right moving on to number three uh this is actually another take account takeover uh using parameter tampering it's very similar to the previous one and again our web applications are very flexible uh when we're dealing with passwordless authentication is not one single request for instance when you are talking about username and password usually the username and the password

are sent in a single request right so and that's it but this is not the case when we're talking about passwordless uh usually it needs several requests uh you need to send a challenge then you need to sign it then the application is to verify it really uh it needs a lot of requests and in this particular case uh the application was doing a lot of validations but again they missed just one so the attacker was able to provide a valid authentication for the user but authenticate it as a different user instead this also has a CV and you can go ahead and look at if you need more details all right um let's let's talk about one more uh

this one is about account creation uh again web applications are very flexible you can do pretty much what you want with them and this time we were talking about a demo application uh you know companies sometimes provide demo applications so they can uh teach their customers how to implement passwordless or how to implement anything else right in this case this particular vendor was used in a demo application but in the user creation flow the application was not checking whether the user existed or not so essentially what happened is that you went to the login page you try to register you provide a username and the application wasn't checking if it existed or not and essentially since

we're doing a Fido authentication you were adding a device to an account so from a passwordless standpoint it is the same as adding a an authenticator to an existing account or to a new account because you are simply adding a new device right so what happened is what you were able to add new devices to any existing accounts and uh you were able to take over those accounts again this is a demo application but still I think it's pretty uh pretty good to know that we should be doing all of these validations and uh lastly the August of 10 we have to remember that we are talking about web applications here so we don't have

to we we must not forget about doing the basic validations uh that any other web application needs to happen right we have to do input validation we need to do security checks and every single endpoint and so forth and so on all right uh so now let's talk about some recommendations uh for Developers if you already have a secure sdlc uh I think it's a it's really good that you you have it if you don't uh you should Implement one what I mean by that is that you should do you should have some form of security testing in place um maybe some code scanning uh some dependency check and all of those things that are very good to increase the

security of your application also you should do some testing and then more testing and then more testing uh yeah I I mentioned this because all of the issues that we talk about they were found by humans uh there is simply no way that a tool can find these issues and if there is uh please let me know because I don't know of any tool that can do this uh probably not even AI so the best way to find the security vulnerabilities is by having testing done by humans also in breast testing for customers a lot of times companies don't want to be allow their customers to do pen test because they think they're going to find

issues and you're gonna have to fix them that's sometimes is true but I mean uh I think that's best because you can find those issues before an adversary does or before remote customers do or before you have a leak so uh well it's true that this is gonna mean more work if you have issues I think it's best to do it because otherwise uh it's best to fix it before everybody knows and uh that way you're gonna prevent breaches and security incidents for Enterprises uh if you're thinking of implementing passwordless uh in you are thinking about doing it uh in-house versus a vendor if you're thinking about using a vendor I think it's really important to hold

those vendors accountable uh you know it's important to realize I mean how many times are they getting testing I mean are they just doing one pen test a year for compliance are they doing uh multiple rounds of testing uh you know do they have a bug money program do they uh are they testing new features do they have a public CBE program so I think this these questions can actually uh make it apparent if that company is actually doing something about security or even maybe they don't even have a security program so uh yeah this is for all the Enterprises for pen testers uh yeah actually when I was first testing uh password resolution I was kind of intimidated by it I was

thinking well this is using public key cryptography right I'm not a cryptography I'm not going to be able to find any issues here right well no the truth is that at the end of the day these are just web applications uh they have the same issues as any other web application my personal recommendation for any pen tester is going to be just go for parameter tampering this is where you're gonna find the most issues uh and also unauthenticated apis so yeah all of the issues that we talked about before they are related to web applications so basically all the same things that you're doing uh are gonna work in a passwordless application and my recommendation would be not to spend too

much time on trying to break the cryptography I mean if you can go for it but uh it's probably going to be a lot of time and you're not gonna get any results there for users uh actually embrace the future I mean I talk about how passwordless can be uh can have issues but uh actually it's way more secure uh whenever you are presenting with the option to use passwordless uh you should do it I mean I think uh I think that passwords have way more issues than passwordless and as you can see here uh whenever an issue is found It's Quickly fixed so you know passwordless is a it's a very secure option all right so I think I went really

really fast because uh that's it uh so that means we have a lot of time for questions if you have them uh if not uh thanks a lot for being here and thanks a lot for your time thank you [Applause] I'm doing the example you gave of

uh not really uh the question was if uh you as users can protect yourselves against somebody else adding an advice uh really there's no way you can do that uh the application is the one that has to be providing that additional check uh to make sure that the user who is adding that the device is the one that owns that account so really as users we don't have any defense

there we go

yeah do you have any recommended pattern oh there you go do you have any recommended patterns for account recovery uh what we find I guess when we see most the implementations we see leave old school methods in place for recount recovery so it'll be like you still have a password if you need to recover your account or use totp or something like that or use a phone number and we find that most vendors aren't fully disabling Legacy Security in favor of passwordless so I'm wondering if you have any fully passwordless account recovery mechanisms sure uh well this is not a Blog I mean this is well we do have in my vendor we do have

a way to do that so essentially what it does is that we are providing a way to verify the identity of the user essentially uh we have a specific website that you can go you type your username and then we do an identity verification uh right now we're using it by using uh essentially it's like a meeting I mean I can just I can describe it better uh by showing a demo but essentially we do have a way to do this uh it verifies your identity it connects you where your manager and uh basically it can do face verification or phone verification that's up to the to the manager and once they have verified that you are the

person who owns that account they send you a magic link and you can add a new device any more questions no right all right well thanks for being here uh feel free to uh I didn't have a Twitter but feel free to add me on LinkedIn if you like uh feel free to reach out to me if you have any more questions and thanks again for being here thank you [Applause]

that was too fast and uh thank you everyone for being here uh two o'clock we have a talk with MacKenzie Jackson on our secret safe and how mobile applications are leaking millions of credentials which would be pretty good and then hopefully if we all follow this awesome with Aldo here then um you know we won't have a conference next year here for your passwords so we'll do passwordless

but but everything was clearly

[Music] thank you [Music] thank you [Music] thank you [Music] thank you [Music]

[Music]

[Music]

[Music] thank you foreign [Music] [Music] thank you [Music] foreign [Music]

[Music]

[Music] thank you [Music] foreign [Music] foreign [Music]

[Music] foreign [Music] foreign [Music]

[Music] [Applause]

[Music] foreign [Music] [Applause]

foreign

[Music] foreign

[Music]

[Music] between myself

[Music] don't wanna overthink it baby [Music]

[Music] everything don't leave me [Music] but I don't wanna jinx it baby [Music] so dead

[Music] thank you [Music] baby [Music] foreign [Music]

[Music]

[Music]

oh oh [Music]

[Music] thank you [Music]

foreign

[Music]

foreign [Music]

[Music]

foreign

[Music]

[Music]

[Music] foreign [Music] foreign [Music] [Music]

[Music]

[Music]

Move Along

[Music] thank you [Music]

[Music]

[Music] thank you [Music] thank you [Music]

[Music]

[Music] foreign [Music]

[Music] foreign [Music] thank you [Music] foreign [Music] foreign [Music]

thank you [Music] foreign [Music] thank you

[Music] thank you

[Music] foreign [Music] foreign [Music] thank you [Music] thank you [Music] foreign [Music] foreign [Music] foreign [Music]

[Music]

[Music] foreign [Music]

[Music] thank you foreign [Music]

[Music] thank you [Music] foreign [Music]

[Music]

[Music] thank you [Music] foreign [Music] thank you [Music]

[Music] thank you [Music]

[Music] [Applause]

[Music] thank you [Music] [Applause] foreign [Music]

[Music]

[Music] you're giving me wind away [Music]

[Music]

[Music] don't wanna overthink it baby [Music]

[Music] don't leave me [Music] foreign

[Music]

[Music]

[Music] oh [Music]

baby [Music] baby [Music] don't leave me alone baby you'll get me [Music] some kind of butterfly baby

[Music]

[Music] oh oh [Music] [Music] foreign

[Music] foreign [Music]

[Music] thank you [Music]

[Music]

[Music] moving up

[Music]

[Music] foreign [Music]

[Music]

[Music]

move it up

[Music]

[Music]

foreign

[Music] thank you foreign [Music]

[Music] foreign [Music] foreign [Music] foreign [Music] oh yeah [Music] foreign [Music] foreign [Music] foreign [Music] thank you foreign [Music]

thank you [Music] foreign [Music]

[Music] thank you [Music] foreign [Music] thank you foreign [Music] [Music] thank you [Music] foreign [Music] thank you [Music]

thank you

[Music]

[Music] thank you [Music] [Music] thank you [Music] foreign [Music]

[Music]

[Music] thank you [Music] foreign [Music] foreign [Music]

[Music] thank you [Music] foreign

[Music] [Applause]

[Music] foreign [Music]

[Music] thank you [Music] thank you foreign [Music]

[Music] dream come true

[Music]

you're giving me Wind and Rain some kind of butterflies baby you'll give me [Music] everything don't leave me [Music] but I don't wanna jinx it baby [Music]

[Music]

[Music] thank you [Music] baby [Music] potato [Music] don't leave me alone [Music]

[Music]

[Music]

oh oh [Music]

[Music]

thank you [Music] thank you [Music]

[Music]

[Music]

foreign [Music]

[Music]

[Music] foreign [Music]

[Music] [Music]

[Music]

[Music] thank you

[Music]

foreign [Music]

[Music] thank you [Music] foreign [Music] thank you [Music]

[Music]

[Music] thank you [Music] foreign [Music]

[Music] all right [Music] Hallelujah [Music] thank you [Music] foreign [Music] foreign [Music] foreign [Music] foreign

[Music]

[Music] thank you [Music] thank you [Music]

foreign [Music] foreign [Music] foreign [Music] foreign [Music] thank you foreign [Music]

[Music]

[Music] foreign [Music]

[Music]

foreign

thank you [Music] [Music] thank you [Music] all right [Music]

[Music]

[Music] thank you [Music] foreign [Music] thank you [Music]

[Music] foreign [Music] thank you [Music]

[Music] [Applause]

[Music] foreign [Music]

[Music] foreign

[Music]

foreign [Music]

[Music] dream ing of myself

[Music]

some kind of butterfly baby [Music] don't leave me [Music] but I don't wanna jinx it baby [Music]

[Music]

[Music] thank you [Music] baby [Music] everything [Music] don't leave me alone baby

[Music]

[Music]

oh [Music] oh [Music] [Music]

thank you [Music] foreign [Music]

[Music]

foreign [Music]

[Music]

[Music] um

foreign [Music] foreign [Music] [Music]

[Music]

[Music]

[Music]

moving on

moving up yeah [Music] thank you [Music]

[Music] foreign

[Music] foreign [Music] foreign [Music]

[Music]

[Music] foreign [Music] foreign [Music] oh yeah [Music] thank you [Music] foreign [Music] what's up [Music] foreign [Music] foreign [Music] thank you

[Music] thank you [Music] thank you [Music] foreign [Music]

foreign

[Music] foreign [Music] thank you [Music] thank you [Music] foreign [Music]

[Music]

[Music]

[Music] thank you [Music] [Music] thank you [Music] foreign [Music]

[Music]

[Music] thank you [Music] foreign [Music] thank you

[Music]

[Music] foreign [Music] let's go upstairs [Music] [Applause]

[Music] foreign [Music]

[Music] thank you [Music] foreign

[Music]

[Music] baby [Music]

[Music] don't wanna overthink it baby [Music]

[Music] baby you'll give me everything don't leave me [Music] but I don't wanna jinx it baby [Music] so it's okay

[Music] thank you [Music] baby [Music] foreign [Music]

[Music]

[Music]

oh [Music] oh [Music]

[Music] thank you

thank you [Music] foreign [Music]

[Music]

[Music]

foreign

[Music]

[Music]

moving up [Music]

anymore [Music] foreign [Music]

[Music]

[Music]

[Music]

Home Alone

[Music] thank you [Music]

[Music] thank you [Music] foreign [Music]

[Music]

[Music] thank you [Music]

[Music] thank you [Music] foreign

[Music]

[Music] Hallelujah [Music] thank you [Music] thank you [Music] wow [Music] foreign good afternoon everybody and welcome to besides Las Vegas we are here for the password con appreciate you coming to hear from Mackenzie Jackson who's going to be talking about are your secrets safe no they're not we know they're not finding millions of credentials and mobile apps just a few things we want to say thank you to our sponsors especially the diamond sponsors Adobe and our gold premise Primus Prisma sorry Cloud blue coat Toyota and conductor one it's their support along with yours that makes this what it is cell phones please turn them off we don't want to hear it if somebody's calling it better be God

also if you have any questions save them for the end because we he doesn't want to be interrupted so on that note let's get started Mackenzie all yours thank you

that was uh that introduction is probably like going to be the highlight of this talk so but thank you all for coming here I'm really excited to be presenting at b-sides uh Las Vegas uh this has been one of the one of my goals to be able to present at this conference I have a funny story before we start is uh last year I was a volunteer and I was on the registration desk um and I was chatting to the people on the registration desk to me and I was with three other scissors so if the Sciences are the people that are volunteering here I'm kind of terrified to know what the audience members are

but I'm uh really happy to uh to be presenting here so my name is Mackenzie a little bit about me before we get started I'm from artero in New Zealand but today I live in the Netherlands and I work for a French company so there's a range of uh of uh of countries there you can find me anywhere on social media at the handle at Advocate Mac and I also am the host of the security repo podcast it's my mum's favorite podcast she hasn't missed an episode uh and it'll be really great she she recommends it to you uh if you want live dangerously there's a QR code scan at your own risk uh to take you to

that all right we'll get into the the topic so what we're going to talk about uh in this session is really discovering secrets so we're going to talk and originally initially about kind of Secrets now we're in passwords cons so I don't need to spend too much time about this why they're a problem then we're going to look at discovering secrets and source code I'm not going to spend too much time about this because the presentation after me is actually going to go deeper into it but this is kind of what kicked off my interest in mobile apps specifically and then we're going to talk about discovering secrets in these compiled applications downloading them and finding them uh finally we'll

talk a little bit about how to secure these store secrets and then we'll we'll have a go into some questions all right so just quickly um I'm sure everyone here is familiar so but just in case to get everyone on the same page what do I mean when I'm talking about secrets so I'm talking about digital authentication credentials uh these can be things like API Keys security certificates credential peers and the key difference here is that these are made to be used programmatically and generally machine to machine right so I have yet to successfully memorize an API key and use it these are meant to be used by our systems to authenticate themselves now why that's important is

because when things are made to be used programmatically they often end up hard-coded or in the wrong place because humans still handle them even the machines use them and so that's really what we're going to be talking about now is identifying where they've kind of leaked out of where they're where they're meant to be how we can identify them and how we can use them so why do Secrets exist a question I ask myself every day um but if we take just a kind of a modern application a mobile application um you know secrets are used because of a shift in how we build our software you know our applications aren't monoliths that do everything we connect to lots of

different services so you know the easy one to talk about is something like OCTA where are you going to build your own authentication or do you Outsource that to a service that has has is doing just that you know that or to use our goalia for search do you do your own credit card processing or do you Outsource that to stripe especially if you're trying to get around the 30 fees that the app stores charge um but um applications quickly end up of these are compiled of these different services and they all communicate with Secrets but it doesn't stop there because we have to have back-end infrastructure we have to have testing we have code that

we need to deal with so then our infrastructure also uses lots of these secrets now our mobile application needs to talk to these as well uh potentially through the back end but they still exist as secrets and then it doesn't stop there because we want to monitor it we need to have monitoring of it perhaps we want to have crash logs being sent somewhere from the app so they we need secrets to be able to do that and get information and these are all potential access points and we haven't even talked about the micro services that we create so your simple little mobile application very quickly turns into a collection of all these different Services all doing

different things and we need to authenticate with each of them and we do this through Secrets but every single one of these points if I'm an attacker this is a potential entry if I can gain access to to something even if it doesn't seem that interesting you may not think that me getting access to a slack channel is important I'm going to talk about that exact exact example later but as an attacker I can do lots of different things to be able to abuse that and leverage that to elevate my privileges and gain access into you know lots of different areas all right so how should Secrets be stored this is going to be a very very

simplistic example um but just before we get into all the bad things that happen we'll talk about how it's meant to happen so we have a front-facing applications for Android it's an APK for Apple this is an IPA we shouldn't have any secrets in these now unsurprisingly we we we're going to discover that there's lots of secrets in here but this is we really shouldn't we should have a secret stored in our back end you know perhaps through a sequence manager um or just in our Cloud infrastructure they often have secrets manager we want them loaded into our local memory and that's what communicates with the third party services and sends our data securely to our application this is how

it should be set up but often lots of things uh change people cut corners or they feel like they're doing it more efficiently although come up with lots of arguments as to why the insecure way they're doing it is actually secure I'll talk through some of them as we go through but this is really how it should be done but it's not often how it is done in practice so let's get into the first part of this finding secrets and source code as I said I'm not going to like go too deep into this if you're really interested the talk after me will go deeper but I want to address this because this is really what made me start thinking about

Secrets inside source code inside mobile applications based on some initial research I was doing in source code so to give an idea the status secret sprawl is a report that git Guardian the company I work for releases every year and one of the things that we do is we monitor public code repositories to try and identify if secrets are leaked in there um now the biggest one is obviously GitHub so GitHub has huge amounts of information there was more than a billion commits or contributions made to public repositories in GitHub last year and we scanned every single one of those to try and identify uh how many secrets are out there in public repositories so last year we found 10 million so 10

million secrets and we've validate a lot of these so this isn't 10 million random higher entropy strings that look like Secrets but it just you know URLs or unique identifiers now this is 10 million secrets that we're fairly confident um our true positives so this is a huge amount of information now if you want to win some candy if you remember this number for the next presentation it's going to come up um but we looked at the file extensions that we had and I got an interest because we're going through lots of information and I wondered how many of these how many mobile applications how how many of these would actually concern mobile applications because I had heard

a lot about mobile applications being breached pastors being found in there so I wanted to use this research and this information to get a deeper understanding of how these mobile applications are actually using Secrets now truth be told I'm not a mobile Dev um I've I've driven into this into this topic uh from a security perspective and learned along the way but there's a bunch of files that really are specific to mobile applications that kept coming up in our research so if we look at some of them the dot properties now obviously these aren't only exclusive to mobile applications but ones that we're when looking into the frequently related to mobile applications XML files were often

related to mobile applications and the plist file which is nearly always related to iOS development and so when I had a look into these I did some further research to find out how many of these uh contain secrets and what are some of the file names that we had so if we're looking just at Android applications the main one that we were discovering secrets in was the Android manifest.xml we found 23 000 nearly 24 000 secrets in this one XML file um you know other ones as well as string sort XML was a big one that was related to Android developments uh there's a long list of exit of these files that we can go down here are some of them and

just the last one I always like to add in a funny one API key dot properties feels like something that probably shouldn't be in a public git repository uh definitely not but we still find you know 65 of these uh Keys being leaked uh so this is interesting we found similar results when we looked at iOS Android application specific the main one that we found was the Google services info.plist uh this is a a file that's generally always related to Google services and namely Firebase Now by default this shouldn't really be that sensitive because it should only contain your Firebase ID which might be useful for attacker but not really but then people started doing really weird things

they started adding secret Keys into this file as well I guess maybe it's handy to have both of them together and we started seeing lots of weird Secrets inside these and also lots of other ones and again API keys.plist feels like someone that shouldn't be in a git repository but often is so when we looked at all of these files it really got me it got me thinking is to if these are the types of files that are containing secrets in public git repositories then it really has to be that in private git repositories the problem is much worse and if this is the flow that means that the application at the end is going to have those secrets

in it so looking at this source code uh really got me thinking of to that ultimately these secrets are going to end up in the app mobile application um and so that's what kind of started me off with this first I want to talk about exploiting uh secrets in these in these public source code I'm not going to spend too long about it but just bring it up how would an attacker that's specifically looking at exploiting a mobile application uh be able to discover these types of files in public places like GitHub so generally when Secrets leak for applications they're not they don't leak in a official repository they leak in a repository attached to an employee that maybe

accidentally leaked something or is starting their own project that doesn't realize their secrets in the history that belong to their organization but there's a couple of ways so firstly this is the kind of my least favorite way but I'll talk about it briefly because it's the easiest you can just use the GitHub search feature what we call GitHub docking so we know that the Android manifest.xml has lots of files so if I'm looking for to exploit an Android application I might narrow this down to specific keywords I'm looking for an API key and there's lots of different types of these docks that we could do this isn't a great method the reason why most of the secrets that

you find in source code are in the git history when you're using the GitHub search feature it doesn't search the history it just searches the top level or the kind of the latest version on the main branch so there's a much better way that we can programmatically try and find these keys inside mobile applications and that's using the GitHub API so we have this uh events API if you anyone can go to it you can do it on your phone right now it's api.github.com forward slash events everything that happens publicly on GitHub is on this Ledger it's on this API so what we can do is we can start using the public events and the push events to

try and find code that or code for for things that shouldn't be in GitHub for instance narrowing it down to the Android manifest and strings this is a huge amount of information to digest but if you're trying to exploit a mobile application if you know that it's going to be an Android manifest XML or strings.xml that's going to give you your most amount of uh most amount of prizes then we can narrow it down and all of a sudden this fire hose starts becoming digestible uh so this is really some of the ways that we can do it and also if I'm trying to exploit a specific mobile application then what I might do is I might discover

what employees are working for that if they have personal github's accounts and then abusing this API to try and find uh files that relate to this specifically for them source code all of this sent me off down this journey of trying to figure out how mobile applications can be breached can can they can we find the secrets inside of them and exactly how do we go about doing that uh all right so let's get into that so firstly uh on the Play Store this is probably something of what you see and we look at these and we're trying to figure out what are mobile applications in their raw form if I how can you download it so uh most people make the

mistake that non-human readable means secure so when we've submit an APK file to the Android play store or an IPA file to the uh to the Apple Play Store we all think that okay this is this or a lot of people seem to think that this is some kind of black box it's not human readable you can't really extract any information from it from just that file so that means it must be secure it's totally unhackable but that's absolutely wrong with so many things like this same with packages same with containers uh so what we started doing is trying to first step is to turn these files back into something that's human readable and it's very easy so there's two types

of files when we're looking at mobile applications so the first file that we we have is the IPA from Apple and the second file is the dot APK from Android and so what are these These are basically glorified zip folders that we can use to extract a source code and once we've extracted the source code from them then we can we can start looking into them to try and find any sensitive information that may be in here so how do we actually find these secrets so I'm going to run through quickly if you guys all say a prayer to the demo Gods I'm going to hopefully this will all work but the first thing that I want to do is

I just want to show you how easy it is to extract these files so here I have two I have my Android app.apk and my iOS app.ipa these are real files that I've downloaded from the respective Play Stores or app store so uh that I've kind of chosen at random but I've removed their name because there's some sense of information inside of here so I don't want to get in trouble for disclosing uh just give me a minute

okay so the first step is we need to try and get this back to something that's uh human readable so to do this well I guess the first step was we need to download it so you'll notice that you can't download these on your computer you need to use some kind of uh mirror or some kind of tool so there's an easy one for Android applications called uh gplay downloader so I use that to download the application then I'm going to use a different tool called uh jdac to which is a decompiler to get this back to its original form so just to show you what that looks like uh it's just going to take a few seconds

and then that's going to be able to spit out and take me back to the source code because once you have the source code then really you can actually start doing some interesting things and and looking for some files so now that we've done that let's open up what we've just created here so this here is the source code of the of the application that we've downloaded so here you'll see the Android manifest we've talked about this file so this is where I can already start to find some interesting information and go through in here and also we know that there's some other files uh very interesting like a strings.xml uh which we can find in various files

um here to try and get information about uh and see if there's any strings hidden in there the the problem with this is that these are really massive files and buried under all of this may be something interesting maybe uh and hackers or at least I am very lazy so we're always going to find a better way to do it so what I want to do now is just show you what I would do in real life um is this is what is this scan this file uh for secret so I'm using a tool called GG Shield uh fear warning this is a tool that uh get Guardian my employee creates so I'm wildly biased as to why I

use this but it's definitely the best totally um

whenever I'm on stage I forget how to spell the most basic words

I told you I was just checking to see if everyone's awake you know making sure everyone's paying attention

feels wrong but uh let's give it a go

all right so this is going to take just a little bit of time oh

okay this is going to take just a little bit of time to scan so I want to go back to something else so I've easily shown you how to extract the um the the Android application now I do want to warn you the iOS application is much more complicated so I hope you pay close attention and take lots of notes as to how we can do this so what we need to do in our Android application is we take here our DOT IPA and we change this to dot zip and

we have a look in here

and we have our source code from this so hopefully hopefully you took uh lots of notes on how to do that I can't do it again um but yeah that's really how interesting so when I mentioned that these are glorified zip folders particularly for the iOS version I'm like absolutely dead serious these are literally glorified zip folders so now that we have the source code we've extracted that it's really easy if there's a hard-coded secret in in your source code it's going to end up in your application and we can easily extract it using simple tools or in the case of iOS not even using any tools at all so I'll try and see what my scanning is doing

uh we'll wait a little bit these are all okay we're all done so we'll go up to the top and these are some of the secrets that we've found I want to stress again this is a real application downloaded from the Play Store um but I have hidden all the secrets so you can't do anything malicious with it so the first stick with the ascent we've got some valid Google API Keys these can be interesting these are in a Java file so that we know that these are hard-coded when we decompile things obviously we don't get the original names we lose a lot of information it's human readable but it's uh we don't have everything that was there but we can see

that we have these Google API Keys here we have something very uh and we all can also can see that these are valid so we've checked to see if these are valid so we can use these we also have Google oauth keys we can't check these automatically but these can be extremely sensitive depending on the setup we have Facebook app keys and we also said we have here some slack web Hooks and those either these are valid I find slack web hooks literally everywhere because people kind of assume that they're not that sensitive and in some ways that they aren't so what does this slack hoop allow me to do this leg hook allows me

to be able to post information into a private select Channel what's the use case for this I'm going to assume probably that there's maybe some error or debug uh logging that's being sent or when something happens it kind of gets sent a crash log gets sent to a select a slack Channel that's interesting but me as someone that's malicious I can do lots of things like I can create some clever uh phishing messages in there to say that hey this integration has disconnected please re-login or something like that go to a mirror of slack and get people to give me their credentials now it doesn't always work but it's it works a lot of the time

because how else would I have that slack web hook right you've broken down some barrier of trust I'm not a Nigerian prince emailing you I am I'm your own system telling you you need to update your information so this is the info this is kind of what we get and it keeps going down some more web hooks in different areas um and we have a couple in I'm trying to find this interesting files here and here we have the strings XML so this is obviously what we found so what I found originally on GitHub that led me down this path to kind of wonder if this is interesting you know translates into the app people were putting secrets and

strings.xml and putting it into a public git repositories then I wondered hey does that mean they're going to be in the app turns out it exactly means that they're going to be in the app as we can see right here so this is just a process that I used for Android I downloaded it called Jeep with a G play I decompiled it with jadex and then I scanned it with GG Shield so you know this is very simple there's other tools out there too and and this isn't that complicated honestly I would say that this is definitely at the level of a script Kitty or even below um where you can you can just go through

go through this you can pipe these all together and just download download apps all day long and try and find sensitive information I'm going to talk about when we've done that and what we've found later on but we definitely uh this is my backup demo just in case it failed and the similar process for for Apple apps I used a tool called IPA tool to download it Apple won't let you wait they won't let you download anything unless it's on a mobile so what this tool does is basically trick it into thinking it's on a mobile it uses your Apple ID and you can download it there there's also lots of mirrors online where you can just

download files uh you just change the extension you don't even need a tool to decompile it and then you scan it with GG Shield so again really simple about the level of a script Kitty to be able to do this so we're not talking about highly sophisticated attackers if we were I probably wouldn't be presenting to be honest so let's talk about when uh this breaches a breach that's actually happened so this one here is from Jason haddock this is a story that he told on the on the security repo podcast that's a great episode if you want to check out uh and this is about when he was doing a penetration test on a bank

application so if you don't know Jason he's absolute Legend uh in in the field he's he's uh been a very highly regarded uh penetration tester himself uh he has a checkered pass which you can hear about on the dark net Diaries episode with him um but this is an interesting interesting example to show that this happens in real life so there was a bank that had a mobile application and Jason decided was doing a penetration test and downloaded that application and started playing around with it this particular Bank application had a functionality so apparently in America United States you guys still use checks that baffles the rest of us um but but one of the features of this was

that you could take a picture of your check and cash it you know all in the app so when Jason decompiled this he looked at how this and found that actually these images were being stored unencrypted uh on on you know in in its in the mobile device um so he kind of figured well if that's being stored unencrypted somewhere that means it's being sent unencrypted somewhere and if I can get access to that that would be great uh so what he found that there was a bit uh S3 big bucket uh S3 bucket that these were being sent to inside the app and again we're talking about it we're talking about a tier one bank here just we're

not talking about Bob's financials um we're talking about a a bank that I can't disclose because he didn't disclose to me um which was probably a good call um and then in that there was a hard-coded uh secret for that Amazon S3 bucket and then with that Jason was able to access that Amazon S3 bucket and find 10 000 images of unencrypted unencrypted inches of checks from this so this isn't something that just like happens to you know small applications this is something that's widespread uh where Secrets often end up inside our mobile applications because we honestly think that we don't understand how easy it is to do it and when you're putting something in an APK online you're

putting up your source code you should absolutely remember that so you want to make sure that your source code is basically considered open source and and expect the same security measures from that so this is just a real life example of when this has happened um uh out in the world so uh what about the what about other examples of discovering secrets on the Play Store so there's actually a few research uh that have been done on this zednet published that there was 12 000 Android apps that they found that contain secrets and cyber news last year did a great a great study on that and I'm going to shamelessly steal uh their information and talk

about that I did a webinar with cyber news we did a full hour on their research so I'm going to talk about it for five minutes if it interests you again scan at your own risk there's a QR code if you want to live dangerously uh on there on on their YouTube channel of that webinar uh that we did together but I'm just going to briefly talk about uh some of the findings that they they did so the main statistics how many applications contained hard-coded credentials of some kind about 55 so more than half now before before everyone gets too shocked by this not all secrets are the same not everything's going to give me access to

your Amazon S3 bucket there are some secrets that we discover that I would still definitely consider secrets that I wouldn't want an attacker to have but won't necessarily give you access to the kingdom but it would certainly be used as part of a a bigger a bigger attack to be able to understand and get more information but still I would consider that the the correct way would be to have zero so the fact that we have nearly 56 percent is definitely very alarming so what was the main uh the main secrets that were found so this this study uh actually only looked at the strings.xml file it didn't look it didn't scan the whole source code because that would

have taken too long for the the research project size um but we're we're going deeper into that now the number one was Google storage buckets so this is really closely in line with the example that we just saw data being sent from the app so the reason why just taking guesses of these things but the reason why I think that is because I mean you're sending data directly from the application uh to to the bucket it feels uh unnecessary to go through the back end or something like that it feels more efficient but it's a terrible idea if you're going to hard code those credentials other ones Firebase URLs uh people found lots of these these aren't necessarily uh uh

exploitable by default however it requires that you've configured your Firebase uh really well and often what we find is that things like this with like Firebase the the default is insecure the default was poorly configured why because companies want you to be able to get up and running quickly because if you can't if you have to sign your app and get all get and get lots of security working to send a mess to send your first message on Firebase then maybe you'll use a different service because it's too complicated to get started but then we don't go back and actually correct and make it more secure so lots of things like this uh Facebook app ideas lots of different

types of secrets that we've found uh on that were on here the top ad critically accreditation Cyber News found health and fitness was up there but honestly no category was immune you know we talked about the example of the financial sector so this shows that it you know really everyone is suffering uh from these programs uh almost equally so there's no kind of Avenue of that someone that's doing really well on health and fitness uh uh sucks so and just to talk about it um a little bit after we did this research I decided to have a look it was at the time where there was endless amounts of chat GPT yeah checks did it right chat GPT uh

credential uh apps popping up where they're basically marrying it so that you got chat GPT on your on your mobile application I scanned about 10 of these and I found nine of them had their open a open a uh open API Keys hard-coded directly into them so there's lots of different areas that you can find that one I found particularly shocking all right so how do you securely store your secrets on your mobile apps how do we hide these effectively so that the attacker can't actually use them so the correct answer is that you don't you don't put your secrets on your mobile application you put them on the server side um so and then people will often say

well if it's on the server side I have to connect to that server yeah you do and we just do that like any other application right you probably have some kind of login area you log into your mobile application that authenticates the user with the back end and then you can do the heavy lifting on the back end that is the only way there's no way to be able to hide your API Keys effectively if they're on uh if they're on their mobile application so only caveat to that is if the mobile application themselves is creating the keys so what I mean by that is in the example of the bank where they were they

didn't encrypt the images the app could create an encryption key store this securely on the device and the hardware that's designed for that and use it to encrypt it but we can't use API keys to connect to anything outside of that app and the other one is if you have a public application with no login well then you have a public app and you should be receiving public information nothing sensitive should be coming from that so we don't need to worry about it and please don't hard code API keys in there all right so uh here's some arguments that people always get to me uh what about encrypting keys and base64 and if you want to throw something at me for

that statement don't worry I'm coming back to it um uh what about splitting keys so this means uh what about putting keys in two different areas and then joining them together uh non-sensitive Keys application keys this is what I was talking about with encryption and what about if I obfuscate my code so no one can understand so first one base64 is not encryption it's encoding and it's completely useless for security the tool that I use GG Shield automatically unencode space64 so this is absolutely no level of security and please don't call it encrypted encryption uh splitting keys I'll give you that splitting Keys makes it slightly harder for the attacker in the sense that I

can't just use a g something like GG Shield to scan it because of the keys are split it won't know that they're partial keys but if you're relying on slightly harder as your level of security then you've probably got a big problem um and I just suggest doing it anyway non-sensitive Keys uh yeah uh I agree that there are some keys that don't give me complete access however they tell a story they let you know information I'm all as an attacker I'm always wanting more and more information uh to try and put it together so I can coordinate it uh in in an attack that that I can effectively use so having any kind of keys in there and it also means

that uh it gives me opportunity that if you've done something wrong if you have misconfigured something on your end uh then that's opening the door for me in lots of ways application keys this is acceptable as long as you store them in the key store or somewhere similar and code authorization it's useless to maybe slightly harder but it doesn't really matter so this is what like code notification looks like there's some options you know in the Android manifest you can you can select Minify this kind of makes the code less human readable but it wouldn't it wouldn't stop me using a scanning tool to be able to find these secrets um it just makes it harder for me as a

human to look through and understand what the secrets do this is the tip of the iceberg there's lots of examples where you can do to make code obfuscated but really uh don't waste your energy on trying to make it harder for someone to understand put your energy into making it secure uh so some things that you can actually do so signing your application so code signing you know is basically proving that this application in its current state hasn't been modified that the author is correct and that and you need those signatures to be able to authenticate with services this isn't in replace of storing them securely on your back end this is another level of

security that you should absolutely have it means that if you make a mistake and either attack or find your API keys I can't simply use my machine to manipulate them and extract information limit your IP addresses to know and machine so if you have a service that's only meant to connect to the seller service make sure it can only connect to that other service limit your API keys to minimal scope uh so the amount of times that I will find read write access or admin keys for uh for a job where someone's just trying to ingest information so you're just trying to pull information read information from an Amazon S3 bucket and you've given me

admin keys to do that the some people argue that this is more secure because you have less keys to manage but that's totally uh excuse you're just lazy so please make sure you you if you need to create more keys create more keys to limit their IP uh and then set up access rules so what I mean by access rules is in things like uh Firebase you can set up security rules to be able to understand exactly uh uh what people what people can do and restrict the limits that you can so we we should be doing all of these none of this is to replace storing the key securely that's kind of number one but

what happened here is none of these companies none of these apps that I've scanned think that they have secrets in their mobile applications no one's doing it really on purpose it happens because people don't understand the technology and how it's all compiled um so we need to add additional layers of levels of security to catch it when the mistakes happen when not if when the mistakes happen because they absolutely uh definitely will happen uh and there's a couple more things uh that we can do to prevent uh our secrets from from leaking so the number one here is a change in mindset and that is kind of assume that you're going to be breached assume that your source code is

is public uh if you adopt the attitude that an attacker is going to get into your get into your application get into your source code then you're going to change your mindset about how you handle things in there for me this is really one of the the number one rules that we should absolutely be dealing with is kind of changing this mindset so that we're assuming that we're going to be breached we know our source code is going to be looked at from malicious things and if we put that head on then it becomes a lot easier to be able to secure it and we know a lot more about how to do that uh use automated sequence detection so

mistakes happen Secrets end up out there in the wild what you can do is use tools like GG Shield to scan your applications after they've been compiled to scan your code repositories and make sure no secrets are in there as an attacker I'm always going to be using automated sequence detection there's heaps of bug Bounty tools out there like get leaks like travel hog that we can that we can use as attackers to to find secrets so we as security as blue teamers need to adopt the same approaches when it comes to identifying them and preventing them from leaking out in the first place uh restrict access we've talked about this you know uh whitelist Services

short-lived credentials uh one of my biggest gripes is people not rotating credentials if you're using uh Secrets make sure you you have rotation policies in place this does two things one it means if a key gets leaked then it shouldn't have a long life span so it's not valid for long and two and if you're regularly rotating Keys it means you know how to do it so if a key gets leaked you know what that key is for and you know how to rotate it the amount of times that someone will find a key and no one has any idea what that gives access to and you don't know if revoking that key is going to shut down and

Destroy production so you kind of just ignore it and hope the problem goes away but if we have a rotation policy in place then we can regularly rotate these keys so that they're not a problem uh use a honey tokens is one of my favorite things at the moment is just using fake credentials to identify when when you're being attacked um and give give the attackers something to to to chew on honey pots are fantastic ways lines of Defense ensure your secrets are always server-side don't put any secrets in your mobile applications even if you think that they don't do anything and the other one is uh we find a lot of these because there's misconfigurations you know

misconfigurations in your Firebase misconfigurations and your S3 buckets um we absolutely want to try and reduce that use infrastructure as code because it means it's replicatable and then use infrastructure as code scanning to try and identify these misconfigurations I know some of these are a little bit outside of the talk today but this is just kind of uh uh my opinion of kind of things that we absolutely should should do so I'm a little bit early uh but thanks I have a if you found this talk interesting I actually have another talk tomorrow at five to five forty five um where we're going to be looking at how we can explore lots of things like

how we can do basically the same thing there'll be lots of demos of how we can abuse Docker images and package managers um find misconfigurations and find secrets so if you're interested in this topic I'll be speaking again tomorrow in this room at 5 to 5 45 and with that I would like to thank everyone for paying attention and being here with me today it's been really really cool for me to be able to speak at besides Las Vegas one of the one of the goals I had so thanks everyone for being here and making it possible [Applause] thank you and uh if anyone has any questions uh preferably easy ones then please let me

know yeah just a second please oh sorry sorry that's my fault uh I'm I'm actually kind of like behind on like the Mac OS operating system can you just download iOS apps to Max these days no you can't you have to use an intermediary so if you're like uh if I go back uh I use the tool here called IPA tool um so that's how so that's how you do it but um it can be tricky to kind of like use these things if you just want to do some experiments there are Play Store and App Store mirrors that do allow you to download it just be careful because about 80 of them are super dodgy

delivering malware to you um but there are there are some legitimate ones too yeah yes uh your proposed solution is effectively backing for front end it's effectively backing for front end yeah yeah and so it puts a lot of um onus on the user to make sure they're using the correct application right so you could have a forged app that tricks the user into presenting their credentials to the back end and then the back end gives the forged app the credentials right yeah but I think that's I mean that's an issue like a forged application is an issue for regardless of whether you're you're doing this because I mean how have you go to download a mobile application you

need to authenticate somewhere so you're going to be uh you know asking your user for credentials to log in um so that I mean that does put the onus on the user to make sure that they're using the legitimate application uh but this is but this is absolutely an issue that we're facing uh regardless I don't think that forcing it to go back end changes that um because if you don't need to log in to an application then what your back end can do is send your your data so you've got a public ad you just open it up and you can do stuff you can still fetch public data so like it might even

just be an RSS that your backend is producing so that you can download it if you don't need to to log in but by putting the credentials in the front in the front of that you know you're you're you're asking for trouble of people being able to manipulate that yeah yeah over there I think it's more for the the videos yeah so uh would you suggest there's any value at all at putting either certificates in like an mtls type situation or cert pinning situation or an API key in an APK or app at all or is because of the the ability to pull them out so easily there's really no value there I would say it's absolutely that

you know value it's the same as if you know I think you just assume that the the source code like if you can you put your source code in a public GitHub repository if the answer is no then that's not uh at a hygiene level that you should be creating an APK from from from that um now there's lots of people that will say things like what about uh whether uh like Maps Google Maps API keys or you know weather API keys or something that really offers no value for what I can do and the answer to that is like these things still charge now I'm not going to be able to get into your infrastructure

because of a weather API key or a Google Maps API key these are the ones that we find everywhere uh but you have given me a button that if I press this button I'm going to charge you one cent or whatever 0.01 right now if I really dislike you I'm going to push that button a lot and I'm going to automate that pushing of the button so that you get a big build so it's still not like okay I'm not going to be able to shut down your application but there's still lots of things that you can do to be able to do it just to be honest for your competitors to run up bills for you make

you unsustainable I mean even a thousand dollar Maps API bill is is going to be very suspicious and kind of painful to pay so I think there's no value people always argue with me about like certain types of keys and and whether or not you've configured it so that nothing but your app can use it and I agree that all that's good but you're relying on the fact that everything has to be perfect all the time your infrastructure has to be perfect 100 of the time I don't like the saying that attackers don't need to be right once because truthfully they need to be right lots of times to get there but you know it is kind of true in this

sense that you're you're you're one mistake away from from that key being useful to me and as an attacker you know especially if I'm malicious I can do stuff with it so I'd say absolutely no value you can argue what you want but I don't think my position will change yeah just wait one second for the for the mic yeah how about storing the mtls client keys in the app in the world like you mentioned Secrets manager yeah so is that uh standard practice you know like putting putting Secrets see there's different levels the talk after me is going to is going to discuss sequence maturity model uh like different levels of what to do and

there's different levels to it um now I I differ from the the General Security opinion of this so if we look at the very highest level we use hashicot Vault we're using Dynamic secrets so secrets are being generated and destroyed after one-time use all our secrets are centralized there that's the best practice but is that reasonable for a small company to be able to manage a heavy tool like hashico volt and that so then you go down a level um and I say like no no if you because the chances are you're not going to use it properly and you develop it you're you're not there yet in your maturity so you shouldn't try and force this crazy

tool on them if you go down and you have secrets managers if you're using AWS or gcp they have address managers they're less secure with less features but they do the job you know and and so I'm I'm an advocate for be reasonable with what you want okay aim for the vault have a path to get there um but if that is what you need because the argument that I have is because people always argue with this I'm all about finding secrets in weird and wacky and wonderful places so um you're arguing should I use Vault I'm saying just don't put them in your damn APK like you know like let's start there don't put them in your source code we

can talk Vault all you want but we're not at the stage where we can you know so just use whatever you're going to use vault is the best or other there's other services out there like one password has some good ones or they've got some new developer tools Doppler a keyless lots of tools out there that do great great things to managing Secrets but just pick the right tool and actually use it and just make sure you're not hard-coding Secrets they start there and then we'll we'll work it worry about the rest here hopefully that answers your question

well Matt this one's going to be hard I can feel it no um I'm really kind of curious um where areas of do you see for improvement for that secret scanning aspect of it because the last things look like passwords uh so um how do you see you know areas that need additional research to make your job easier to find these secrets yeah so I mean secret scan is actually getting better and better uh all the time I think a few years ago secret scanners were kind of creating lots and lots of false positives I mean Edgar getting we've put lots of work into reducing there how we do it for it's easy for secrets that you know it's

like an AWS key or twilio key because I can check with the service hey is this real yes or no yes okay good send me a second cast a wide net it gets tricky when you're talking about generic Secrets secret setup for systems that you've built that I have no idea exist and so how we've managed to do that is we've we've created flags and post validation so we find a high entry string looks like a secret what else clues in this you know do we know can we understand what this code is doing to flag it and we've been doing that but in all honesty I think that sequence detection and high quality Secrets

detection over the next four years is going to become an absolute given like you you won't get away with having crappy uh secret protection AI is helping in this to be on a digest set so we have a record of uh everything that's happened on GitHub for the last seven years publicly so now with some machine learning and AI we can really start training it I think that the detection is going to get as good as it possibly can quickly um but I think where the problem comes in particularly as large Enterprises is not so much the detection of it it's more like how do you actually effectively remediate this issue you've got a thousand Secrets you've got two

appsec Engineers like you know like how like what are they going to do all year all they're doing is rotating secrets and investigating them so to me it's all about streamlining that remediation process and improving that area so that we actually know what to do and then not just putting sequence detection in like your source code or code deposit trees put it everywhere along the along the way do get hooks to prevent them getting in the code repositories do scans after your application has been compiled uh all of that kind of area so I mean there's lots that can be improved I think detection itself is getting better and better across the broad not just

with us but everyone even the open source tools are getting really good so I feel like that's going to be a given which hopefully will help but where it's going to get complicated is what what you do after you've found them how do you actually deal with that

any other questions everybody let's give it up for McKenzie thanks everyone [Applause]

I think Jessica [Music] thank you [Music] foreign [Music] [Music] thank you [Music] foreign [Music]

[Music]

[Music] foreign [Music]

[Music] [Music] thank you [Music] thank you [Music]

[Music]

[Music] thank you [Music] foreign [Music] foreign [Music] foreign [Music] thank you foreign [Music]

[Music] [Applause] [Music]

[Music] thank you [Music] thank you [Music] [Applause]

[Music]

[Music] thank you [Music]

baby [Music]

appetite don't leave me alone [Music]

[Music] good afternoon welcome back welcome to password con room let's get excited for our next speaker Dwayne McDaniel who's going to be talking about do you know where your secrets are exploring the problem of secret sprawl and secret management maturity let's go sponsors we want to thank you especially our Diamond sponsor Adobe shout out to our gold sponsor blue cat Plex track and Toyota is their support or other donors and you all which make this possible please turn off your cell phones nobody wants to hear your Bone Thugs-N-Harmony ringtone okay if you have any questions save it to the end without further Ado Dwayne what's up hey thanks very much for that intro um if you don't want to turn off your

cell phones uh just turn off the ringtone but yeah please take pictures if you want and post them on the Internet please thanks for all coming this afternoon I hope everybody is everybody having a good B-side so far awesome it's so so apparently um so I'll before I get my intro I have a prize hat up here I'm asking specific questions who saw the last session in here all right you guys are going to get a chance to win some prizes from me based on what you heard before uh because I work with Mac um so I live in Chicago I've been a developer Advocate since 2016. they're about uh depending on how you count

developer advocacy in the early days but mostly come from the devops space the platform play Space security this is my first b-sides Las Vegas so thank you all for having me I'm very nervous to be here uh it's the biggest imposter syndrome I've had this year this is like my fourth b-sides so you guys are all special um if you want to hit me up on X or Twitter or whatever they're calling it these days it's MC Dwayne and it will be until they take the service down I'm also on Insta and all the other places MC Dwayne uh including GitHub I should probably put that on here because that's a more reliable platform than all of

them and hopefully it's more reliable and hit me up about anything if I'm talking about it today or if you want to talk about karaoke which who's going to karaoke tonight okay it'll be me singing by myself um but if you want to come hear me sing by myself or you want to talk about rock and roll uh you want to see a rock and roll show please hit me up I'm always happy to talk about rock and roll um very quickly I work for get Guardian they're going to come up a lot in this talk because a lot of the research was done by them in fact all of the research because that's where I work

that's just how it works um so we have a booth over there we can talk to you about hard-coded secrets and just checking how they leak and all that um but that'll sleeve that for the hallway over there so in summary just don't leak your credentials and you're good and you can go home and sleep well at night you've done it success unfortunately I have to keep giving this talk and other talks like it uh so two-thirds of this talk is just going to be a bunch of facts and figures that might be terrifying especially if you weren't aware of them before and some people will just nod along and say hey I've heard this before

I've heard things like this before specifically the guy that talked right before me Mackenzie my colleague who's also a developer Advocate at get guardian we didn't plan that that's how the schedule worked out but if you do feel panicked or fear like you're talking about my company uh just recite the litany against fear and everything will be okay after all we're talking about the internet nobody really dies on the internet do they uh that's maybe a not a great joke but uh Uber they're still out there they still exist but this happened to them uh anybody remember this from 2002 or 22 um so super admin gets fished and they're doing it right they got two MFA

on or two backer authentication on and the theory is he got flooded with so many requests that his thumb slipped we'll never know maybe he just got mad at Uber and said okay I'm just gonna let this one through anyway the request went through uh the spammer or the attacker uh who was from the lapsis group which is a UK based hacking group um got in was able to take that Superman's credentials go through the VPN long story short found Powershell scripts Shock full of hard-coded credentials for everything and he got into everything including their hacker One account and told hacker one hey I hacked into Uber I'm cool and they're like yeah this is that's

probably not true so he floods there uh their slack with memes and takes over all the channels and they're like yeah this is some prankster the next person to talk to is the New York Times that's why we know about the story what did he actually steal we'll never know did we see a wall of new security job postings from Uber the next day after that sir yes we do we did um Circle CI who got affected by this who had stuff in production for this is anybody here a developer does anybody use Circle CI wow that's the that's a first that is literally a first for me given this talk that or this bringing this up that no

one nobody got hit um basically the kerning messed up and I'm really mad about that because that was fine 10 minutes ago anyhow um long story short attacker hits a remote developer remote developers environment gets compromised attacker gets into the circle CI internal framework and internal platforms plants malware that start stealing credentials that then takes those credentials and gets into customer accounts the same day that they come out and say hey we had to rotate all of the access Keys sorry if it took down your production instance but we had to rotate them uh same day a uh independent researcher said hey something's going on Circle CI because all my honey tokens went off and now we know why someone actually got

a hold of their credentials and was attempting to exploit them I don't have fancy graphics for everything else uh it was one of my favorite stories uh not in a good way but just I think it's interesting uh anybody I drive a Toyota do you have the T Connect app yeah that's the right answer uh Toyota back uh in 2017 subcontracted someone to do maintenance on it good idea in general for some reason that subcontractor pushed a portion of the t-connect code onto a public GitHub repo why we'll never know it takes the security researcher until 2022 to figure out that that contains an actual data key that affected 296 000 customers nothing super sensitive so email

accounts and some identifying information but no my credit cards so Toyota Japan puts out a really nice blog post and statement saying hey you're probably going to get fished be on the lookout it might not be us just be aware but for five years it took him five years to figure that out uh AstraZeneca all we know are the very scant details that got released but it was The Perfect Storm Tempest in a teacup kind of situation where a developer pushes the test environment credentials into GitHub public no big deal it's test credential like what what's on Earth can happen in a test environment well somebody else inside AstraZeneca pushed actual customer data into the test environment

we don't know how big this was we just know what happened because it's HIPAA they can't just come out and say this many customers um and they didn't unlike Airline crashes uh that come out immediately and say hey we got a crash we are going to fix it and here's exactly what went wrong and here's exactly the steps in security we all just kind of get ah we'll make best effort sorry and move on this one's kind of scary to me because I am an astronautical customer we live in this world of constant cat and mouse uh literally people trying to get in and get our data and our Hardware resources our machine resources Alfred the creator of cyber cop the

first commercial Honeypot that ever got released um said this and I think it's a great a great very nice Summary of why we should take this stuff seriously I think it could be reduced to this attackers and this this is supported by like Verizon dbir report if you haven't read the dbir this year go read it it's a great chock full of things ransomware is kind of ebbing and flowing and it turns out that log4j we got mitigated and it's kind of not it's still a problem but anyway dbr is full of good info but 80 plus percent of all attacks are organized crime at this point of malware attacks um yeah there are nation state actors

like we all know about Microsoft and their forged key that happened last month um those are the China backed hacking group attacked them and guy was trying to get in the State Department of Commerce Department that stuff happens there are hacktivists that just want to make companies look bad like the kid from knives this he was 19 so I say kid um young man from lapsis uh who did it for the walls apparently and just to make Uber look bad publicly but most people are after the top two anybody you know want to guess this isn't for the prize Hat by the way uh but anybody want to guess what number one why they want

your Machinery sources why do they want your data money but and how do they get your money ransomware my favorite joke I heard all of last year was how did the hacker evade the FBI they ran somewhere yeah thank you thank you that's the exact right response thank you all um the one thing all of those attacks have in common that I went through and again I'm not trying to scare you all this is a reality is that there was a credential somewhere along the path that gave the attacker access to other things or a credential that was misplaced because it was hard-coded when it shouldn't have been hard-coded attacker is going back oh for here for a

second what we see them commonly do to get these things is lateral slides lateral movement through the environment trying any door they can get their hands on and then escalate privileges up that's take over the machines escalate privilege as high as we can and then go see what else we can get into and escalate those privileges lateral slide escalate up time and time and time and time again but because because of that we kind of know how to defend for that or we should and that's what we get to in the later part of this but when it's just a quick definition we I use the word secret and credential interchangeably because they mean the

same thing to me in my head but this is what I mean by it it is an API key a username anything that gives you access to another system or decrypts data that's it like if if it's an EMV file a CRT or what have you an actual SSH key something that should be secret you should not be sharing that out in the world this is how it commonly ends out in the world and I do this all the time anybody want to guess why we do this it's easy and when do you need to do something easy because you're in the middle of a rush testing and debugging like I just need to make sure this

credential works is this the right password and if you do it locally and you take it right back out of your local file who cares that that's how we debug things that's what we do in life the problem is then we add commit and push and that's where the problem really escalates to the point of we put out this report every year feel free to download it if you want that's the first big third of this this talk is just going through these um so we as a platform since 2017 have looked at every single public git commit you can as well if you want api.github.com timeline or maybe events I always get the too confused but

there's a public API you can just go subscribe to it's in Json it's over 6 billion 60 billion commits deep it's insane um last year alone there were over a billion commits that we scanned again just public GitHub so not public reflet or gitlab public or anything like that and we're seeing a giant growth in GitHub year over year uh 27 increase in developers just in 2022 um at the time the fastest growing language we were seeing was HCL uh anybody know what that goes to terraform yeah that's great you guys are awake I'm loving that this is just far enough after lunch that you're still awake and the coffee's kicked in from the Three O'Clock Coffee Rush all right

first question from the prize hat and probably heard the stat if you were in the last talk who uh who knows how many secrets were discovered last year in public GitHub by get Guardian who said that uh you get a piece of candy there you go all right yeah 10 million uh that's a terrifying number and that's a fun way to deliver it uh yeah we found 10 million hard-coded credentials this was a 67 increase over the previous year and I have a slide that lays that out but this isn't cumulative this isn't over all of time this is just new ones out of that how many billion 1.27 billion commits how we did it that's public as well we

built our own detector engines but we documented it all out there and that's we can go look at the actual how we did it so they're just saying it's all supported out there all right second question on average how many commits per thousand commits contain a secret on these hands now because I can't just hear them all the way back what 100 a little high anybody else 10 a little bit a little bit High still anybody else 56.9 that's a great specific but no anybody anybody else I will give it Price is Right rules over there I'm not going to throw it but 5.5 you get an Apple uh you can you can come up and get it later or if you want to

pass this back here here I'll throw it to you and you you can pass that back to them you can an apple uh 5.5 out of every thousand commits on GitHub public last year contained a credential hardcoded credential that means one out of every 10 authors pushed one you can see the previous years like there was a little over six million we discovered in 2021 uh over almost 3 million we discovered in 2020 and again this isn't a cumulative number this is brand new ads and when we stop and think about like the fact that GitHub itself only grew 27 percent this is startling so who's doing this stuff turns out it's everybody it's not can't

just be new people yep that's the answer um so this is where they're coming from and I don't want to point fingers but it kind of makes sense when you think about General populations but does anyone see a weird missing Nation off of this list anybody any security researchers in here that investigate um where origins of attacks come from Rush is all the way at the bottom it's on the list but there's one missing oh yeah they should definitely be on there as well but I think that's tied to the other answer uh so North Korea isn't doing public GitHub that's the other reason they're not on here um Indonesia so go look at where attacks are coming

from and you're gonna Surprise by your logs you start tracing IP addresses at least my research I found like a shocking number of like Indonesia and like why are they coming from Indonesia and I've talked to other people about this and there's no clear answer maybe they're trampolining maybe North Korea is using Indonesia as an attack Vector all we know is that Indonesia isn't leaking their secrets onto public GitHub that's what this data tells us anyway moving on um so if we look at specific categories of oh wait that was supposed to be one of my question slides wasn't it oh no oh I moved it around sorry that was supposed to be a question slide

um so if we look at the how which specific Secrets got shared out there what types um other is the biggest category because other is literally the bucket for all of the other like two percent five percent seven percent that weren't big enough to show up at 3.8 so let's let's ignore the other for a second and I should have just taken that out but data storage is number one Cloud providers number two which totally tracks to what we originally talked about attackers are only after two things and they're disgusting uh because they want your money messaging systems why do you think messaging systems uh are important what off others

yeah um I think when I heard between those and well the answer I'm looking for was flat it out um is there's a lot of companies that think okay I'm just going to pass this key between developers through the system it's better than email right slack yeah it's slightly better in email anyway we still look at generic detectors as well so we have specific sectors look at like specific providers and then General generic detectors that look for things that look like hydrogen passwords things that are Basics before encoded Bearer tokens things like that generic passwords is the vast majority of that now I'm just getting into stats and figures on it but all right we'll go

back to the specifics who thinks they know the most commonly leaked specific Secret in 2022 I I would thought so too but it wasn't anybody else what oh sorry he said AWS any other guesses nope no they're any other big providers out there anybody want to guess who's that Google you get a marker that if you spill something on yourself it'll wait it out on a Titan pen um uh yeah Google API Keys nine point seven percent of all the 10 billion or 10 million we detected were Google API keys uh you can see the rest of it splits out again other skews everything and it's weird and the colors choices were not mine uh when we built this but you can

see like that's how it spreads out all right this is I can talk about stats and figures all day but the what I want to get to is later here uh what's the fastest growing leak secret we've seen in 2023 technically open API but I'll give it to you you get progress software kleenexes I didn't just clean out my bag to do this I swear I had candy in it and didn't eat all the candy um but yeah uh this so far this year we found over 50 000 incidents open AI Keys it's not the largest number but it's the fastest growing considering last year there were basically none um it was not even detectable last year

I went and dug in the stats and I forget it but it's a very small number that's in the other category all right but what does this mean for your Enterprise these are all interesting stats sure and for you as developers to think about but what does this mean for the companies you work for well we went out and asked 107 people that said bitter decision makers in I.T and we put together a report what they said and the voice of the practitioners state secret and appsec it's a really long title um all right only got two more of these I promise um what percentage of it leaders said they experienced a secret leak in the

last 18 months a little bit lower it's Price Right Rules by the way 100 I think that's more accurate no what did you any other guesses 26 higher 36. that's a good guess too 75 on the nose good you get some notepads that say Adobe on them um 75 percent of respondents said in the last 18 months they had experienced a secret leak 60 of those or six percent of all people said it somehow impacted their company and employees that's the reality and that's why we need to get on top of this and solve it because we're starting to really impact the bottom line for our companies um 94 said they're planning to improve their secret practices the next 12 to 18

months this is my ultimate problem with survey based research is people can say a lot of interesting things and this is proof we this is select all that apply uh anybody see a problem with the math on this like immediately yeah because all these it leaders like yeah I'm probably gonna fix it um yeah there's probably dealing with this yeah but we're all we're all dealing with budgets only so much time in the day but everybody's got this plan that one day we're gonna fix it one day we're gonna do something about it we know it's a problem and that's what I keep harping on um for the people that did experience and I know I went through that real fast I'll

leave this up here for a second longer um for the people that did experience that 27 admitted that they were relying only on manual code reviews who thinks that's a good idea yeah I've never had anyone who use their hand on that I love manual code reviews I think there's a serious value to the manual code review how else do you teach Junior devs anti-patterns how else do you teach them this is not winding up with the business value of our company that's what code reviews are great at the human to human connection pair programming that's that's get overall Better Together finding an individual string that's wrong man humans are terrible at that that's what machines are great at

um so for people that did report they had a leak how many this is their self-reported number average number of occurrences and by occurrence I mean if it's the secret is shared within a repo how many places in the repo did it appear and any other guesses Price is Right rules it is three point 3.95 last piece of candy ready who's I thought I said eight you said three all right that's the last piece of candy and that is the end of the prize hat for now I got one more thing but I don't have any more questions all right um so uh done with the prize hat now we're done with the silly Part

um I did that to kind of because this is all terrifying news right like another student about you could sleep well at night um so what do we do about it and that's where I want to end with the last 10 15 minutes who here is familiar with Dora metrics Dora metrics the uh accelerate anybody here read accelerate okay there's a book called accelerate uh and the research they did in there uh formed the devops research um I don't know what the a stands for not Association but um basically this is how you measure the success of a devops organization actually let's just real quick Dora metrics tricks nah tricks yeah how to measure software delivery so

you can go to well this is not the one I was looking for because the Google has the official one there we go Google Cloud um but we can take how all devops organizations perform and break it down into these four leading metrics uh deployment frequency assessment that's what it is research devops research and assessment I never went with the as4 the point of frequency how often an organization successfully releases to production lead time for changes how fast they get a change out the door change failure rate the percentage of deployments that cause a failure and the time to restore service if something breaks how long does it take to fix it you can take just these four metrics and

judge the overall health of the devops organization so says their research which is also survey based but half the book of accelerate explains why that works Nicole Ferguson um and other people wrote this Gene Kim a lot of really smart people so go read accelerate so we took that idea of like well how do we map out what a successful organization looks like and we said this is wrong this isn't good enough this is what Google's entire developer documentation says if you leak a secret simply just go re-architecture app and say good luck I don't recall saying good luck uh this isn't good this doesn't tell you what you need to know so we think that it starts with hey use a

vault cool uh either like something off the shelf like Ashley Court vaults or if you're using Azure Azure key Vault or AWS Secrets manager or what have you that's a good start but that's a start so how do we map out how we get to success and I think it comes down to really three pillars you gotta have people invested in the process that trained and they know that hey this is a problem we need to fix we got to stay on top of this you got to have clear processes that are documented that show this is what we should be doing and here's exactly how we do it and then you rely on the tools because

if you just throw developers key vault you're not going to have success key vault is good but it's not something you do on an individual basis if you're working in a team it's a team effort same thing with hashtag Court Vault yeah you can use hatch Court Vault completely on your own by yourself as a developer but if you don't Implement as a team you're not going to find the results you want and if you try to automate things completely on your own without having the processes in place everything's going to fall apart it's the three leg problem remove one of the pillars everything falls apart but anyway we think it boils down to five levels and of course computer

science we're off by one so start with zero uh and this isn't never meant to be an accusation so never feel like hey you have to be better this is our general guide of like here's how we think teams can improve over time and a general road map for you so just a place to go gauge where you are as an org and then how do you have that conversation with your devops leads and your operations team and your csos and your CTO say here's where we think we need to get to based on where we're at at the moment so level zero is a lot of organizations that's why we put at the base of the

pyramid on the secret management side they're hard-coded credentials and we already went through all the rigmarole in the first part of this talk of why that's bad it's just everything's unencrypted you're throwing things just willy-nilly into your code base throwing it through your CI environments and there's a password in place somewhere that's the hope for the best or the best answer you can have of what are you doing for security oh we have a firewall that's not good enough and then the secret detection side how do you know if people are leaking credentials here and there you don't you just don't you're taking the word for it you're not even doing manual code reviews at this point again not an

accusation just this is where we see like the very bottom level if you're starting a project and you've never thought about this stuff it's probably where you're at and then you get people that are starting to think hey maybe maybe I need to use an EnV file because if I store my EnV file outside of the repo that's what AWS tells me to do with the 8.80. AWS credentials file that totally works and it does problem is sometimes people move that into the repo and then they forget to set get ignore or they modify get ignore that it allows it to be shared or they copy paste the entire repo or the entire folder and the EnV folder into a

bucket last year there was a firm that found 1.5 million git repos that contain secrets in public S3 buckets because people just threw them there because they can store them out there for cheap digital or does not work outside of git servers um but okay but we're all moving the right direction let's take um our our config files uh and let's group them together let's make sure we're storing things externally from our repos and you're starting to think like okay maybe we shouldn't hard code that's a good first step um oh yeah and log data we finally realized that your plain text credentials are in our log data maybe we shouldn't do that maybe we should build

sanitizers in there please um but Secrets aren't really scoped like everything is just fully allowed to do everything because that's fast and easy and now you're starting to periodically look through and say like okay manually did we did we check for this and secrets are in fact rotated manually sometimes when you remember it when you think it's a good idea the alarm goes off hey time to rotate that AWS key from last year again better than where you were before you're on the right path that's the conversations you should be having it's like how can we do this better if you have include one of these one percent every day you're gonna get to the next level pretty quick it's

intermediate people now we start using vaults now we start using secret managers now we're starting to scope things correctly so if someone does get a hold of it the only thing it can do is a very specific limited thing that you cannot escalate it doesn't allow escalation uh showing things dynamically uh our nose you're loading the secrets in dynamically from the vaults correctly everywhere so if someone does get to your code they see vault.projectname.secretname.secret that's programmatically calling it in that's that's awesome like if I was an attacker I'd think oh man I don't got time for this I've got to figure out some other way in you're starting to scan things at the developer level automatically and you're

starting to uh you scan continuously at the pr process the merge process the pr the merge request process that every time the code's gonna get merged into your main branch yeah we've we're scanning to that point and trying to look at your build outputs to see if any secrets got jammed into there so scan your Docker images your Pi Pi packages and that rotation is still manual but uh they're rotated at least at least periodically you can't see where this is all going and I'm starting to run out of time and I do want to leave time for questions but uh level three is okay now we're gonna start doing this for real everybody is going to use vault

all right there is a clearly defined process around how to use it and rotation is clearly defined and it's a regular scheduled event and we know when it's going to happen now we start actively monitoring logs because we know that hey this stuff's going to get leaked let's start actively looking for it on the regular with some tools that let us do that datadog log IO logs IO they're all there's not one winner here there's a bunch of ways to do this uh and now developers starting to use tools to scan before they push this is where a trekolog comes in git leaks we have GG Shield there's if you're just looking for Pure detectors there's a lot of solutions out

there on the open source AWS Secrets uh our AWS Labs get Secrets is another good one if you're all in on AWS um but you're starting to build that into your git hooks system so every time you go to make a commit automatically check the remediation process involves the developer now you're getting asked like why is this here and you start getting into uh holding a developer's account for why they're doing that maybe it's a test credential maybe there's nothing to worry about maybe you're maybe they did it by accident maybe they didn't know maybe they weren't trained maybe they didn't get the notification maybe they didn't read the memo we can start fixing the people and the

processes and the tools all together all right and then we get to our expert level no matter where you are here this is where we think you can get to because we know companies that are at this level we work with them day in Day Out there's a central Vault Storage or hardcore um hardcore Hardware managed device storage that has complete clear access logs you know exactly who's called what Secrets when who changed things who was in the system at all it's all clearly logged and actively monitored secrets are Dynamic meaning they change constantly the best passwords are the ones that do not exist and we've gone to a system that is relying on IAM rules uh wherever

possible and the only passwords are the ones that are mandatory but those are very short-lived so someone gets a hold of it they've probably already expired the one they have the hold on um you're starting you're checking throughout the entire pipeline at every step that's what I'm saying the open ID connect you're starting to replace um passwords with tokens short-lived tokens um yeah and you're enforcing things so that's the the process and the people part like hey you didn't do this right thank you um how how do we fix that how do we as a team make this better now you're scanning every single commit every single time it moves even before it's made consistently

across the board everybody does it if not and it still shows up there are consequences to that uh not be a heavy-handed but it's hey why are you pushing Secrets continually we told you not to do that here's all the training here is exactly the tools now it's a you problem a specific person problem and monitoring is ongoing continual and secret automation or Secrets rotation is completely automatic has anyone ever flipped is anyone here flipped on AWS automatic rotation or I guess AWS users in the room how many of you use the secret rotation automation it's theirs in the documentation it's one config right on please be proud um with all the other tools you can do it

there are systematic ways to go about this so again you can download this but this is what we would love for everybody to go have that conversation with their teams like hey we've we've here at a level one but we're almost to level two what why don't we talk about processes to get to that to get to here we want this to be a road map of where you can find success not a judgment not a we should feel bad nobody should feel bad it's a journey nobody's born knowing this stuff not even Linus so in conclusion it's a cost of cat Mouse game and the stakes are very real I was talking to a colleague

who said their company leaked and AWS credential they caught it within five minutes they got a bill for 150 000. because they caught it they pulled it out of the code but it still took a while for them to rotate it that's the stakes we're playing with now uh open AI there's a dark reading article I didn't quote it in here but actually I did quote it in here uh but it's uh reflect um reflet has does not have any automated checking for their code uh if you leak open AI secret onto GitHub public there's a deal between GitHub and open AI that will invalidate that they'll send in a validation request immediately and you're safe

you leak that same credential on replit it will live until someone tells takes it down like you or then send your own a validation request uh yeah somebody got a bill for 110 000 dollars because a hacker group founded on replit and passed it around in Discord and everybody got a free AI that day but that's the stakes we're playing against so again don't make anybody feel bad about this stuff and I don't really want to scare anyone but bring it to your attention have the conversations internally I'm Dwayne I live in Chicago I've been developer Advocates it's about 2016. hit me up on Twitter about anything and if you want to sing some karaoke or talk about rock and roll I'll

be around wrong direction thanks [Applause] and with that open up any questions and we don't have a mic because we're on the internet right there

hi great talk um you mentioned you scan all of public GitHub and you find these credentials I imagine the vast majority of people putting secrets into public GitHub uh small developers small companies people that aren't going to have tremendous amounts of professional experience you you also scan private repos as well do the metrics line up uh sorry is there a difference in the like the ratio of the metrics from like professional developers in Enterprise versus what you find on public GitHub is it comparable or is it this actually really interesting to bring that up um because we just are talking about that internally what I can share publicly because again we're dealing with private repos and

customers and ndas and all that what I can share is uh we have severity scoring on it's automatic on the system so if it's valid and it's public um it automatically it's a critical score uh we do see a a curve um I can't say exactly the numbers but we do see a curve downward of um the more professional the organization the higher they are on that list but I think that's uh the the more that that curve so it's um less severe less criticality from those teams but that's a result of their maturity in our opinion that's what we think is actually happening is like yeah they're consistently using hash Court vaults so they're leaking a lot fewer

secrets that hit that criticality curve so it's a combination of those things so it's not like one thing I can point to it's like oh there's just Juniors or it's a brand new company um but yeah I can't I don't actually can't release those numbers on the individual but that's a good question but yeah we do see people that are checking consistently are higher on the elevation scale and there's nothing controversial or a secret about that any other questions I did have one other prize and since you asked the first question it's a chord minder sorry almost hit you in the head didn't mean to do that there you go reminder so you mentioned about a manual code review and

um oh right there yep yep okay so you mentioned about manual code review and it's kind of you know okay but not not that good but uh any automated tools that can actually scan this and give us some more detailed uh um uh input on the on the code there's a number of providers out there full disclosure get Guardians one of them that looks for hard-coded Secrets uh truffle security is one another one I would name off the top of my head uh if you are all in on GitHub GitHub Advanced security is pretty darn good for what it is um but it only works on GitHub it's like the the other thing on there thanks very

much for coming um uh but yeah there are other providers out there and then there's the whole world of sa St on static code analysis um static application code testing s-a-c-t s-a-t I'm getting acronyms because I'm really at the end of a talk uh but yeah then there's a static code analysis tools out there and those range all over from sneak to um I just there's a ton of them out there sneak comes to mind first when I the ones I always think of when I think automatic code review um but yeah if it's a combination if you use test I believe you use testing Tools in conjunction with humans that's the best outcome because manual code reviews

again how do we improve the business logic how do we teach Juniors not to code anti-patterns manual code reviews bus for that but it's like back in my website building days regression testing like pixel to pixel my eyes ain't that good I can't tell you the difference between two shades of purple I just can't uh backstop JS could tell me every time and it was automatic and it took seconds and I would use it for that but for copyright yeah um for an actual copy I would want the person that has an English degree on the team to actually read over what I wrote not not just trust a machine okay um so an instant I've actually had a my

place at one point was I'm sorry an instant I had a my work at one point um a while back was we had a contract uh come in from a you know consulting company uh to help us build up and install a product and a few years later one of our guys actually went hey do we know about this git repository and found that on public GitHub the guy had uploaded um you know a bunch of like infrastructures code type stuff including Secrets um in that thankfully by the time we discovered that that test environment was gone now the passwords were active but theoretically you know at a point in time when that was uploaded you know there was potentially

a test system where there were these passwords there and like you know you can have all your uh you know we we actually use Vault um and you know we've got you know our own git server and you know even if we had scanning on that what's the sort of approach when you do have you know like no what's stopping a you know a junior Dev or some external person might ever just get push to you know a public repository like how do you find that you know when it's not really in your it's not your repository it's who knows right like if it's pushed privately you're right that the the greatest thing that gives us is the

ability that everybody that touches the code has the entire repository the most terrifying part of git is that everyone that touches the code has the entire repository this is the the double-edged sword uh if you actually type man get into a terminal it says get the stupid content tracker it's dumb it has no idea what it's doing um if they do it publicly there are tools that will look publicly again that's where we started from that was the first thing that our Founders built was let's look at every commit on GitHub and find out what strings there are so that's our public monitoring product is literally look for your strings uh look for your vars look for your secrets and let you

know when they pop up out on GitHub public uh we're not the only people that do that and other solutions that pop to my I don't have I don't have another solution that pops to mind immediately but I do there are other systems I believe crowdstrike but if there's anybody from crowdstrike in here let me know if I'm right about that but there are other tools that also can help you monitor public places but it's that's the Toyota problem is that their subcontractor pushed a repo publicly and didn't tell anybody and it took a security researcher five years to figure it out um there are other systems that are constantly scanning and monitoring and that's only gotten better

GitHub itself will probably tell you at this point depending on what secret it is their secret detectors are again pretty good uh and if it's an AWS key AWS is going to find it immediately AWS is constantly scanning the entire internet constantly because they have to any other questions because we're right at time but there's nothing else after this so I got a couple more minutes oh yeah a question let's get the mic up here it's for the internet you can be loud in the room but the internet can only hear so much so usually when we're talking about like credentials we talk about like MFA or two-factor authentication what level do you think that would come come to if we

were say you're adding a passphrase to your SSH private keys or maybe there's other uh Protections in place so even if in the code a uh you know it is leaked maybe there's like network controls so that the server can't be reached from a external Source 100 that's a great Point um and that's something I guess I don't really I don't talk about enough in any of this stuff because I'm so focused on the credential end of it but this is one piece in the larger equation um if you talk to us at the booth the first thing I say is we are a code security platforms that's what we're focused on network security um the biggest thing I think people

should work on is egress like know exactly where that traffic should be going and if you don't know where that traffic is going go fix that right now fix that before you fix this don't tell my boss I said that um I was on the internet damn uh but go go fix things like that um because yeah if the call shouldn't that's the the talk I saw earlier on um MFA issues with MFA if the call is coming from somewhere weird don't accept the call um but this is all part of it so yeah overall comprehensive security strategy yes you should have your network so locked down that if someone does get it it doesn't work but part of that

strategy should also be we rotate so fast that they get the code from yesterday it doesn't matter that um that line and Return of the Jedi uh they're code checked out it's a little old but we'll about to let them through like that wouldn't even work in ancient Rome like that's a dumb idea like a password should be immediately invalid as soon as it's used or it just shouldn't exist um but yeah that's a good point the more secure your network is the better security or like adding passphrases yeah yeah password SS Keys should be a mandatory practice I 100 agree with that is that like level two level three oh if I was gonna if I was gonna play back to the

talk and stuff if I was going to map it um I would say it's somewhere between one and two yeah but definitely three you're gonna be using it consistently because three is where you go you start the developers are starting to do things consistently level four is where it's mandatory across the board everybody does it and it's building the process but I think I think level one is where you start having that conversation of hey do we do we do that oh we do now okay that that's yeah let's go along that path because there's no real hard boundaries here it's like overlapping I like that if I have time to uh one further question is if you're doing all

this scanning um how does disclosure work for you when you find like hundreds of repos and hundreds of companies that are leaking these secrets if it's public yeah we email the committer okay and say hey did you know this is here um if whoever signed it so if you're signing it with an email that doesn't work know that we can't reach you and we can't tell you about it makes sense um so if you're some people really think that's a security thing like I I don't I don't want to use a real email and there's no git has no way to check but that's the disservice you're doing like you as a committer can't be reached

now so who could tell you about any issues they find so always sign it doesn't have to be a good email it could be like a protonmail but just make sure you're signing with something where we can actually reach you all right thank you great talk and thank you any other questions other than that oh one more one more and then we'll call you the last question and then we'll all go get some coffee uh I I just wanted to point out with the SSH thing like if you use a volt or any other sshca like you don't really have to encrypt your SSH thing like you can use oidc or whatever to grab yourself an

SSH key and then it rotates or whatever so if if you use a sshca like so volt can act as an sshca okay or or you can use ssh's own CA um but if if you use volt as the ca then you can use like uh ad or any other authentication method to grab your grab the key from volt that it will then use to log you into the server so then you don't really have to worry about password encrypting it I mean at that point you're signing it with the certificate Authority or that that's taking care of at that point you're using some other mechanism to give yourself the key and then it's signing

it for you I think we're all on the same page the more security you throw at it the better it's a matter of technical implementation at that point that no you're right if um if it's if you're that advanced on it that you're using it to sign then yeah all right hopefully uh hopefully that helps some people at home all right thanks very much everybody give it up for Dwayne Dwayne McDaniel don't forget there's a happy hour right now in the middle room tell everybody Dwayne sent you I think you get a free drink yeah I'm just playing so no talks in here until five o'clock God I forgot to tell everybody I got stickers you want stickers Dwayne has

stickers even if you get a prize thanks [Music] foreign [Music] foreign [Music] foreign [Music] thank you [Music]

[Music]

[Music]

foreign [Music] all right [Music]

thank you [Music] thank you [Music]

[Music] foreign [Music]

[Music] foreign [Music] foreign [Music] foreign [Music]

[Music] foreign [Music] foreign [Music] [Applause]

[Music] thank you [Music] today [Music]

[Music]

[Music] thank you [Music]

baby [Music] about my appetite don't leave me alone [Music]

[Music] I overthink it baby [Music] giving me Wind and Rain some kind of butterfly baby [Music] [Music] but I don't wanna miss you baby [Music]

[Music] foreign [Music] don't leave me alone baby [Music]

[Music]

[Music]

oh oh [Music]

oh oh [Music]

[Music]

[Music] thank you [Music] foreign [Music]

[Music]

[Music] foreign [Music]

[Music]

[Music] foreign [Music] [Music]

[Music]

[Music] thank you

[Music]

[Music]

[Music] thank you

[Music]

foreign [Music]

[Music]

[Music] foreign

[Music]

[Music] foreign [Music] thank you [Music] no no no [Music] back

[Music] grounds [Music] almost

foreign [Music] foreign [Music]

thank you [Music] foreign [Music]

thank you [Music] thank you foreign

[Music] foreign [Music] foreign [Music] thank you [Music] foreign [Music]

[Music]

[Music]

foreign [Music] foreign [Music]

foreign [Music] foreign [Music] all right [Music]

[Music]

[Music] thank you [Music] foreign [Music] foreign

[Music]

[Music] foreign [Music] thank you [Music] [Applause]

[Music] foreign [Music] foreign

[Music]

[Music] thank you

baby [Music] you're giving me winter rain there's some kind of butterfly baby

[Music]

[Music] don't wanna overthink it baby [Music]

some kind of butterfly baby [Music] don't leave me [Music] but I don't wanna jinx it baby again

[Music] but I don't wanna miss you baby so close [Music] fly [Music] baby [Music]

foreign [Music]

[Music]

[Music] oh oh [Music] [Music]

[Music] thank you foreign [Music]

[Music]

foreign

[Music]

[Music]

[Music]

thank you [Music] foreign [Music]

[Music] [Music]

[Music]

move it up

[Music]

[Music]

thank you

[Music] thank you foreign [Music]

[Music]

[Music] thank you [Music] thank you [Music] foreign [Music] thank you [Music] foreign [Music] foreign [Music] foreign

[Music]

[Music] thank you [Music] thank you

[Music] foreign [Music] foreign

[Music] foreign

[Music] foreign [Music] [Music] foreign [Music] foreign

[Music] foreign [Music]

[Music]

foreign [Music]

[Music] foreign [Music] hahaha [Music] [Music] foreign [Music] foreign [Music]

[Music] thank you

[Music] thank you foreign [Music] thank you [Music]

[Music] thank you [Music] [Applause]

[Music] foreign [Music] [Applause] thank you [Music]

[Music] thank you baby [Music] you're giving me wind away [Music]

[Music]

[Music] don't wanna overthink it baby [Music]

some kind of butterfly baby [Music] don't leave me [Music] foreign

[Music]

[Music]

[Music]

maybe you'll give me [Music] the way [Music] baby [Music] don't leave me alone [Music]

rain cause some kind of butterfly baby

[Music]

[Music] oh

[Music]

[Music] foreign

[Music] thank you [Music] thank you [Music] everything [Music]

[Music]

[Music] moving up

[Music] foreign [Music]

[Music] [Music]

[Music]

[Music] moving up

[Music]

[Music]

forever

[Music] thank you foreign [Music]

[Music]

[Music] thank you [Music] thank you [Music] foreign [Music] foreign [Music] good afternoon good afternoon we've made it through almost the first day of besides Las Vegas how y'all feeling fantastic fantastic we have the amazing Dr Matt Ware with us today this afternoon password 911 authentication adventures in healthcare so somebody earlier said he saw him talk 16 years ago so let's give it up for this man he's been talking a long time okay all right well let me get some stuff out of the way real quick first of all your cell phones turn them off again unless it's God or somebody's dying we don't want to hear it or at least turn the ringer off okay I want to take a shout

out for our sponsors the diamond Adobe our gold prism Cloud blue cat Plex track Toyota all that good stuff so without further Ado Dr Ware take it away [Applause]

in Sweden and a few things happened in the healthcare world since then there uh so it's really good to be back here and talking about passwords and Healthcare which are two of my great big passions here so in my day job I'm a cyber security researcher focusing on trying to figure out the Cyber resiliency for medical systems so I both look at kind of the impact analysis side so like a vulnerability comes out you know what is it going to be the clinical or the patient impact of this vulnerability I also looked at this from the defensive side as well it's like how do we go ahead and Design Systems so when that next vulnerability comes out they still

are you know resilient patients can still have care and we can still go ahead and do a good job so as part of this too I'm also a member of The biohacking Village so if any of you will show up at Defcon over the next couple of days uh we're gonna have some medical devices for you to be able to hack and learn on maybe apply some of these lessons that I talk about here too aside tangent um I was really pleasantly surprised by how easy it was to get medical equipment through TSA security like I I was coming through here I was like okay this is gonna be the cover story this looks dodgy as hell you know you know how

we're gonna get this through here and um you know no problem they're like okay it's not a bottle of water we don't care uh so just right on through um so as I was mentioned earlier I I have been doing this a long time I'm a long time member of the John Ripper uh passer cracking team because while um I'm passionate about medical advice security uh I am absolutely obsessed about password cracking so you know whenever I get to you know combine these two things together here uh it's just a great day for me so I'm kind of talking about how long I've done this here I was one of the first researchers to study the rocky

password list so I was able to go ahead and grab it online uh I published you know some analysis of this here other researchers were interested so I shared a list with them so if you're sick of the rocky password list I'm really sorry okay um but uh if you want to ever have some good stories about that uh talk to me afterwards about this so one thing though I really have a disclaimer here is I am not a medical doctor don't let this lab coat here you know fool you um I got my a doctorate in uh in computer science and was really trying to model how people create and use passwords and now I've just came this

pcfd password guest generator and I've been talking about that for a long time so since it's a password con so if you'll just you know uh give me a little bit of latitude I want to talk about this a little bit more oh and also before I can continue uh if you were asking like why I don't actually get any work done you know it's just because of these uh you know two uh little hellions here uh so they're always walking across my keyboard and generating really good random passwords for me thank you so when I talk about the you know pcfg password cracking you know the kind of the base question is you know what

actually is that and so the the official term is it stands for probabilistic context-free grammar and it probably doesn't mean anything to anybody here unless you're really into computer science or a grammar nerd or something else along those lines and so I went ahead and I renamed it uh the pretty cool fuzzy guesser and this kind of better describes what this actually does here so basically you train this on a sets of disclosed passwords it uses machine learning to go ahead and figure out probabilities associated with all the different types of mangling rules that people use so you know sometimes you'll put you know uh it capitalize the first letter they'll cap order but they'll also capitalize the last order

at certain probability you know people usually add numbers to the end of their password but sometimes they add numbers to the beginning or they replace letters with numbers and so what this uh pretty cool fuzzy guesser does here is it goes and takes all the information and then generates passwords in probability order so it'll get generate the first most problem get password guess the second most probable password guess and so on and as you can imagine this is really useful if you're trying to go ahead and crack passwords so I'm not going to talk too much about this because I really want to focus on medical device cyber security but there has been some work in this uh um here at

diets one I briefly mentioned so the first things I've had really recently here is a default Russian rule set because I believe in kind of sharing this all around not just focusing on all of us English speakers uh so it's got a lot of new support now for you know cracking Cyrillic passwords and you can find out more about that if you go ahead and read the developer's guide I also added in honeyword and synthetic password generation so what this does here is instead of generating the most probable path for guess and then second most probable one it actually generates it according to distribution that you would normally see in a normal population so it generates essentially

fake password dumps so these are passwords dumps that look very realistic what you find in the real world and the question is like why would you want to do that there and really the kind of the main use case for this here is being able to help provision quickly large deception environments because no one wants to go ahead and you'll generate a thousand you know active directory user passwords by hand and so this is a really good way to be able to make that you know be able to fool an adversary uh based upon this though I also added in what's called a weighted random walk guest generation so this generates password guesses same way so it will go

ahead and just kind of randomly walk through the grammar and generate passwords so by default they'll generate you know the most probable password guesses first but you can go walks around and it causes a lot of you know duplicates as well so the question is you know why would I go ahead and do that and the simple fact is that the pcfg passer gas generator is slow it's really really slow so like it cracks passwords but you know you generally want to go ahead and use other attack modes if you're not targeting something that's really computationally expensive was this weighted random walk though um it's fast you know you don't have to keep track of states and the memory

requirements don't change over time uh it's highly paralyzable because each gas is independent of each other and what this really means is that it you know it really kind of hopefully scales to a GPU passive cracking type of a method here so just to give you a preview what I'll be talking about next year hopefully is I really want to go ahead and get this included into hashcat there so that's going to be kind of my goal there but that's all really kind of to the side you know because you know what I really want to talk to you about though is passwords in hospitals uh because this is something that's really important there you know with password

cracking you know we're we're getting better you know that's I won't say it's a solid problem the next talk we're gonna hear is going to be really trying to get into that last hard part of it there um but you know there's a lot more problems here with you know trying to secure hospitals you know how do we go ahead and uh be able to provide care in a kind of a hostile threat environment and it definitely is a hostile threat environment too hospitals are absolutely under attack so the the primary people that are targeting them right now is your standard kind of ransomware type of uh you know criminal organization there uh because they like the money and

hospitals have money I mean that's really kind of what it comes down to there so you know kind of if you look at what they're doing right now it's kind of a standard that you see across any other industry here they they break in they they encrypt the files they demand a ransom payment in order to decrypt them um in order to do that uh but um hospitals have luckily and this is a really good development here have gone better at being able to respond to this because being able to provide uninterrupted you know clinical care is kind of core to their mission so unfortunately the attackers they don't go well okay now this is getting

harder we're just going to walk away and Target other Industries uh they've definitely been going ahead and changing their tactics as well uh so recently we've been seeing them go ahead and start trying to Target the hospital support structures like voice over IP systems and you know it's really hard to run a hospital if you don't have a phone um you know different departments need to call each other patients need to be able to call it so when a phone system goes down to hospital it's a really bad day uh the other new tactic that's really been kind of popping up lately and this is a really um just horrible one is extortion to patient medical data so

they'll go ahead and break it into the hospital they'll steal very sensitive patient data and then the hospital won't pay them because they were going ahead and they have recovery procedures in place to be able to respond to that there so the hackers then reach out to the patients and say hey look we're going to go ahead and publish this very very sensitive data about you online and then tell everyone you know about about that sensitive data unless this Hospital over here that has all the millions of dollars goes ahead and pays the ransom and so they're getting to patients now to try to you know pressure to hospitals in order to pay to Ransom so that data

doesn't get leaked and so you can see this is a this is a really important problem for us to be able to solve because we're all future medical patients hopefully you know that we you know if we live long enough we're going to go ahead and end up in these types of systems here and so and so it seems goes for everyone that we love and care about uh so being able to provide safe and how effective Health Care is a really important topic that I'm very passionate about now one thing and one reason why I want to give this talk though is because our Solutions have to fit into a clinical setting there's no if ands or buts about

that there and so it this doesn't really cause a lot of problems because a clinical setting is very different from your typical enterprise system there you don't want you know someone to go ahead and reboot the you know the infusion pumps or something like that there when you're in the middle of get receiving treatment there you know you don't want your anesthesiologists to forget their password as they're trying to log into their computer or something like that um you know what you know password change policies that suddenly break systems that are providing care so trying to figure out you know how we can go ahead and secure the system and apply you know strong password you know

policies and authentication uh mitigations into this here is really important so I could keep on ranting about this but I really talked rather talk about you know case studies here so the first one I really want to talk about is control substance infusion pumps so these are very common devices you'll find across hospitals and are used to dispense pain medication uh so you know any of the really good drugs uh there that you can think of there they generally fall into this here and you don't want patients just to go ahead and open them up and start reaching in and start squeezing that you know bag of morphine really hard there so pretty much the defining feature of a

controlled substance infusion pump is a lock and key so like here's one model here there's where the key goes here's where you know well I guess that's a lot but you know you know there they go for that so if you go ahead and see a device with like you know dispensing drugs that you know has a key that's probably a controlled substance infusion pump so this is probably the most you know Common basic security password type of a you know dime ever you know you got something that's really valuable you don't want people to go ahead and just grab it so you put a door on there you put a key on that door or lock on that

door and you call it a day so this is really kind of a standard uh you know password type of a problem so what if I told you though that this is also kind of a standard issue that comes up a lot is about key reuse and specifically it's a common practice across all these different manufacturers here that they will use the exact same key on all of their devices so uh for example like if you go down to the hospital down here on the Strip uh you know you know what type of infusion pump they have you can go ahead and get the key really easy and go ahead and log right in and open it right on up and as

I said this is very similar to for all these pumps so this is pretty much a universal thing I'm not trying to give you know individual manufacturers a hard time the only kind of disclaimer that I have is that you have a lot of these kind of third-party you know plastic cases they put on there and there's not really standardization of the keys for those simply because there's not standardization between the manufacturers so this is the audience participation part here to talk but it's just a problem okay we have some really you know you know important stuff we don't want patients going ahead and squeezing it we don't want people stealing it we put a lock on there we use the same key

for every single lock all around the world um is this okay no well I would actually argue it is okay um because you know you have to be physically close to this in order to be able to do this this is not a remote attack and if you start going ahead and opening up these infusion pumps and squeezing that bag really wicked hard you know the nurses might want to stop you as well and you know so you know this is not a huge problem and at this on the other side there Physicians and nurses and everybody else need to be able to get in these pumps very quickly and so these Keys have to be very

reliable uh because you don't want that to jam when you're going ahead and trying to go ahead and change you know a patient's medication you also can't really change your treatment plans while that key is locked as well so you have to be able to unlock it because you don't want patients pressing that button as well so making sure that lock is really reliable that can be open to all types of situations it has you know a strong failure mode and stuff like that there don't have to worry about it's really important for providing patient care so there's not really been any uh you know examples as far as I know either of you know patients actually compromising this

so you know the chances of someone you know having that key around their wrist when they're getting a motorcycle accident is pretty low so this is kind of an acceptable trade-off that you know most people take when it comes to password security or you know authentication issues when it comes to infusion pumps so let's take this up a notch though should your pacemaker have a password I know yeah everyone's kind of sucking it's like yeah that that seems like something that would be really important there you know we don't want people just going ahead and logging into pacemakers and pacemakers are absolutely very connected um like right here you see like a cell phone okay you don't want just everyone

you know walking by you know logging into your pacemaker there and this this is very important to enable people always ask like why are we enabling this well this is providing information to your your doctor about you know how that pacemaker is functioning uh often pacemakers need to be tuned around and stuff like that there because it's you know it's a it's a very delicate type of a saying so being able to provide that level of reporting saves lives so we can't go ahead and say we're going to go ahead and disconnect all these devices make them dumb devices we have to be able to enable this but like even looking at this like how

does that pairing work like there's not a button on there I can push like you know when I go to like pair my cell phone what's that do I do need to do like 15 minutes of cardio or something like that there in order to like get my heart rate up and uh you know accept that request and I mean the short answer and there's a couple different ways to do this here is that you'll have an app you'll go ahead and you'll enter in your serial number into that app it goes out to the manufacturer and the manufacturer gives you kind of a rotating pin that you can then use in order to go ahead and pair

your cell phone with your pacemaker so I know we have a lot of Security Experts watching this right now and you're like okay sms okay that's a problem sometimes there's challenges with this you know how random is a serial number that that's actually being generated for someone to be able to guess there's a lot of different rabbit holes that we can go down into the security of this here uh but before we do I just want to point out another different use case that will kind of uh negate a lot of that and that is the pacemaker physician programmers so the cell phones generally just report data about that that operation there uh but you know

Physicians need to be able to modify that treatment plan of those pacemakers as well so these programmers not only can they read data but they can go ahead and set the settings of your pacemaker as well and so that's really serious and the other challenge is that often these devices need to operate in kind of an offline environment you don't want to say like I'm sorry we can't deliver life you know giving care because our internet is off uh so you know these these position programmers need to be able to access pretty much you know anybody's uh Pacemaker and this can happen whether you're here in Vegas whether you're at home whether you're in Australia or anywhere else you might

need to be able to receive a medical care and so these uh programs are kind of interesting each manufacturer has a different one so you'll often see them they just kind of stacked one upon each other in the hospital there uh each of these will also be running a bunch of different apps on there for each different model that they want to be able to be able to support if you look at that GUI you're like wow that's really old yeah you're right it's about you know it's three years old there uh but you know you have a GUI that works in a hospital you know Healthcare System why bozzer to update that there so

how do you secure this okay this is something to have to be able to access pretty much anybody's pacemaker you have to be able to go ahead and distribute this you know all over the world basically uh you can't really have that functionality to phone home to a centralized database to be able to receive these you know whatever security code that you have like how do you like how does this work with passwords um so like one thing they might start to think though is like maybe the manufacturer really has some proprietary algorithm that no one can reverse engineer or maybe we like hard code the password onto here and you know really try to keep these programmers safe you

know we really try to lock them down and ensure that they don't get down to the wrong hands and oh well that's not going to work um so you know these things show up on eBay as well you know maintain being able to maintain physical security is these devices is very hard so one thing I just want to highlight is you know I've looked at this here I actually haven't bought one of these uh doing uh pacemaker research is actually really hard because you don't want to be like oh I wonder what this does and then someone just drops dead outside your hallway um but uh you know this is definitely something you need to worry about when we talk

about you know the Threat Level to be able to do this here you know adversaries if they really want to be able to get into this here uh there are definitely ways that they can be able to obtain these types of devices now I want to take a kind of a step back though okay uh and it really just kind of um ground us because um I don't want anyone to leave just scared okay these devices they save lives full stop I have people that I know and care about that have connected pacemakers and I'm okay with that because the the value of this the care that they're receiving from that there is way outside the risk that they can

potentially be happening there's also no evidence of any implantable devices being exploited in the wild so while we're basically talking about theoretical attacks that could potentially happen they're not saying that's currently being exploited and in the level of expertise and resources develop these working exploit is non-trivial okay it does take work and luckily the Venn diagram of people that have the technical skill to be able to do this and people who want to kill people is pretty low uh you know as far as overlap goes and but I want to have a better story than this and this is why I'm giving this talk here it's because I really want to inspire everybody here to really start

trying to figure out better Solutions uh because you know we have to enable disconnect conductive functionality we have to even add to this connected functionality make it even better make it more connected so that we can get started getting data and uh in order to save people's lives but we also want this to be safe and we don't want to have any caveats we don't have anybody worried about whether they're gonna be hacked by their pacemaker or not so we really need to get together as a community and figure out you know Secure Solutions in order to be able to you know address problems like this so some topics I'm going to be really talking about here and kind of harping

on is you know first of all when we talk about medical devices and stuff like that what we're really talking about is medical systems like everyone focuses on the end device that you can buy on eBay but it's connected to all these other different types of systems and that's the whole reason why we're making that connected in the first place so trying to understand the whole security of the system of systems is extremely important the other thing I really want to highlight is the idea of resiliency so you know devices are going to get hacked keys are going to get leaked the the real important question is you know how do we still you know maintain and

deliver Safe Care to people when these problems happen so for example like what's the pacemaker option we might not be able to have a password with that so what else can we do maybe and I know this is a hard problem to ask for you know at a password con conference but like are there non-password solutions that we can have you know uh for example like can we have better reporting so that way we can say you know if something does happen we're able to detect it very early and real respond to it one thing that manufacturers do is they have safeguards in these pacemakers um mostly they're from a safety standpoint you don't want the physician

to like hit the wrong key or you have to walk over their keyboard or something like that and also and you know for the cause harm to a patient so a lot of times these devices you can give them you know you know bad data but they'll fail into a safe mode that will still provide you know not great care but at least some level of care uh for them so maybe you go ahead and start doing that in order to be able to better secure it so even though you know anybody can log into this uh pacemaker uh they have a harder time actually causing actual physical harm but what I really want to

inspire people to do is start thinking like as a system of systems and outside the box of how do we go ahead and secure and enable these really hard uh um uh clinical use cases so let's have another you know uh case study here store credentials we've heard about this quite a bit actually uh from all the previous talks that we've had throughout the day here and I'm sure we'll be hearing more about this later too so medical devices store credentials just like everything else so here's a cve and it's basically for an infusion pump and I'm sorry about all the words um but really what it boils down to is that the cve came out and said that you

know and hacker may be able to obtain unencrypted wireless network authentication credentials to the hospital uh by disassembling one of these infusion pump units and just reading the flash drive so the caveat There Was You Know accessing the internal flash memory requires specialized tools and then like nurses might get a little bit annoyed at you too if you're going ahead and just you know unscrewing these uh on the the ward there um so that was really kind of what it was saying there and you know the kind of mitigation saying that this wasn't a really big problem so my question to the group did I want this to be a little more interactive you know is this a problem you know Wi-Fi

credentials to the hospital stored on a medical device that you have to physically open up in order to extract a flash drive in order to read who thinks that's a problem oh we got a few hands here okay guess what I'm with you there okay because it that doesn't have to be in a hospital when you go ahead and extract that data so here's a bunch of diffusion pumps in that cve listing there um they're a lot cheaper so we went ahead and bought one um and so here is that infusion pump in our lab uh we name everything after Simpson's characters in our lab because that brings me a lot of Joy uh and so like you know there's me opening

up there's a tamper evident uh seal that was just a sticker and I'm actually kind of cool what's that security feature because um you know it doesn't I'm not gonna report it myself so having a really good taper over the steel doesn't is important it's probably worse that one penny so the board biomed just doesn't start unscrewing it in the you know and when they're messing around with it and stuff like that you know there's the flash card uh in it um and so there's me with a proof of Life of that flash card there so you know being able to uh you know uh pull that out is a very realistic you know type of a threat there

now one thing I want to say though is you know after all that work there I actually didn't find any uh credentials on that particular model there but what I will say is I have found lots of Wi-Fi credentials on other eBay purchase equipment that we've bought as well there uh so this is absolutely a very real thing um the disclosure is always a pain as well because she's like you know it's like oh my God I hope there's not patient data on here as well um but uh so you have a lot of fun talks with lawyers when this happens here uh so this is an example though of device retirement uh so this is

something that all devices go through you know uh the device you know gets used and someday it doesn't get used it gets you know replaced it's either thrown out or sold on the secondary Market um and then you need to make sure that um you know that's done secure later because you know people don't change their Wi-Fi passwords very much especially in uh you know high value environments there so one thing that I really kind of believe in is the idea of threat modeling and so this is something that you know our team's really been involved in quite a bit a couple years ago we published A playbook for threat modeling medical devices uh This was um uh sponsored by

the FDA uh to really try to get threat Molly involved in early on in the development of medical uh devices there and throughout this we have the idea of kind of example of high value data flows and these are common things that like a lot of medical devices do they have a lot of security problems so like device retirement that has a lot of security challenges that can occur there it'd really be nice if we started seeing the you know more Advanced Techniques like TPMS or you know other types of things like your cell phone you know is pretty good at being able to secure all these different things even if you lose it you

know somewhere um so there's a lot of different other kind of high value data flows that you know tend to trip people up as well so in this Playbook what we did was we have a bunch of example uh you know fake but you know realistic kind of looking medical devices um and we start walking through kind of you know how you go ahead and do threat modeling of these devices as well as common problems that occur so like a big one that we have is software update you know software update is a really important thing we want to be able to make these medical devices be uh responsive uh to attacks when they come out so being able to update

software provides a really good mechanism for that there but it's really hard to do secure software update um there's a lot of things that can go wrong and so one other research question I kind of like to put out to the whole audience here is um you know how do you do this and how do we make this easier for medical device manufacturers to actually Implement because trying to implement secure software update especially for some of these you know not real-time operating systems and stuff like that there is actually non-trivial and if you're wondering like you people get them wrong absolutely okay uh so here's another uh cve this was also for another infusion pump and basically it says application

does not restrict the upload delicious files during firmware upload update so there was no device signing there's no authentication you just pop an executable in there it's like sure sounds good and it goes ahead and installs it so like I'm not the greatest hacker in the world but you know even that level there I can go ahead and be able to deal with so this is another example though from the credential management and stuff like that there of being able to you know properly sign software updates be able to distribute them out there correctly and stuff like that is a really hard problem to be able to solve that pops up in pretty much all these different types

of medical devices out there now I've been giving a lot of different uh infusion pump examples here and um you know the initial thing might be like oh my God you know infusion pumps are really insecure and not completely wrong uh but uh you know the reason why we see all these different vulnerabilities pop up about them there is really it comes down to their cheaper than MRI machines we start looking at other Mac devices you see a lot of same problems uh that's actually one of the things where I love being in uh cyber security for medical devices is because it feels like I went to a like a time machine and I'm certainly back in the 1990s like oh look

telling that hey cool um so you know there's a lot of different you know there's a lot of room for improvement uh in medical device Security in order to be able to get them to start using more modern protocols and uh be more resilient here um but I have to say I'm cautiously optimistic um because you know the security research that we're doing right now is having a huge impact and I really do believe that security research acts a lot like a vaccine there you know you're you're basically stressing uh the organism uh in a way that's you know safe and it develops defenses against you know what those stressors are and so in this case it's cyber security we're

stressing these infusion pumps here and across the industry we're really seeing the infusion pumps in infusion pump manufacturers are really kind of taking uh notice so there's a lot more Guides of how they go ahead and secure devices securely and they're starting to implement much better security there uh the really big problem though is Legacy systems you know hospitals just don't go ahead and update their systems uh you know all the time there so these systems that you hang around for a long long period of time and then the other thing that I really think is kind of important is how do we you know translate lessons learned from these types of devices and this type of

research there to all the other types of devices in the hospital because we don't want to go through this long painful process with every single device in that hospital it'd be really nice if you could kind of learn from everything and go ahead and develop more Secure Solutions and really apply that to other people as well there so I've been on my high horse a little bit too long here so let's talk about more about you know other types of examples uh that you know can pop up here in uh Healthcare so Health electronic health record systems are really kind of the core of any sort of hospital if you follow any sort of cable from any medical device in

a hospital and you follow it long enough through all the different wiring closets and stuff like that there eventually you're going to end up in an EHR system okay because this has all the data about all the patients that you care about so this is a you know what we use in a lab we have an open EMR because it's free and once again that's kind of nice uh but this is actually a an EHR that's actually used in a lot of smaller organizations as well because it's free and you don't want to have the billion dollars to spend on Epic or Cerner um so um we also just um and it's kind of a quick shout out we use a synthetic

patient records for this so there's a project called Cynthia that allows you to generate very uh realistic synthetic patients so this will have you know they'll have the proper comorbidities so you're not having like a a baby with dementia um so uh if you squint it it looks like you know a real estate patient and that way you don't have to worry about dealing with an API when you're doing any of your research there so these electronic health record systems though they need to be connected to a lot of different other types of systems and so he starts thinking about this and you look back at the couple you know previous talks that we had in this

track here and that means apis you know everyone's you know a favorite you know saying that we're talking about when it comes to computer security so there's a lot of different ways to be able to connect to these types of systems um and um basically one of the newer ones that's really kind of interesting is called the fast Healthcare interconnect interoperability Resource number one and it's a it's a fairly new standard it's only about 10 years old um and number one as you can see it's you know they're still trying to build it out but it's a lot better than the previous uh standard called hl7 uh which um it's it's like Vibes of a standard kind

of it's not really a um so you know it was a fire it's actually it's a pretty good uh way to be able to be able to transfer medical records between different devices and because it is a newer protocol it does support other types of you know newer you know authentication mechanisms as well like oauth 2. and so Oasis it's pretty good actually okay I mean I see a couple people naughty like okay you know that's uh and that's not horrifying uh so that that's really good there uh so so that's that's really good and that's really optimistic there uh but the problem is what happens when we take you know these protocols that we're

using in a hospital and we start trying to go ahead and use them to enable other types of healthcare delivery Opera options here so a good one is kind of Aging at home you know we want to go ahead and move more you know medical devices and you know Healthcare you know uh systems into the house so that way you don't actually have to go to the hospital you can get better care at home you want to be able to see your patient records which is really important for everybody as well and then we also have a lot of different other areas too where you we want to be able to um you know interconnect different uh

companies together we want to be able to interconnect different Services together we want to connect us to the cloud so you know how does this go ahead and work and the short answer is there's some Growing Pains um so uh recently um or a couple years ago actually but so it was a number of really kind of interesting vulnerabilities came out called uh playing with fire and um the work that Alyssa Knight and her team did was uh really kind of cool they found a lot of different challenges with basically trying to analyze uh web apps you know on your phone uh that you know connected back up to EHR type systems there and while they found a ton of different

problems really the core one that I kind of narrowed in on there was that a lot of these patient apps here focused on authentication but not authorization so they would make sure that they authenticate you really well they would make sure to get an account on the system that you were actually who you were talking about and you say I want everybody's record it goes sure you know there you go you know you can have whoever's record that you want there uh so what really had come boiled down to is they didn't Implement a whole lot of server-side controls so they they make sure that you actually had an account the door but once you're in the door you

could go ahead and take out everything that you wanted was that and kind of the question is you know why didn't they think of that there you know that seems like a pretty really you know basic or important type of a thing and uh my hypothesis I'm kind of looking at some of like our EHR systems here is hospitals really don't work that way you know a hospital you're not trying to say okay this patient only has access to this one piece of data what you're trying to say is this medical device has access to all this type of data here so okay this one can have access to your dental records okay this one can have

access to your X or x-rays here or this one here can have access to you know your billing information for some reason um so when you look at you know how you go ahead and specify authentication and authorization and OC's devices here it's really these broad categories that these really focus on so this works good in a hospital it causes a lot of problems when you start taking these protocols like you know willy-nilly and trying to move them to kind of a patient-facing app where you say you know you really only need to have access to only your data and so I suspect that this is going to be an area where we're going to continue to see a

lot of challenges with you know over the coming years and so from a security research researcher standpoint I would say this is something that you know you should probably look at when you're trying to analyze these Health apps is are they actually you know having any sort of authorization as well as authentication that they're doing there so getting back to the patient bedside though let's talk about service pin codes so if you find something in a hospital and it has buttons on it it probably has a hard-coded unchangeable service pin code that you can use in order to enter into kind of some restricted mode in that and this is this once again this is a

universal issue it doesn't matter as I said if it has buttons it has a back-end you know hard-coded password on this so this is kind of once again here's our lab uh of course because I can't remember these uh we you know wrote them down on a piece of paper and just tape them onto the device here um so now I have to say like I've been doing password cracking for a long time uh I know a lot of you know good reverse Engineers too where I work um so the question kind of comes down to you know how do we find out this data here you know do we have to go ahead and

you know crack this open you know pull out that flash card again start reversing this you know extracting these passwords uh there was a previous talk about your reversing mobile applications do we need to go ahead and do this in order to get the you know that those service pins and uh the short answer is uh I'm lazy so no I just go ahead and I download the manuals and then you know like somewhere in there this was like in chapter one here's all those PIN codes for that right there so I I heard some people say no no here uh I guess my question to you is is this a problem uh who here thinks this is a problem

right now okay let's get some hands up here oh we got a couple of them yeah we got some wavies as well um I actually will say I don't lose any sleep about this here okay I don't think this is the problem because I kind of like factors around to like having that physical key like if you're like physically there and are able to start pushing things into the button and is that there like if if I'm connected to that device I'm not gonna be doing that because I don't want to be like oh I phoned the device and you're dead you know um yeah so um uh you know I really don't have too much of a problem with you know

these you know physically you have to be physically there in order to be able to access these modes because it's really the main goal for this here is just to make sure that like a nurse doesn't accidentally like change the IP address of a device here or like you know uh you know a patient just doesn't like you know start messing around with it and get into an unsafe mode here uh but it's not really a real secure uh you know access control that needs to be kept secret from everybody so what about hard-coded remote passwords so this is a a a problem here so I was kind of curious about you know trying to figure

out like how big of a problem this is so um I work at miter and so we're all about our different enumerations we have you know a million different ones there you hear about attack cve cwe and stuff like that there so cve common vulnerability enumeration is a really good way to be able to kind of see you know what vulnerabilities have come out in devices before but there's a lot of vulnerabilities that come out so being able to narrow your search for them is a little bit hard sometimes so one thing I want to kind of highlight is cwe common weakness enumeration so you can say you know I want to find out this particular type of

vulnerability like you know use of hard-coded credentials so why don't I go ahead and search all the cves for this particular cwe and see which ones pop up here and the short answer is a lot okay um and actually I was surprised by how few I found because I've done a lot of work with medical devices and I've definitely seen a lot more make devices that had hard-coded unchangeable remotely accessible passwords uh than what pop up here so this is a real problem okay and this is definitely something that you know we do need to be able to address and this is something I'm a lot less forgiving for from when it comes to trying to

assess the impact of these security vulnerabilities here now there's definitely other mitigations that can be put in place in order to try to limit the damage that's done by these here but you know this is something that you know you know going forward I would really like to see less of here so you know we talked about a bunch of different types of password you know vulnerabilities what's issue uh you know different medical devices here and I didn't offer a whole lot of solutions as well I have to admit and um I'm very um aware of that and that's actually one of the big things that really spurred me to have this talk here is we need more

cyber security researchers in the medical field um there's a lot of different work that needs to be done and I don't want anyone to hear it leave here you know afraid I don't want you to be afraid of receiving medical care I don't want you to be afraid for your um your loved ones here when they they're receiving medical care but I want you to leave here inspired I want you to leave here really determined to kind of look into you know different types of medical devices and really try to start doing this on this research and it's fun work it's impactful work and there's a lot of low hanging vulnerabilities for you to be able to

deal with as well there so one other thing I really want to be able to highlight though is that the solutions that you come up with have to fit into a clinical setting okay you have to really always keep that patience you know uh Care at the Forefront of your mind so it making sure that's available making sure that these workflows are able to work is probably the number one priority there and so we have to make sure that we have to change our security practices around that versus trying to change you know these clinical care workflows in order to fit into our security uh methods here um I also really am a big fan of threat

modeling uh so um I really think that you know if people just start doing threat modeling that really can be a you know Help win the war a little bit here I mean at least try to identify what are the threats what are the trade-offs that we're making and you know how are you going to go ahead and address those threats there as well the other thing I really do strongly believe is that cyber security is too hard right now okay medical Heights manufacturers are not going to go ahead and you know focus on cyber security that's not that doesn't you know make money for them there uh so what we need to do as a community though is make

cyber security easier so that the way it's easier for them to actually Implement devices in the right way uh than just trying to roll their own solution here so come up with like common practices and common libraries like how do we go ahead and do authenticate for you know pacemakers and stuff like that there and come up with standards for that there I think it's a really important area of research for us to go forward with here as well uh but really at the end of the day there you know it's really important that we we can't say no to all these different options like someone's gonna say I want to connect you know this

medical device to the cloud and I mean I got feelings like no don't do that don't connect that to the cloud I don't even like connecting my home you know uh you know smoke alarm to the cloud or anything else along those lines uh but we really need to be able to enable this we need to say yes we need to be able to do it so that people are able to receive effective care going forward so that's really kind of my uh you know stump speech here for the importance of this work I hope that you're you know inspired by about this here I love talking about this so feel free to hit me up really whenever and I will you

know rant your ear off about this and I really want to learn from you as well here uh and also come visit the biohacking village later on if you actually want to hack some of these devices uh for real as well so what's that does anyone have any questions we're in the back there I used to be a oh thank you uh Matt I used to be a paramedic and now I'm in the cyber security field and I'm curious I got two questions one's long one short first one is how do you deal with the what's your personal philosophy on dealing with cyber security and Medicine of that like emergency situations need to have you

know break glass in case of emergency those keys versus keeping things secure so that's the first question the second was are you guys hiring so I'll answer the second question first yes uh so uh talk to me afterwards uh for the first one um I will say uh I am all for break class uh at the end of the day uh I I once again kind of going back to that bullet point earlier there I think we have to be able to enable those clinical workflows like I I forget my password all the time I mean that's just the case there I don't want to be like in the doctor's office like I forgot my

password and they're like well I guess we gotta operate you on you you know take out that medical device there that's just not an acceptable answer that I'm willing to take uh so uh we really have to be able to enable all those break class options there and that causes a lot of problems because that means that password's a lot harder you know we have to be able to allow people that may not you know be good actors in order to be able to break that class as well so coming up with other resilient options so when that glass is broken um you know your house still doesn't get you know completely ransacked you know

maybe uh you know saves for your important data or something like that there uh so long story short I'm absolutely on the break glass of functionality there I'm you know availability and being able to provide care over you know following just kind of a a standard security thing we need to update security in order to make enable those uh that that functionality uh yep you have a question right there yep uh thank you Matt great talk obviously um so kind of to Pivot with that same break glass emergency sort of thing um I work for the defense health agency um and so one of the things we all be the first thing we always say is patient

care is first right what happens when our medical professionals start to weaponize that against us so when we start to make what I consider the very basic cyber hygienic policies in place they fight saying like oh this might kill someone I was like sir your ability to watch YouTube videos is not going to kill a patient but I can't but like all the veterans in the room I'm really sorry that's an actual real case study um so so like how how do we basically in your opinion like what's the right balance of like no no we really want you to save people but also don't be a dick about security um that's a real hard question and

certainly when I'm certainly struggling with myself here uh quite a bit because I like watching YouTube videos too um [Laughter] um but I think it's really a conversation and that's where I'm actually kind of a fan of threat modeling uh if we tell people like you guys Implement a security uh process and they're like no um you need to come back and say okay here's why and you kind of walk them through it because if you tell people like okay uh you know the hospital over there got hacked because you know they enabled you know internet access to this you know clinician facing the computer here and malware gone to that and then it ransomware to hospital there uh

people because can sort of start to understand it and you still have to have a conversation because they're like well I'm not gonna click on that link um but um you know being able to really kind of walk through these types of spreads really try to walk through the procedures and then maybe come with an alternative option like okay you need to be able to show YouTube videos to your patients because there's a lot of useful Health Care information why don't we go ahead and make another laptop available for you to be able to uh you know um uh you know have this really important interaction that you're having with your patients as well so I certainly haven't

found a good answer to that question it's a it's one that's going to continue to pop up all the time um I mean I know like I'm a bad user too like when my Corporation says like okay you need to change your password I'm like I've done a lot of studies to say you don't and uh it's not helpful um uh so you know having those uh you know interactions and stuff like that there uh it's really tricky but we need to kind of come to the table with them as well to try to figure out all the network flows for them uh so um botnets have been a very very uh critical uh you know attack Vector for

uh all these vertical devices so um the the problem lies with the device manufacturers not abiding by a process supply chain uh you know validation and stuff like that so how do how do how does pitre take this to a Next Level to better educate the actual vendors of these devices and supply chain uh so that's a really good question there um uh so um everyone has a role when it comes to the medical device security you know patients have a role doctors have a role hospitals have a role medical rights manufacturers have a role the thing is when you go further Upstream there um the the more resilient the system becomes because like trying to expect

users to be secure is just you know not going to work you know trying to expect doctors to be secure you know you're gonna have exceptions here trying to expect hospitals to be secure you're gonna have Exceptions there so trying to move that up and you know really focus on making Place manufacturers can have a huge impact throughout the entire industry there now I mean the challenge there is you know how do you actually inspire this and so um one way is uh you know what's legislation so you know there's a whole I am the calvary track here that really focuses on policy um but you know that's a that's a dirty r word there you know you don't you know

you have to be really careful about that as well uh so the other thing I think is really kind of important is just making it easier uh for make levice manufacturers there uh so you know expecting them to have a cyber security and password and authentication expert on our team it's just not realistic I mean there's just not a lot of those people out there and let alone you know for these smaller device shops here so you know having practice guides so they when they go ahead and say you know I need to make this device I need to get it through you know FDA certification I need to do go and do all this stuff here

hey I can go ahead and use these libraries I can go ahead and use this you know follow the standard practice guide that's been put out there um you know that's just easier yeah you know than trying to roll your own solution and do it so if we can go ahead and make security easy that's really the way to be able to inspire them and so there's different resources available for medical voice manufacturers uh so the nist a national cyber Center of Excellence uh publishes uh some implementation guides that are really useful for different areas there so they have one for like Telehealth one for Infusion pumps and so on um as I mentioned like we've you know

authored A A playbook for Molly medical devices uh so and you you know now we're starting to need to be able to do threat modeling if you want to get your device to recertification so being able to kind of use some of the Lessons Learned in that that Playbook there uh can be very useful for you as well um and then just being able to use kind of standardized libraries and you know there's a lot of overlap now with like iot so you see standards like matter they're coming out for the iot setting and uh medical devices look a lot like your smart door locker or something like that sometimes too so especially when you're starting to move them into the

house so as we start to see you know security features being used across other Industries trying to figure out how medical device manufacturers can