PW - Protecting Against Breached Credentials in Identity Workflows

so we have with us matthew woodyard uh from oath zero uh first time of our pastor's comp but he's been to besides before um not trying to intimidate or scare you or anything i'm really happy to have you here and with this talk protecting against breach credentials in identity workflows so go ahead all right thank you so much for the introduction since it's my first time here i'd like to tell you a little bit about myself so professionally i was in finance for about a decade um and worked through a lot of rotations within security so it could have been compliance um fraud uh architecture really anything you can imagine um i probably did it at least

for a little bit of rotation i co-founded a threat intelligence startup called bad packets and currently i research emerging threats at auth0 based a lot on the data that we have i also do some academic research with the university of glasgow mostly on botnet tracking um i live in chicago i love to bike i've started gardening but i'm really bad at it mostly it's just weeding i mean honestly like if your partner wants a garden you will weed the garden so that's how it goes um so about us uh auth0 part of octa we're a customer identity access management platform which means that we focus mostly on things um facing customers so you like sneakers you go to

site to sell sneakers you hit that log on box that's what we do and anything past that though completely opaque to us right so we don't know anything about the web apps downstream we don't know about their appsec posture um so that gives our intel team the motivation to track botnets track credential leaks aggregate insights from the logs that we get from all of our customers and maintain a robust credential database and integrate it into um identity workflows before they hit applications because that's where our control stops all right so that's me um i want to talk just a little bit about credential stuffing you know what it is um keep things a little bit informal really

because i know that a lot of people in this um room may have some background here on what it is already but nevertheless we'll go through it i'll keep things a little bit informal i'll stop for questions like in 3x so if there are any things that need clarified please definitely do that so we'll start with the what why and how of credential stuffing the state of affairs and what we're seeing from our log data and controls that we can put in place so what credential stuffing is exactly um essentially it's the reuse of credential leaks and attacks from unrelated leaks to the target account so by credentials here for the scope of this talk i just mean things like usernames

email addresses and passwords used in some combination um or or something like this so related attacks like session hijacking other types of brute force attacks gonna be out of scope for the purposes of this um presentation but i think president um credential stuff you will be with us always because people will continue to reuse passwords sites will continue to leak the passwords that you used on them um so why do threat actors employ this attack in the first place besides just his efficacy like what are they after what are the goals so a lot of times there's an economic objective so i'm going to give as an example say that an attacker wants to do

an account takeover and their goal is to use a password that you had used before say that it was leaked by ashley madison and you re reuse that password on your favorite sneaker ecommerce site that person wants to get those sneakers resell those sneakers and there you go so a second example of why you would fall victim to credential stuffing is attackers themselves want the personal data so there's sort of this feedback loop to the attack itself where if you have access to the account you get to learn more about the subject you get access to their sneaker account you find out where they live you can then launch a more targeted attack and similarly from a workforce

standpoint homestead among us has not reused their work email on another service maybe you don't reuse your password as passwords con we don't reuse passwords here but you probably reuse your email and it's really great for lateral movement um if somebody's domain is octa.com that gets leaked i wonder where they work all right so let's put it all together i'm probably going to walk for this one hope that it works okay cool all right so an attacker would start with a list of passwords from the usual sources so you know um dark web uh whatever um public leaks and their own previous attacks so this is kind of the feedback loop that i'm talking about here um they would load

that into a database and either through scripted or unscripted means like humans and we'll talk more about that in a little bit typically through a botnet or something to obfuscate them they'll hit your identity workflow and then they'll get to your precious web application so separating out the identity workflow for the purposes of this talk is going to be at least a little bit important because that's the data that we're looking at right so i'm going to take the first break for discussion if there's anything about that that wasn't clear that you have questions about maybe it went too fast let's go ahead and take a second to talk about that and if any of you want to

volunteer kind of what against your organization attackers are after this is a great time to do that so um is there a mic runner was there a question my money all right your attackers are after money really glad to hear that but is it is it primarily a monetary objective in this room like uh is most of it they're trying to steal some material good or you know take money directly or what

okay yeah so what you're saying is that a lot of the attackers that you see they're kind of getting in dorking around figuring out what's there in the first place and they're not coming into it with like it right yeah so they're not coming into the attack thinking oh man i'm gonna get some sneakers out of this deal right they're just that's what you see a lot of okay great yeah so that gives me an idea a little bit on what to focus on as well um so thanks for your input on that gonna talk a little bit um about how bad things are and um how everything you see presented was generated so with only a few exceptions

um all this data is aggregated and generated by the authentication logs that we get as auth0 and the insight that we've gained as practitioners in the threat intel team so will you be targeted uh science point to yes so we did a look at quarter one of this year and saw how many of when i say applications um i mean tenants and we can break down to what that looks like um in the q a if you like but 48 saw seven or more attempts of credential stuffing against their applications so what will that attack look like to get a feel i'm gonna give an example here of a real world credential stuffing attack so the graph shows a number of

authentication attempts before that's in blue during that's uh one where you see the big red bars and then after where we've got blue bars again um of an attack against one of off zero's customers so during this time frame the malicious attack ended up being around two times the number of median login attempts from the previous i think six months um so i mean really depending upon what your identity paradigm looks like it almost looks like an l7 attack against your authentication right so extremely common attacks um that's generally what we see um how they work so what's the mechanics of how we get there i'm going to go through a timeline of a story based on a real world

attack that we saw and a little bit of insight from verizon's data breach investigations report the dvir so the dpr dvir um showed that over 60 of breaches were actually detected by victims within 24 to 84 hours the bad news of that is that they normally didn't detect it themselves they were disclosed by the attacker by a law enforcement agency an external cert team whatever right so in this attack that we saw first the attacker obviously stole the credentials we can't clearly know this time but the agency that we had worked with um had estimated that it probably was about 24 hours between the breach the confirmation of the breach and the first attack and in the next 24 hours the leaked

credentials were um identified by the detection response team of our customers verified by us as well and we were able to put the credentials in our credential stuffing protection workflows and within really just minutes of putting it in there we already saw use of the breach credentials only from that one threat group though we think you can never be sure about these things um over the next 72 hours we did um ongoing monitoring and really again saw it mostly coming from just one group but once we got to i put 84 hours but it really is 84 plus because it took about a month for us to see this in the wild just very broadly

used so that really i think gives you an idea from the amount of time you would see going from breach to opportunistic so um this leads into specif um sophistication and i kind of want to just really show how low the bar is there's a lot going on this chart i'm gonna walk over there and talk through it so what we see a lot of is like does the attacker even try so these dots down here is just normal um authentication traffic right this up here the size of the dot is um number of ips um in requests and what we're looking at is the number of unique user agents versus um the number of attempts right so this

attacker up here is using a botnet it's not a great botnet they're not even messing with the http headers right this is part of what gives us an idea of whether or not sophisticated group is using breach credentials and here we have one that is at least a little bit more sophisticated because they're trying to use user agents that look like humans right so really this is how low the bar is when it comes to attackers and their sophistication so i kind of break this into two personas here um i think about the opportunistic attacker we'll call him sam skitty and they are always using the older leaks a lot of times public leaks um very common leaks

so if you just do a count of all the times you've seen a email address for example and just do a sort that's what they're going to be using oftentimes they don't even try to rate limit circumvent very common controls and then ada ato we'll call them works on fresh leaks like what i showed before they use bespoke botnets we'll talk a little bit about what that sophistication looks like and they emulate human activity either by being humans or um scripting and other a little bit more clever methods so take a break here to pause have you seen credential stuffing attacks in your org what do they look like do they look like what i just

presented do they look totally different obviously i'm giving a very simple picture here because i'd really like feedback on this topic yeah and if the mic doesn't work i'll just repeat the question hopefully that worked out well got one in the back i've seen it tons of times but i'm not partial you know i'm not neutral in this one

hi yes we've seen my employer has seen creation stopping attacks and actually with a shorter timeline they are showing here and on the more advanced side of it so yeah it's real so it's real and kind of what i presented matches up with your experience more or less okay um yeah i'm honestly happy to hear that because as a researcher you're never really sure if you're on the right track until everybody else tells you you are or i don't know if you have low self-esteem it kind of works the same way so if there are no other questions i can just move on to talking about controls that might be put in place we are way

ahead of time so this should go pretty well um i want to revisit what i presented for before on what a slightly more sophisticated look from the identity side might look like so you still got the attackers aggregating leaks they're still using a botnet but in the identity workflow we've got the breach credentials detection and that's really what makes a difference right it is so important to have at least some kind of list there to some kind of tracking of the logs and analytics and have some kind of action taken after that so what we have are things like notifying users forcing a password reset sorry forcing a password reset some people do things like mandatory

education right where it's like hey i see this has been reused this is where it's been reused you may you may want to look into such and such a thing or try such and such a thing right so i think really the solution here is defense in depth though because just having the protection that i showed before is really not going to be enough so bot detection i think plays a huge role here so when i was speaking of more sophisticated attackers if somebody knows the area in which their target lives so back to the person attempting to purchase sneakers they would want to err on the side of using a residential botnet that is actually a residential ip

in the area of where the target lives this is a way of circumventing fraud controls downstream and it's a great way of circumventing bot detection controls because often these botnets are ephemeral fairly short-lived take advantage of iot devices or routers and and they just look really really real so i think this is an important control to have in-house to protect your identity workflow if you don't have it in-house then you might want to buy it just figure out what your budget looks like for that second is going to be rate limiting i i'm always shocked when i see how few attackers even try to circumvent this control i i really am and the linkage the

credential stuffing we always see is is very strong very strong a lot of credential stuffing attacks are people trying very hard very fast but we also see things like people sprinkling in attempts not trying too much again that's the more sophisticated attacker and rate limiting isn't going to work on that neither is something like impossible travel where you might look at the ip of somebody who signed in and see like oh wow you're signed in from norway and las vegas like that's pretty weird we're going to throw up a flag there if you have a more sophisticated attacker not going to work if you have a site where your users care about their privacy generally also not going to work

because they're going to use vpns probably in short amounts of sessions right so i so dislike impossible travel that i did not even put it on the slide but nevertheless it is worth discussing if you're talking about botnets and rate limiting and if you have the capability the next move would be to rule and risk-based pre-login assessment or assessment throughout the process so when you're looking at that attacker if you're able to do constant authentication using mouse usage maybe analytics about how they use their keyboards etc that's something that is an extremely effective control that frankly your id provider probably can't give you if it is an external provider right they're not going to see that

downstream that's going to be an application security concern and finally there's a reason i put it to bottom is multi-factor authentication there is a trade-off in this control that really depends on the market power that your company has to be honest with you i use some intuit products um not going to call many of them out but every time i log in i need an sms second factor which is a terrible second factor that i never asked for that makes using hey how you hydrated hey you know what i could be more hydrated maybe i should have gone with the stand-up routine or improv in chicago

i can't juggle i cannot juggle that

i go i go fast i do i get through you know what i probably can't i drink like a diabetic camel you requested you get it i didn't even put that on the request that's your mind reader um a flamingo dance would have been better but i'll take the water yes thank you thank you for the water appreciate it so multi-factor authentication honestly said don't like it a whole lot really don't but if you're into it and you have markets really cornered okay yeah what's

up but like you know you've got things like the email email section factor there's an option as well which is a little bit lower barrier but you know for obviously like uh sticking your own organization your staff members like surely it's impossible yeah well that's a okay let me repeat the question if that's okay so the question in essence is why don't i like multifactor that much and i'm going to put it more generously towards me um because i realized that saying you don't like multi-factor a whole lot can be a real bad look at a security conference but i also look bad so you know i guess they go well together um so i think it comes down to market power

um and where you are so with workforce mfa always do it um if it's not email or sms and something stronger that's great i really think you should do it right and so like the octa part of my brain does not agree with the statement i just made but with workforce you have to ask the question where are my workers gonna go i'm not gonna lose workers over this right but if you're selling sneakers and somebody closes that tab that is that's why you have workers in the first place so that's i mean i really think that is a core trade-off um i don't know does that make sense and more importantly is that like totally incorrect in your view

because i'm open to vote i think we're trying to protect against different types of things here my security problem your security problems are very interesting in my research yeah we can um we also have like enough time to keep things online too um that's like exactly we're actually i think maybe one slide away from the last so that is why i'm here in the first place um i really do want it to be discussion i think part of being a researcher is carrying on the conversation right the great conversation that's pretty much what we do like it or not in service of capital so you know basically though i think that's what it comes down to is a risk-based decision

and me as a user what i don't like about sms and email mfa being forced on me at every login is you've introduced friction with frankly not a huge value add in security and if intuit had any competition they probably wouldn't do it right they would probably lean more on risk-based controls or give me the option of what i want to use like when it comes to mfa uh they put water so i can't put my laptop up um it's really cool to have something where you can push a button and if you've got workforce and you have a postal service at your disposal you can mail them hardware security devices you can have them enroll that device

right so i really think the friction is really important um we talked about market power yeah you gotta ask what's there to lose what are they after in an enterprise you may have like a whole lot to lose i've worked for really really large organizations and if you're like one of the biggest banks on the planet like what is there at stake like don't want to be too dramatic but the world economy i think we've seen and what's the cost of implementing the control finally i think is really important like it comes down to organization size so show of hands how many work at an organization smaller than 100 workers okay awesome uh you just volunteered

yourself to be to have a question asked so from your standpoint of all the controls we've mentioned and maybe others that you could think of which do you have the budget to implement and how difficult is it

if you worked at a company 100 individuals or fewer what you know kind of what would you run up against right like we've talked about what what the highest roi probably is right so what what aligns with your budget uh yeah you're gonna need some friends you have 100 friends at best yes [Music] could you give him a mic i'm sorry i yeah when i hear mssp i'm like all right let's get on the record here so uh all right it's working i run an mssp and we run up to accounts not quite under 100 but 500 or less and a lot of them have that question of you know what do we implement what's the best roi um and

despite our best efforts not all of them want to do you know risk-based discussions and go through that and really pinpoint so we had to come up with like our essentials package which mfa is on there because in a lot of cases it's free with the tools that they're using um if not it's cheap through octa or duo or whoever security awareness training endpoint endpoint protection email security and then if they want to go one step further vulnerability assessments and all that but um that's that's generally those those are the first five controls that people will reach for and usually if there's like you know no more than um 500 employees that's usually within their budgetatory

grasp i mean at least for your customers right yeah you can see i'm a little you know kind of narrow-minded because of my my topic or my specialty mostly being siam but yeah i think i think that's great insight for workforce and it's good to know what they can afford and i honestly do agree if you're a smaller org and you're using a lot of cloud-based services it really does often make sense to just outsource that one way or the other right maybe there's this agreement there too but but honestly that's my view is that if you're not an identity company you may not be that great identity so that's just what i've seen

um yeah so i mean really that's it um if you want to talk about whatever protections you have in your workflows what you think works and what doesn't um if you want to get really in-depth because i know there's a lot of password con people that are like super duper smart about passwords and cracking them i would really love to hear your views on um did i leave it on i didn't leave it on on this thing right here so managing what's in that database securely actually doing a good job in your identity workflow of taking the credentials that are entered comparing it to what's in the breach credential database insight on that um of course would be extremely welcome and

um does anybody like have any thoughts on that or what they what they do in their own experience i can see you smiling under the mask yeah out of curiosity do you make a difference between credential stuffing and password spraying in this talk no [Laughter] um but do we track that yes right um of course and it's i don't know all the brute force sort of attacks have some kind of commonality um but password spraying is almost closer to opportunistic botnet activity i think um it's it's not very good it's not usually super targeted um but with credential stuffing like at least you know something your attacker has a starting point um but definitely tracking that obviously

extremely vital in any application i think at least is is looking for bridge credentials is that part of your service and is it integrated or is it an option that i can purchase additionally you can give us as much money as you like um i i don't want to plug services too hard because um i don't i want to be asked back [Music] yeah well i'm not asking for a quote most definitely not i do have an opinion on the use of uh services providing bridge credentials uh information from be it dark web or wherever you find it so yeah okay yes so i i will say yes to that um i i think i

can safely say that you know as you saw from the example we work really closely with our customers and they want that stuff in our breach password protection database too so we get a lot of good stuff from that um you know we buy data from lots of cti vendors as you might imagine when it comes to credentials um honestly we're not running crackers but i would love to hear the roi on that uh because i know there have been lots of great talks on cracking passwords here but i didn't i don't know if there's been a lot of talk of like okay here's the buy versus build like here's how much really you want to spend on having

a gpu or whatever that could actually crack you know certain passwords hashed a certain way or whatever um that's a discussion super happy to have and this is definitely the venue for it um but yeah i i know that more than like talked about your question but but i i hope that helps at least and um honestly if if you're not gonna buy it um you really should consider building it like honestly

yeah so maybe i should go back to our controls worth it because like is x worth it is always something of great interest to me and not just because i'm at like a third-party vendor now um part of like if you have principal or staff or director in your title is like being able to bs enough that you can like put that question into your budget ask so yeah i just wanted to ask a question so if you have a bridge credential detection why don't you notify people proactively and using the database to actually you know uh reset those credentials ahead of the time before the attack is happening i mean i understand that you can

use that database maybe to detect the attack itself and identify the malicious ip but as far as the users are concerned wouldn't it be better to notify them ahead of the time hmm that's that's a good question honestly so for example if we had an email address we know it was associated with a a credential that had been breached wouldn't it make sense to notify that person rather than waiting for them to log on to a downstream service that's really the heart of it right um i think like honestly part of it is is to have it in product um to be super frank but also when people actually try it that's what gives us confirmation that we're on to

something because if you're using that email and password in multiple contacts we don't know if that password's still like relevant you know maybe the leak was from 2019 like a linkedin leak or something like this so that's kind of i hope i hope this answers satisfactory to you but those are really i think the two biggest reasons that we don't proactively notify people who may have been involved in breaches um honestly said i also think that's a responsibility of the company that was supposed to be securing your credentials in the first place

no one to scare them off um yeah i suppose that's true

hey i just uh was curious bit about your risk-based decision making in siam um so what kind of practical implementations could somebody look at if they wanted to do risk-based policy for customer identity yeah so i think that it really comes down to where you sit in the flow so i want to say that right out the gate right so are you at the login box and you stop at the log on box or do you have the application um but for something wrist space i would really look at any multiple factor that you may have what risk may be associated with that so if a phone number is you is being used like for

passwordless or something really really helps to have some intel on that so there are lots of great vendors that provide things like this i would definitely include that as risk space we have internal machine learning models which you can actually read about tons that we have resources that i can give you after this talk i would recommend that if you have it right but like first step is is having that log data at the ready and if you've got the data um there are lots of predictive tools that would be at your fingertips are you up here because you want me to eat or you just up here to walk it around i have consequences for you i'm just

okay if i should ask them hello afterwards i'm i'm here all day and we have i think at least a few minutes and oh yeah yeah i mean i can wait yeah well it's an interesting topic i can tell you that we've had several talks at pastors.com before about protection against password spraying financial staffing and so on all talks are available online on youtube for free some of these talks were held in europe over the past few years so a certain possibility people in the us haven't seen them and vice versa but there's some great stuff in there personally i'm working for a financial services company or id provider in norway and when you use us there's a

mandatory to find on location one of the things that we do that i have pretty much never seen anyone else do in the world and i'm not saying we are amazing we're just incredibly good is the fact that when you log on with us you log on in an order that is different from how much most people do because usually when we're two-factor on application you do username then password and then an otp we ask username and then otp and otp is not a push notification to you it's a hardware device that you have where you get the pin and if you are not able to enter that you will never get to the password

prompt so credential stuffing and password spraying is absolutely pointless towards us it doesn't work it can't because you can't get past the otp without actually coming to norway and stealing a hardware device from somebody using bank id and this is something that you can do at you know if you have a ubi key social engineering you and i just yes all of twitter because i convinced you that i was part of it did i mention fido web authent as part of the flow of this so nope doesn't work but i mean this is something that you can do as easy without you know if you have a hardware key like ubiqui or a google titan key or something and you're

using secure shell you can set up a pan module in your linux box saying that whenever i ssh into my box enter your username and then they will ask you to use your ub key first and if you don't have that or if that doesn't work you don't get access i mean i set this stuff up like 11 years ago on my box at home when i got one of the first ubikeys made more or less and just for the fun i put up a box available on the internet with telnet and ssh available and i said use them first and then you need to authenticate using the otp so i just saw i don't know how many login attempts and

all of them fail because you can't get past the otp that's a brilliant way to solve this well not solve it but at least you're reducing the uh attack surface by huge amounts so it's a good tip well we i in my case i decided just to go for you know only do the ubiqui so no password at all but again i run passwords con i have a passion for passwords i'm saying they are never going away so i also say that while having a memorized secret in here as well is something they really should have and for those who doesn't remember what happened to rsa and security rita you need a password as well

more questions or comments from the audience oh yep yeah i want to say also i could not agree more um i wish in the u.s and internationally we had is it is it like a requirement in norway that banks issue that or is it just totally unique oh well that makes it easy yeah yeah your flow your flow is much better yes yes please so this is a not exactly a question more a comment because i didn't come here to represent into intuit but in fact i i knew there would be somebody in that sense so so we we do have a very intensive monitoring of what's happening with our customers and i can tell you

that the thing that you're you're talking about this mfa is something that we are well aware of uh this morning i just got here late because i was on a call exactly about this issue uh you know we have a lot many threats and many security mechanisms and everything that we essentially know what we have to do but you know everything that we're doing we're doing very slowly in order to not uh damage or open the open the door for for additional fraud so i i'm sorry for what you're experiencing but uh you should know that this is something that is being handled i accept your apology and i thank you for your service uh

[Laughter] uh yeah i mean that truly i wasn't honestly trying to pick on you there's other large us banks that actually do the exact same thing and it really really drives me nuts in fairness this is like in a small business smb context so i can see why they do it right like small business account takeover is a really really big deal and even though intuit like ultimately may not be like responsible or on the hook you still have that duty to your customers and and i appreciate that like i i do right um i was just being a baby about friction and i think that it's more of like a scion problem honestly so i don't know that's

my view i hope that's a little more generous um i have two questions uh one is uh but you said you didn't like the impossible use case or impossible impossible but i didn't um i didn't get well the reasoning behind or what you don't like about it and then the second is um do you have any good usage use cases for a situation where somebody an organization is using a soft token as their second factor and it's compromised okay um you'll probably have to ask the second one again because by the time i get through the first one i'll probably i forgot but um what what don't i like about impossible travel um if you are a

provider with a lot of different customers and a lot of different places with a lot of different use cases um i hate to give as an example but you know web3 companies that are startups really like to outsource their identity because you know they got to move fast and like not all web 3 companies are good at identity but what talks about impossible travel for them is that a lot of their customers are really concerned on privacy do use vpns and within the same session lake very well may change ips throughout countries but if you are doing something like workforce or finance impossible travel probably makes complete sense honestly so that's that's what it comes down to i

think um is like what is your use case what are your trade-offs where's your revenue um and so the second question ask it one more time um compromise soft tokens uh do you have like use cases to detect you know credential stuffing plus compromised software things so the question is around like what what situations have arisen or like what how how do you how to deal with it or

i'm trying to think as a login box um directly and i wish i had an answer immediately at hand um you uh yeah yeah yeah so my question actually ties into his question but um we use a siam um provider service called glue glue identity so i was going to ask for your comments on that but to speak to his point um glue is actually java based and you might have heard of something called log4j so um you know there's a tie in there did you want to comment on this question um so commonly this question i think that um you know ultimately downstream application security is probably where i would see that and i know that's not an

answer that probably satisfies anyone but that's the best i can give you yeah yeah so just wondering if you if you had heard of glue identity and if you have any thoughts on open source uh you know providers to help to solve some of these problems um i don't really have too many thoughts on them so for workforce protection for that use case uh potential could be some some providers use kind of like an 802.x like mechanism where you have an endpoint that's recognized as a friendly endpoint and so you can enforce that at your organization where you're only allowed to actually authenticate with a work device that's been proven to be a work device

yeah right yeah workforce is is very nice in that you have control over the endpoints or could at least usually though yeah well yeah uh in this world of remote work control is kind of a a weird concept in the first place okay then we're all done thank you for coming to peace rights and pass-through scope yes thank you for having me it it truly was a pleasure um i hope that i get some insults from you uh shortly after that we can do insults but now it's lunch time and i hope to see you all back at two o'clock thank you