BSidesSLC 2021 -- Anatomy of a Distributed Credential Stuffing Attack -- Speaker: Bryson Loughmiller

okay hello everybody uh my name is bryson lofmiller and uh this is anatomy of a distributed credential stuffing attack or uh botnets cred stuffing and password dumps a podium story um so my name is bryson i work as a manager of security engineering at podium which is a tech company over in lehigh we do interaction management for local businesses and for four years before that i was at adobe doing various things and i do i have a number of different hobbies at which i am mediocre so i've got a ton of information that we're going to go through here and not a lot of time to do it so this is going to be a speed run talk

and with many tangents so uh off to the races so to start off um i'm just going to go through some basic terminology uh brute force attack hear this a lot brute force attack is when you have a single username and you are trying multiple passwords against it trying to find the correct password like shown here in a credential stuffing attack though you have multiple different usernames usually with a single password usually from a password dump or a data breach of some sort and you're just trying each individual username and password combination hoping that you get something some success along the way uh botnet you've probably heard this term a lot generally a botnet is a group

of compromised devices all controlled by a commanding control server of some sort uh or multiple command and control servers depending on how it's set up uh usually each one of those bots is compromised via some sort of malware that is making a call home and whenever the command and control sends out a signal the bots do what they what the command control wants them to do uh so first tangent uh the mirai botnet so some of you may have heard of the mirai botnet this was one that was uh at its at its heyday in in 2016. this particular botnet was an iot botnet and was mostly comprised of iot devices like cameras you know

any any iot type device fridges uh toasters anything that's on the internet nowadays which is uh pretty much everything uh and so in this particular instance you've got your your camera that somehow gets weaponized and you've got potentially tens of thousands of them out on the internet uh all combined with the other iot things and they're attacking some sort of service so uh in in then you'd laugh at the minecraft but the mirai botnet was actually originally created for the purpose of taking out competitor minecraft servers so the original owner uh ran a ddos protection service and attacked his very own clients uh and hired out the botnet to take out minecraft servers so uh that was its original purpose uh in

october october 21st of 2016 uh they attacked dine dns which was a major you know dns uh hub for uh for many many many different services and uh took it out for 600 000 devices took out all of these different services twitter amazon netflix cnn all the things uh for five hours mostly on the east coast but uh caused significant amounts of panic uh and uh you know sparked lots of inquiry into this which led some of you may know who brian krebs is he's a security uh journalist writes a lot of really interesting stories about different security topics does really good investigative journalism on september 22nd he was hit with a 620 gigabit per second

attack not him personally but his website uh and the whole website was forced offline for four days uh and sometime later ovh which is a french hosting company was hit with a one terabit per second attack from this same botnet all from iot things which i think is crazy but uh so uh krebs after his site was attacked uh decided to go out and do a significant amount of research and if you've never read this particular article on on krebs website this is a very interesting dive but he dives into who this individual was and that's how we found out that he was running this this ddos protection service and at the same time running one

of the biggest iot botnets in in history uh and eventually due to some of the stuff that he put out uh the two two of the people uh pled guilty and i don't think they actually went to jail i think that was a recent development but uh but yeah they they were caught so all right tangent over uh so we talked about cred stuffing we talked about botnets what did podium see and that's kind of the idea behind this is back in december 2020 a podium saw a large-scale botnet-driven credential stuffing attack uh over the course of the month so uh to give an idea of what is normal for a podium at least at that time usually

we would see around uh 1500 unique ips hitting us on on any given day around 3 500 unique usernames in our authentication logs and then in the entire month of november around 44 000 unique ips and 61 000 unique usernames uh throughout the course of the month this is a gra our chart of our of our authentications you can see we've got major spike or a decent spike right here another decent spike here and then the mother of all spikes right there um so on each of those days uh we had 41 500 attempts in one hour 190 000 attempts in one day on december 22nd and then uh between december 24th and december 26th we saw 1.2 million

uh there were yeah around 26 700 unique malicious ips i'll talk about this a little bit more later but actually classifying an ip as malicious was tricky uh and they attempted yeah around 2. 2.2 million usernames um the reason this was interesting uh was that we saw this this initial attempt on december 2nd almost like a a poc of their of their attack uh and then that that second attempt on the 23rd like a second preparatory attempt and then the massive spike on christmas day uh over the course of christmas eve to the the day following christmas which i think is really interesting because whoever was doing this i believe that was very intentional as they assumed

that most of the individuals who were watching these sites were probably off on vacation and probably weren't watching as much so uh i think and actually anecdotally just last week over the thanksgiving break we saw more credential stuffing attacks during the thanksgiving break so i think that uh holidays and vacation times are prime targets for uh attackers to go after uh especially u.s companies us-based companies because they know we're out on vacation maybe not watching as closely so um okay so quick analysis of some of the ip addresses that we saw uh they were all over the world um the the attempts were coming from basically every every continent um and uh mostly out of the united states

or and then the united kingdom whole bunch out of russia china germany india netherlands canada just everywhere so um i think like i said before we had seen those 23 000 or yeah 23 000 ips uh what were the asn's the different organizations owners of these ip addresses again that also ranged we had 425 different potential uh asn's different groups that were owning these different ip addresses that were coming after us so a broad variety of um of ownership across these ips which leads me to believe even further that it was some form of of botnet um we plugged some of this information into a service called ipqualityscore that service allows you to throw it an ip

address and they have different fraud scores classify whether it's a vpn a proxy multiple different things based on some of their information that they have and when i threw all of the ip addresses that we had at it it classified 96 of them as proxies 91 as vpns but a lot of that is um classified based on just the the information that they've seen in the past and uh also from the the traffic that they see on their uh their honeypot devices uh so okay so diving into a couple of these colo crossing was the number one organization on this one uh this is a united states-based co-location provider scamalytics classifies this as a potential fraud risk and and when i

googled colo crossing one of the first things that came up was colo crossing abuse so i think they're relatively well known for stuff getting compromised unfortunately um i uh blocked out a few of the ip addresses here just because i didn't want to actually list potentially compromisable sites on here but um a couple of the ip addresses most of the ones that i went to go look at from the the ips that we saw so like for example this was one of the ips where we saw almost 6 000 requests 6 000 uh authentication attempts come at us uh it's just some random website uh and that was kind of the case across most of

these ips that i went to check out it looked like there were just different varying services run when i jumped on showden you could see lots of different ports exposed so i think that these are just multiple different types of devices across a broad spectrum that have been compromised and put on whatever this potential botnet was so this one was uh a service that hit us with 5000 requests again running some different service out of china um this and and there was a there was a broad range as well we saw some services hitting us some of these ips hit us with 6 000 usernames up to 6 000 usernames some of them hit us with only five

and so there is quite the the range as well and so with this one for example they only had they only hit us with six usernames and it was just some other type of service running on this so going across all of them it was just pretty pretty clear that there was a broad range of different types of devices uh that were hitting us uh that just further indicated that it was likely a botnet so was this a targeted attack on podium itself i mean i i really doubt it um we are out of the 102 sorry out of the 2.1 million usernames that were attempted only 102 of those were even podium users and out of that there were 92 failures

10 successes luckily we identified those quickly worked with the customers got those passwords reset and we reached out and notified those individuals through our processes i i think that i mean obviously this was a large-scale botnet with the the tens of thousands of ips that were hitting us um and with each ip not hitting us with a predetermined amount of uh of attempts it wasn't just like you had one hitting us with 500 and all of them hitting us with 500 i think that this was likely a more distributed attack across multiple different services so somebody grabbed a a bunch of credential dumps and threw them at multiple surfaces so i'm actually curious if anyone else

saw something similar at this time frame because my guess would be that a ton of other groups were hit with this we're not the biggest company in the world and so i i can't imagine that there weren't several other organizations that in your authentication logs you could go back to december of this time frame and see something like this and if anybody else saw that and would like to talk to me about it i would love to hear about it because i think this was super interesting um okay so those were the ip addresses where'd the credentials come from uh there's a ton of different potentials obviously there's password dumps all over the place one that i think is

particularly interesting is called collection number one this was a 770 million record collection that was released back in january of 2019 uh 1.1 billion different email and password combinations 87 gigs of data 45 which it's kind of a steal um the the history behind this one's actually kind of interesting there's a uh i don't know you'd call i guess a hacker named sanix and they were the ones that kind of curated this whole collection they released a series of different collections collection number one collections two through five number one was more credentials two through five was a lot of uh pii uh 25 billion records total a whopping 840 gig of just straight data about

people um and a fun little drama that went down here was sanex had a rival data broker named azatej and as it has just bought it and leaked the whole thing for free so uh they yeah there's apparently drama between your data brokers out there which is interesting and i think both of them were eventually yes yeah both were eventually uh arrested in may 2020 so uh crime doesn't pay kids uh okay so are are you am i or is anyone in one of those collections i mean most likely just from a from a sheer statistical uh aspect and also just being on the internet you are almost assuredly in one of these data breaches unless you are

completely off the grid uh so a very useful resource if you've never used it before havibinpone.com allows you to go in throw your email address in and it will show you whether or not you have been in a one of these particular breaches uh so what i did was i took all of the usernames from the that we saw attempted on us i took a sample of ten thousand and i hit the have i been pwned api uh and ran it through to see what information i was able to find and 87 uh of those emails have i been pwned was able to say yeah they they they were in some sort of a data breach and the api gives you the

different data breaches breaches that each one of them were involved in and you can kind of see collection number one was a very common uh collection number one was also referred to as anti-public but uh collection number one is a common commonality amongst the the sample that i went through um the interesting thing was that there were yeah around thirteen hundred that were not in the havon pwned database which i think is interesting because that means that there's a decent chunk you know 13 of those which obviously are breached credentials of some sort that uh have a binp doesn't have which um there's probably a good chunk on the on the dark on the dark web that haven't been picked

up by the the guy who runs that site yet but um just just goes to show that it's not a perfect means of protection um okay so jumping into some uh recommendations have it been pwned actually has a service a free service called the the domain monitoring the domain search so if you work for a company uh like like we work for podium and you have some sort of a domain if you can prove ownership of that domain you can submit that to have i been pwned and anytime uh one of your emails or anything to do with your domain pops up in a new data breach uh they will send you an email and let you

know send you an alert about it so this i would highly recommend if you have any organizations to go and sign up for this uh you'll get notifications when any of your users may pop up there um okay so obviously the biggest problem here is password reuse which uh just needs needs to stop if if everyone is not aware of why password reuse is a problem this is exactly why right your your password is leaked in any given number of data breaches and it's nigh and too impossible to prevent that from happening and so if you use the same password across all your different sites your other sites will be compromised in these types of cred stuffing attacks uh so i

would recommend get a password manager lastpass and one password are two good ones um they also offer dark web monitoring on both of those services where if one of your credentials pops up in a data breach somewhere they will alert you about it and let you know that you need to rotate that password so good services um brief discussion about detection so uh how are we detecting this kind of thing because it's easy for this to slip through the cracks in terms of the the just large amounts of data mountains of data that you might see in authentication logs one thing that's particularly easy or not easy but a good way to look for it

is to look for the high numbers of failed authentications with unique usernames so if an ip address has failed authentications on a high number of unique user names then that is very indicative that that iep is attempting a credential stuffing attack of some sort and then also just focusing on successful attempts because honestly these types of cred stuffing attacks are going to happen all the time right it's just it's just the nature of the space there are lots of things we can do to protect against them and defend with captchas and rate limiting and and whatnot but uh they're gonna happen so it's not necessarily especially if you have limited resources in your security team

it's not necessarily prudent to go and chase the rabbit on every single time somebody launches one of these but you do want to know if there's ever a success and so if you can identify a malicious ip based on a high number of unique usernames like that and then correlate it across to any potential successful authentications from that ip address uh you can set up an alert and and uh protect your your users so and then just overall anomaly alerts of course if you see large spikes like that then there's something worth looking into often uh okay so in summary implement password managers use your authlogs look for anomalies kind of where we were talking about

detection uh utilize services like have i been pwned have it been pwned isn't the only one but it's a good one implement rate limiting captchas on your authentication flows and be careful over your holiday breaks because uh i think it's a prime time for attackers to come after us so that's it any questions

okay thank you

you