New Talk Who Dis? - Jamie Hankins

i'd like to introduce a very very exceptional talk um jamie um i've known for quite some time through various parts of um industry i asked him he was one of the very first people i approached to submit a talk and he said to me what on um and i said i'm memes have you got any have you any great memes we can talk about and he julie obliged and he's lined up i think about 175 pages worth of memes and one page on wannacry um i don't know if anyone else is i don't know if anyone's heard of wannacry does anyone anyone hear the wannacry anyone sing sing calls anyone never heard of her um so jamie's favorite phrase is effing

wannacry because he loves it that much um but the content is absolutely golden so once again thank you very much over to you jamie thanks for setting me up hopefully i don't [ __ ] this up too much uh i'm gonna be talking oh [ __ ] gonna be talking about wannacry and kind of what happened oh can i not have these pictures please so that's kind of what the big no camera thing was about sorry um sorry no me too um okay so uh that's not too bad okay so who am i why the [ __ ] am i talking why isn't marcus here um you know um yeah so during wannacry i was just an analyst um

post arrest i became the head of friendship um conveniently yes um you know a lot of people probably don't know who i am why i'm here blah blah blah i worked with him for nine months before when i cry on other bits and pieces some cybercrime stuff mirai other [ __ ] um and then during one i cried it was me and marcus who were kind of responsible for running it um liaising with law enforcement intelligence community uh certs everyone and their mothers um by like today i work in the front and tell but by night i'm a drunk [ __ ] poster um okay so

definitely my favorite meme um so i was in vegas the day before the arrest um this is what the villa we were staying at it looked like can confirm all right so me and him gave a talk two years ago at a conference called cyber hagen uh we're kind of talking about bits and pieces but um so yeah we kind of we're like we're not going to talk about this stuff and we kind of went for it but i don't think we told the right story at the time um but also like i'm taking this slide and there's one other thing i'm not going to be talking about um it's hard to talk about wannacry without

talking about marcus so i'll definitely be talking about him and you know the awesome work we did during that time but like his legal case is very much something i won't be discussing uh like no questions at all on that place so i figured maybe some people actually didn't know what happened so i built a kind of [ __ ] timeline um so microsoft released a patch uh for this thing and then it got leaked kind of a month later uh the kind of uh the background on that allegedly is that the nsa tipped off uh microsoft but who knows um 12th of may that great day where like everything blew up and then three days

later when xp got the patch so i started this talk off with a lot of memes but i want to say the context um so for the next 50 or so slides there's a lot um i'm basically talking about like my stressful two weeks of like our lives it was [ __ ] um i also wouldn't go back and like change anything because it was awesome but like i wouldn't do it again so what you're looking at here is virus total intelligence is kind of what the industry is for um tracking like samples um so we've got a laser [Music] here that work cool um so here is the first submission date which is when the sample first appeared

on virus total and you can see like kind of 7 32 7 31 is when stuff started appearing most of this is from like api driven stuff so like you know if you like get a snake oil vendor and they're like put a box on your network um that box probably uploading all your [ __ ] to bt no all of them um so [Music] yeah sometimes it's really useful and sometimes it's like oh now i can read all your internal emails chairs um so during the next kind of eight hours wannacry spread around the internet [ __ ] everything up um no one knew the true impact some people on twitter were like this

doesn't look good you know maybe we should care about this um and we just registered domain names because it looked [ __ ] cool um so we'd want to see what was calling out to those domain names so that's what happened in this case um 305 kevin beaumont or gossip a dog posted a fred on sis does anyone here know what cis is oh well that's like more than one okay so ncse are great but cisp is a bit trash at times but doing wannacry was awesome um we were able to coordinate with like everyone in the uk about this thing it was really cool um also as a side note if you ever deal with logging systems or

incidents centralize on a time zone because otherwise you're on a world of pain um we now use utc for everything but during this incident we did not and it was painful here's wannacry somewhere it shouldn't be i think this is an atm but i'm not sure because there's like some weird [ __ ] on it um the left is definitely an atm and then on the right that's a german train station um more places one i definitely shouldn't have been and it also shouldn't have been on that sign [Music] now i really struggle to find pictures of it in like hospitals and stuff um but this one seems to be legit and it's a picture of it in our radiology

department i think uh not great um i went searching on twitter and found these really good tweets just about the impact it was having a basic thing don't come in it's like useless um this reddit post was awesome so me and marcus didn't really care it's not that we didn't care but it wasn't a priority to us marx was on holiday and i was [ __ ] around with like cybercrime stuff um until we started [ __ ] with the nhs and you know like like compromise a random organization in the uk who gives a [ __ ] compromise like the national health service and people really start paying attention um so this is my like favorite quote from

that reddit statement yeah i am when i sorry when i was um speaking to um uk like intelligence community i definitely sent them that a few times um so at 308 marcus registered the domain name um it's like 4 p.m in the uk uh it's important note yeah at this point it was time to re-only we didn't know it was a kill switch we just wanted to know where this was coming from he didn't actually have a look at the sample either um he just saw the domain and was i need to get to this um which is what we would do today as well so for the next couple hours mark has kind of worked um independently

verifying what he was seeing and the data um we we had quite a bit big like disconnect it wasn't uncommon for like one of us to be working on something and the other one to be working on something else totally separately um so it wasn't until i got a phone call from someone at the uk and csc saying did you know marcus has the domain name and he's working on a pp map i was like that's [ __ ] awesome um literally the first thing he did was a puppy map um we didn't know what the hell to do so we just tweeted because it was awesome um like we had the domain name we're not an av vendor like we we don't

have to deploy patches or push out signatures so let's just shitpost instead um you know it was a massive smb worm spreading around the internet it was pretty bad uh and pretty scary as well um so i managed to find some really old conversations between me and marcus um and so in this case he's asking me to check the sample to see if there's anything in there that's special um that's because a lot of the cybercrime stuff we monitor uses a dga or domain generation algorithm and that means it would rotate out the domains very regularly so like some djs you have to register something like every three days and you might need like 10 or 15 domains every three days just

to have like full visibility it sucks um so 6 28 p.m i was like i think we can stop this we'd already stopped it by registering the domain and having it resolved um yeah we didn't really know what's going on um so this is what i was looking at when marcus asked me to look at it and and this is very very very very very simple apart from when the like fate of the nhs and you've got like the government on the other side of the phone going what's going on normally you expect the government to be giving you the answers in this case it was us giving the government the answers uh it was [ __ ] um

ah of course that doesn't work so here's kind of the biggest function the main super really important function and it basically just opens a website um what is really annoying though is see it returns a void um but if you read the docs it actually only returns null if it was uh unable to make the connection if it can make the connection it returns a handle so here's kind of what it looks like uh basically you've got a buffer copy the url into the buffer you create like a new internet session and then you take your internet session the url and some like stupid flags because microsoft and then if there's a handle you close your

handles because you don't want to handle leak and then you just return zero but if there was like no it would do this like service [ __ ] um which would blow up so at this point me and marcus were like argue i'm not arguing but we're like have we fixed it have we not like arguing over the output of this like ida output because neither of us could really concentrate you know i i just sucks at the best of times um so yeah i didn't do a good job at communicating this with marcus um our priorities though were on point um so [Applause] i i was on the phone to someone at the ncsc and and they were like what's going

on i'll walk and talk like starbucks closes in 12 minutes this is gonna be a really long night like [ __ ] you i'm going to starbucks um no one knew what was going on like i said like the nsa the sky news we're calling this a nsa backdoor it's as if it's that easy um and all we were seeing was like the media twitter and our like the [ __ ] pupae map just lighting up and so each one of these represents a prevented infection um yeah we were just like we need to keep this up oh a blank slide oh no here we go i've been shaved down the front because they were going to open me up kneel by

mouth since this morning and then at harvest one the surgeon turned up and said unfortunately we've been uh hacked and there's nothing we can do we can't operate on you today [Music] there are a bunch of cows sitting behind their screens trying to disturb people's lives i had a lady today who had a severe back pain which potentially could paralyze her account below this we had to divert her to other hospitals i really like the scary music significant issues with rit systems we can't access any patient records results prescription requests so we're here the surgery is open um but we've got very limited services that we can provide for people uh oh

[Music] yeah i mean at least it told me instead of just turning off because that would be a pretty nice thing oh and i've killed my mind yeah this is all part of it and i broke my mic

apologies for your ears let's fix that first [Music] [Applause] all right cool so it was like pretty late um pretty tired at this point so sleep is like a really major thing during wannacry um so let's get this shitty animation so on the first day of one and i was awake for 19 hours the second day 18 hours third day 19 fourth day 21 and on the fifth day i finally slept 15 hours um this might not seem too bad but when you're sleeping on your sofa with your macbook and everything not on silent waking up every time you receive a text skype message slack whatever you don't sleep um yeah didn't sleep in my bed for like

five days because i just wanted to stay awake for this entire thing but that's obviously impossible um next day saturday this is kind of when it's only really just beginning um so i mentioned earlier we tracked mirai or we did um it's like shitty side but it's not even cybercrime it's like children with uh barnett's some of the actors with it were quite sophisticated though and some of them decided [ __ ] marcus and went for the kill switch um it's kind of our reaction to press um accidental hero something that went around a lot but this wasn't really important for us the press was scary and like it kind of communicated how bad this really was to

us but what was really bad was we were seeing legitimate requests mixed in with ddos attacks in our access logs um they were everywhere and they just didn't stop um this is our internal telemetry telling us about the attacks we're seeing my favorite one is this one where they attacked the intel site with the path of crime or [ __ ] um they weren't fans of marcus so yeah we're seeing this being attacked and the kill switches and basically everything um more news the news just didn't stop it was a huge media cycle for them but the impact it had on us was kind of it made everything worse because we were always more aware of what was going on

um which sucked so this is a conversation between me and the co on the saturday where i'm communicating that if our sinkholes go down or something breaks everyone gets infected and he's like who's watching this i'm like the entire world um and like it felt like that very much at the time um i said you know like if our servers go down we can spin up more uh we were using the data for telemetry slime metro was really important because we were sharing this with law enforcement intelligence communities the certs people like that to find the devices and get them off the internet and patched um but he said you know at this point it's

it's worth more just keeping it alive for certain than having that telemetry um so we killed it at that point now this is sky news on a saturday [Music] this is bbc world news today i'm malcolm battell our top stories after the attack the investigation europe says a lot of people were calling this the nhs cyber attack scale a hunt is now on for those responsible we haven't identified the offenders at this moment in time but we are deploying all covert and over means available to us what does that even mean

[Music]

by next week 11 names are being considered and most important

so um i mentioned ncse a few times they were [ __ ] awesome um they were doing welfare checks for me and marcus you know ringing us up making sure we hadn't like jumped off the roofs of where we lived uh making sure we're okay offering us support um they were really good um yeah i mean like there's a blog post on their website with like mauertek as the author and the rest is like the authors like all government people with like first name last initial it's really cool um sunday is what i call [ __ ] the press day um this is the day marcus got doxxed and you know like regardless of what you

think about him and like his case and stuff the fact that like the uk media went after him like this is really [ __ ] um i actually had a [Applause] so i luckily didn't get wrecked by the media but i had an escape route um so i had like a side door into my flat and i was gonna go like go out the back and like someone was gonna like come pick me up and like just stay there for a while um it was a genuine concern for us i think marcus was actually jumping over his like the back um garden gate to get out because there were so many like press in the front

um i went out on the sunday and bought all the papers it's really weird having your boss on the front page of all the major like news um like yeah marcus's mum reads the daily mail and it's like why is my son on the front page um so we had that that kind of pressure and then we had more ddos more and more and more it got a lot worse um this is actually from uh russian ip addresses and there's this like whole meme about attribution and ip based attribution but all we were seeing was like russian ip addresses slamming the kill switch this wasn't like mirai this was like an actual like real botnet with some actual

power causing actual outages um i don't think there were too many issues but we were seeing like actual sinkholes go down uh and we just kept replacing them it was like whack-a-mole uh we weren't sleeping we were just watching this [ __ ] happen in real time uh more ddos um so much ddos [ __ ] ddos um by the 15th it was like how many more days of this [ __ ] that we have like how how long until we have a solution like how long until it stops and obviously it doesn't stop um during the weekend though infosec twitter glorious infosec twitter i had noticed this on an av vendor's website and it was the nhs is totally protected

with insert your av vendor name here i'm gonna yeah can neither confirm nor deny um yeah and then they actually had someone come in during the weekend to update their website like they they obviously cared um now this is where things get really fun we love law enforcement they're great um they seized two of our servers in france um about two hours before we got this letter um don't know why never got them back cheers for that um so here's kind of how the kill switch worked so we registered the main donate domain name with namecheap and then we had the aws route 53 name servers set on that and then they were pointing to some

ec2 instances and some ovh canadian servers and franco's the french ones are the ones we lost uh yeah cheers for that um on top of that we had nginx php fbm and php uh us our sinkhole stack [ __ ] sucked um not we weren't configured for this we weren't ready for ddos attacks you know we were just like sinkholing some shitty cybercrime and like apt stuff um so this is kind of a kill switch code at the time so we get the uri the user agent the host name and then we checked the host and the user agent and we'd play some music if you were a human being uh how do i get my mouse

so we have this playing in the background while intelligence agencies were investigating the sinkhole so they'd see the malware go to the kill switch be like what the hell is this why is it playing this music oh maybe the actors are into this um yeah marcus did that it was [ __ ] great um it's like the rest of the code so we get source ip destination you know all this kind of stuff you use for like detection uh and we play into a udp packet and then we'd send it to a sinkhole and we'd respond with this kind of sinkhole tech where the bots party harder and the research is harder um found that out in vegas

so on the 15th we were getting sick of ddos and we'd already kind of spoken to the ceo about losing telemetry over kind of keeping this up um how many people in the room do you think could keep a website up a static website uh against no matter what 100 of the time okay i'm glad no one put their hands up because i still don't think i can do it um so to this day we're still with cloudflare um we're on their enterprise plan for free uh enterprise registrar they are awesome we wouldn't be sleeping without them um they actually developed some custom software to ship logs from their end to us in real time

while the incident was ongoing they didn't have this they wrote it for us we used it for a year and a half until they developed it into an actual product and now we use the real you know good version um but yeah like huge props to those guys now at this point everything's kind of a blur uh i've not really spoken about wannacry for like two years so when brian was like you could talk about wannacry i was like really um so i had to like go fishing out of every i actually spent like four hours one night trying to find my old twitter logs um couldn't find them which is probably a good thing

so because kind of struggle with data um we've got some really nice data cisco umbrella provided us from the time and you know it kind of you can see it increasing and on the 16th after we moved to cloudflare we got absolutely hammered by a dns attack um i think it would have gone down if we weren't with them when we had moved now this is kind of the statement we made over time basically 150 000 unique ip addresses reached out to us in two days but because of the kill switch and how it all works you've got things like gnats and all sorts of [ __ ] um and basically because of that you can see

multiple hits per an ip address and from there you can kind of explain the number of hosts behind it but it's not hard to do accurately because you know you've got things like people rebooting their machines so we've never been able to like properly do this um yeah here's uh yeah that works so here's kind of first day um it's not too bad none spikes still have no idea what expect um it's the first day as well uh we had some like issues with our kind of telemetry um so the although this sinkhole stayed up the sea plus like infection processor we had at the time didn't and so that crashed which meant we lost data because there was no kind

of buffer um this is terrifying so what you're seeing here is wannacry's ability to [ __ ] you up even now um so on the first day there was one here the next day 26 the next day 24 day after about 41 22 17 20 15 000 hits suddenly um if any of those fail that machine gets infected um it's horrendous and this is still a valid issue today um we keep it up we see millions of requests a month like too many it's pretty bad um another thing that happened math's got free pizza obviously um and vice obviously wrote about that [Laughter] um he actually didn't get to use it that much it's kind of a shame i actually

started hashtag pizza for two sec for you um it didn't really work um some people made pizza memes as well and they were terrible but at least people were making memes to support the effort uh it didn't work so this is why i get the phone call about his arrest so me and marcus around vegas allegedly for defcon and blackhat but i was just getting smashed um um so i was flying out the day after marcus so i get a phone call ceo's like oh i'm with the fbi can you remove marcus's wannacry kill switch access i'm like what the [ __ ] like why would i need to do that it's like oh yeah he's been

arrested like the [ __ ] um so because we had to move his access a lot of kind of the the graphs and stuff were all tied to him and so when we moved his access the graphs went dead and so i had to like time this with the indictments coming out so that people didn't start freaking out over the kill switch going down um and i had to tweet you know saying yeah it's like because of this not because it's dead uh what's really really really weird north korea tried to move some uh some bitcoin the night he was arrested i have no idea what that was about but i'm i'm pretty sure it's just coincidence

um so yeah mark is still banned uh don't really know who's responsible for keeping him banned i guess it's me like yeah maybe the fbi would arrest me if i gave him access um national audit office in the uk did uh an audit into wannacry this is pretty awesome um so 34 trusts were infected 46 weren't but were in fact uh reporting disruption this is because of things like shutting down their systems because they're worried or maybe shared systems that had gone down or were infected there were 21 trusts that tried to call out to the kill switch that weren't locked as far as i know these are 21 trusts that weren't infected because of the

kill switch which is pretty [ __ ] cool um 603 kind of like nhs things uh 595 gps um no nhs organization pay the ransom like thank [ __ ] for that uh so 19 000 appointments cancelled um what's really cool is although we weren't actually name checked we kind of were name checked in like an official government report talking about uh the kill switch which was cool nice a day on this this is from this month idiots are still trying to take down the kill switch like actual government employees see and like kill it it's bad guy infrastructure um people need to change the way they think about infrastructure and iocs we had major

issues with the kill switch domain because people just spread isds and it's like blacklist that um don't want to say how many things were probably impacted by that that's probably quite a few now earlier this week uh otherwise known as [ __ ] verizon um i woke up tuesday and the internet was kind of not working um and i know someone from cloudflame was like hi does anyone know anyone on as blah blah blah it's like why and i noticed there's a really coming from their network and obviously us being with cloudflare um it's just kind of worrying so this is the above graph is what um verizon no this is what cloudflare's global traffic look like and at the bottom this

is what we saw and we compared this to the week before and it doesn't look like we're impacted um basically the ip addresses that they really leaked or the prefixes didn't include the kill switch ips so we got quite lucky there if they'd [ __ ] this up though requests came in during that outage so that's potentially 222 000 unique machines that would have been uh affected 37 000 unique ip addresses in that time uh it's pretty bad we learned a lot of lessons from this so we assume people will [ __ ] things up systems will [ __ ] things up uh law enforcement will attack us like everyone's gonna attack us uh we we know we have to scale everything

infinitely you know like php isn't going to work at this scale we need like something reliable um learning to communicate and running incidents me and marcus had never run an incident before we'd never done incident response we'd never done anything of this scale um so learning to communicate also learning to like rotate your people out and making sure that there's like enough info there that you can swap someone out at any time like me and marcus had died i don't know who would have taken this over like hopefully ncsc or someone um in the kind of in the five days of wannacry i did 92 hours of work um it's not fun wouldn't recommend so

what are we doing now this is our infrastructure now we wrote our own kill switch stack which is like go micro services and then we encode that data send it to another thing and we put kafka everywhere now this means that all of this [ __ ] here can die and it's fine we'll still have the data kafka is [ __ ] awesome if you have any real-time systems i'd recommend putting kafka somewhere in between um here's what we're working on now this is victim notification and we're kind of trying to notify all the victims of wannacry and all the other botnets we track and we do this for free we kind of realized post wannacry that

monetizing victim data is not right and vendors shouldn't be doing it [ __ ] them if they're doing it [Music] [Applause] so for wannacry we saw 58 million hits in the last 30 days from 1.6 million unique source ips from 208 countries this is a global issue and it still exists um thanks for listening to my talk um oh yeah during this talk we responded this is based on last week's statistics because of the uh there's some backlog system for wannacry so we can do in real time but for last week my talk if i'd done it last week would have responded to 18 000 requests from 6000 ip addresses in 122 countries that and

thank you very much jamie okay you're going to stay there for a couple minutes because i'm i'm going to guess that there's a few questions so first of all someone's saying no right okay okay okay just just give me one minute with jamie i personally want to pass on thanks because i think everybody in this room has got some thanks to pass on to you for the work that you've done the past couple of years so if you see jamie tonight he likes diet coke and vodka just just make sure his arms are full of daikon vodka thank you jamie