Honeypot Boo Boo: Better Breach Detection with Deception Inception

i'm gonna turn the mic over to our speaker justin thank you kat and thank you everyone for being here i'm really excited to deliver this talk last month i gave this talk at rva sec and it was really well received and we had a lot of fun and we're also going to have a lot of fun today so my talk is called honeypot booboo better breach detection with deception inception because it's fun to be wordsmithing uh so a little bit about myself i'm a authorized partner with things canary i don't actually work at things canary but i've worked with them for several years and collaborated and i'm very passionate about their product and platform because i've

personally seen it prevent three catastrophic breaches trip up some of the most elite red teams in in the world and have pen tests fail within eight minutes and easy cheap effective stuff and so today i'm gonna give you an overview of what we're doing now as far as breach detection why it's not working and some ideas for what we may try to do better moving forward and also walk you through various deployment scenarios for different types of canary tokens and honey pots and with that let's kick things off so for starters we need to talk about what's a honey trap or honey a honey trap can be a honey pot or a honey token and

a honey pot is a device that's designed to look like a real service and a real system that's running attractive services and you have tokens which are digital trip wires that can be manifested as google docs aws keys slack tokens you name it both of these mechanisms represent the larger discipline of deception technology and this is really fun because the more creative you are the more likely you'll be able to thwart your adversary and at some point you're going to have a situation of psychological warfare where you're going to get them to question their sanity and we'll get to that here in a bit the reason we need something new is because it takes entirely too long to

detect breaches it's costing more and more money every year and exponentially breaches continue happening so we also have tons of burnout you know our the security community is in dire need of people millions of jobs depending on what report you read you have burnout um you have disaffection and there's better ways that we need to be looking at this to help help our security professionals and most importantly if you do get breach because you will it's not a matter of if it's when the idea is you want to mitigate the blast radius because you want to know exactly at that point that you've been breached and not 212 days later so some of you probably know

the pyramid of pain and laundry room viking you know for the analysts out there this might be so so true it hurts but you know the idea is early on you know as an analyst maybe you work 100 hours a week and you're not going to go to bed so you just triage tickets instead and you might get ahead for the time being but you'll soon end up dead so in other words we need to help our analysts because it's really hard and i have a lot of respect for people but a lot of the time they're spending a lot of their uh cycles on false positives and you know alerts that don't have any fidelity so i

feel for them and really like why deception technology of all things i do believe this is the year the inflection point when deception technology really has started to take off because people need a new way of thinking about this and you've just looked back over the last decade at some of the largest breaches you know going back to 2013 with snowden and he had the most sophisticated spying agency and security organization in the world and snowden just rummaged around for long periods of time exfiltrated data and they were none the wiser you go back to solarwinds we all have ptsd in the supply chain at this point everything is about third party risk and trying to mitigate that

you know the effects are still being you know felt today and then you've got a credit score which is every adult here you know you were affected by the equifax breach and now you get credit monitoring and freeze your credit all of these things you have this synthetic identity theft problem that's been popping up and at this point it's like every other day there's some kind of breach and the worst part is things are converging now into the physical world um so we need to do something to to stop the septa pain so if any of you remember jimmy mcmillan from 2012 he ran on a platform about the rent being too high and he was definitely

right but the cost of cyber crime is is also way too high you know 2021 costs about 6 trillion dollars to the global economy the prediction is in 2025 it'd be about 10.5 trillion roughly every 39 seconds a new business is breached fbi last year had 847 000 cyber complaints but i'm here to give you some hope and hopefully a solution on how we can fix this if you look back at some of the largest breaches as far as the number of records go you know it took 212 days on average based on the ibm report to even know that a company's been reached that's entirely too long by then they've run off with the crown jewels and they've

done everything they need to do so if you look at cam4 what's interesting about that service it's a naughty webcam site but 10.88 billion accounts there's only 7 billion people obviously people had multiple accounts because they probably didn't want to reveal that they were on this site but yahoo got breached multiple times you know yahoo sports yahoo fantasy 3 billion and in india in think 2017 this was their government id program that was had you know lofty expectations but was a epic failure because every single citizen of india had their data leaked and the worst part was 401k was tied to their social security to their bank account you know everything just in one identification system so their life is

effectively ruined at that point so with that said let's talk about how canaries fit into this equation you know back in the day until 1986 they'd use actual birds you know when they went mining as an early warning system and that's no longer cool cool but we can use them in our code and canaries they come in two forms you've got honey pots which are known as birds and you got honey tokens which are canary tokens and birds can birds manifest as real devices you can have a scada system a smart coffee machine a dumb terminal database server and these things are so real you'd be hard-pressed to tell anyone that this isn't an actual device

and i'll show you uh what just how convincing this looks here soon and then you've got canary tokens and this is where the creativity is your only limitation because you can literally token anything you can take a piece of data text file dns record spreadsheet even qr codes now and i'll walk through each of these examples and the idea is you layer them in a sort of deception inception model which is where everything will kind of culminate at the end and you end up essentially creating a minefield that's hard to avoid at some point an adversary is going to trip over something and then they're going to trip over something else and along the way

we're going to gather a pretty detailed forensics trail and when it's all said and done we may not have all the information to de-anonymize them but if we provide it to a third-party company we've given them like a huge advantage and you'll see how this works so let's talk first about the birds the bird's word let's talk about from window to the lols i don't know if any of you remember microsoft bimbos it was a japanese computer repair shop that also had a modified version of windows xp that was really secure because all it had was sticky notes calculator and notepad so the attack surface was pretty small and you know a lot of people

organizations that are over 10 years old probably use active directory and active directory runs on domain controllers and the reason you use this is you want to manage your users machines passwords files you name in a central location this is really attractive to an adversary if you can compromise ad normally you can do everything you can read emails log into any computer listen to traffic so of course people are going to want to come to this so the idea here is we deploy a bird that actually is legit not like this and we want to see what happens when this domain controller is interrogated and you'll see so for those that know nmap or don't

it's a classic tool for service enumeration and port scanning so what i did here was i just ran an nmap scan with various ports targeting a windows domain controller right and if you look at each of the service ports you've got very convincing information about the services you know you look at port 80 it's running windows internet information services in fact if you were to visit that ip on port 80 you'd hit an is page and it would even let you put credentials in too it's you know it's so hard to tell the difference you can customize the directory information if you look here under 88 which is kerberos which is used for authentication you've

got realm.kdc and it says ad.thinks.com you can make it be b-sides pgh whatever you want for your company to make it blend in with the rest of your systems and even look at you know like microsoft sql server it's a dumb idea to run that on the domain controller but people do it all the time and even tells you what version you have and so on and then the moment you run this scan a lot of things happen you it lights up the christmas tree for you no need to do it yourself so on the left side here's a bunch of aggregated alerts that you're going to get not only do you get notified that there's port scanning

happening but you're also going to be notified that somebody tried to log in with ftp credentials and it'll even tell you what username and password and they can try all day but there's nothing of value on these birds you even have microsoft sql authentication so if somebody's trying to log in with sasa which is common you know it's going to fire an alert and even custom service ports you know if they run like a telnet command or netcat it'll capture all this information on the right side you can see kind of more information you gathered the user agent so this is already helpful right we know now that whoever visited is coming in on we're using nmap on a windows machine

essentially and we also know we don't have their ip yet but we will actually we do have the ip look at the there's a slack alert right there it tells you the ip so you can start to do some reverse geo location this is actually my ip in richmond and this is our first warning system and it's only going to get more robust with our information profile over time ted stevens he was he was not exactly right when he said the internet is a series of tubes it's not a big truck i mean it's not a series of tubes it's not a big truck it is mostly noise and specifically it's just grey noise and if any of you

have used a service you know that it's really a great way to gather contextual information for birds that you want to deploy in a traditional research mode so the bird that previously was deployed is essentially a production hunting pot and this differs from 20 years ago when honey pots really just sat on the internet and listened for signals and there wasn't really anything interesting to you in particular but what has shifted is these birds that you deploy are in your environment normally one hop away from your public infrastructure and so they're going to only alert you when it's relevant but that said canary has a traditional um honey pot called the outside bird and i

wrote like a very basic bass script on github that pulls in the alerts the ips and it runs it through grey noise and it gives you context so essentially what grey noise does is it says we know showdown has these scanners we know cloudflare has cdns we know all these ips we're going to filter them out we're going to filter out benign malicious and we're going to leave you you know these 4 000 ips will give you 900 so it's a great way to automate for analysts the process of you know manually going through and checking all these ips free service community api highly recommend checking them out and another native integration karen aryan rumble goes together like

george foreman grilled tilapia for sure and what there's there's a native integration so when you're in canary you can hover over the hourglass and it takes you directly to rumble what's neat here is rumble automatically builds an asset inventory and gives you a lot more enrichment and context so it tells you you know services location the autonomous system number on the internet um when it was seen and so you can determine okay did this canary token fire from one of my assets if if so i should probably look into that you know like this was really helpful for a log for shell you could start gathering information on public ips and if machines were communicating you know to

c2 servers and surreptitiously you'd probably you know need to investigate that another free awesome tool hd moore who wrote metasploit um this is his company it's awesome highly recommend checking it out okay with birds taken care of let's talk about the tokens and the tokens all the way down and we're going to start with chris farley and black sheep and recon tokens these are going to be tokens that people use before they even have access to your network and there's ways that you can gather information on them to figure out what they may be wanting to do and why they're targeting you so one of the first tokens we can use is a domain name system token

and the reason domain name system exists is because most people aren't weirdos and like me that memorize ip addresses you want to type in you know b-sides pgh behind the scenes you've got an internet protocol address that's where you know you know how to route traffic on the internet um the way that we can use these tokens really effectively is we can create dns entries and map them to unused or dark network segments so you can create entire virtual lens that should never be used and you can have dns pointing there so if somebody is enumerating those network segments it's going to fire this token and you're going to know snoop dogs in the building

snooping around and when this token fires it doesn't tell you the ip of the host it tells you the ip of the dns server but even with that it was able to determine that i was coming from richmond virginia where i live and i was using cloudflare their one one one two dns server in particular so this is like a great initial token just to alert you that people are mucking around and trying to do reconnaissance so this is really helpful and if we continue along that path one of my favorite tokens is called a web redirect and you can get so much information from nozi o'donnell just by having them visit a site so there's a doppelganger domain

that i bought and transferred to canary called thinks that io and when you would hit things that i o it would quickly redirect you and when you actually buy the service it doesn't say canarytokens.com it uses your own domain so that it looks legitimate but you know it sends you there then it redirects you to b sides and in the process it grabs tons information it grabs your browser user agent ip if you have javascript windows you know after a while you can start to you know shrink the pool of potential people this may be so like hardware surveys get captured all the time if you use steam for video games they always ask you send your hardware

specs microsoft asks you to send your hardware specs so there's a good understanding of how many windows computers might exist here in pittsburgh and then how many people use firefox and you can see how you can start to de-anonymize and have a more targeted forensic pool than if you just try to cast a wide net so this is where we're starting to build the trail until we get to the end and that's when things get real okay downey dodgeball fans out there there's a token here called a clone website token this is actually incredibly helpful for knowing when people are trying to impersonate your your company or they're trying to start a phishing campaign so i've actually

seen this token be effective in allowing for a cease and desist of domains you know like you get notified hey somebody's deployed this contact the registrar and you send them a scary letter with your legal team nine times out of ten they're going to tear this domain down and kind of give up on their campaign this is really helpful all you do is you create the token it gives you five lines of javascript you don't need to be technical just embed it in your html pages and then if the client redeploy it anywhere fires an alert get the ip you get you know some basic headers and yeah this this is good information bert macklin

is on the job apis alert protect investigate or application programming interface but now we're going to talk about api specific tokens and how great they are starting with the the cloud the aws so probably my favorite token is the aws amazon web service api key this looks just like a pair of normal credentials but when you use them you can't do anything but you also don't know that they're they're not permission you can though actually lead people down a path and you can provision credentials that have access fake access to an account and then you can continue to allow them to do things so that you gather more information but in general these have no permissions

but when they use these keys it starts to get even more information it'll start to tell you the commands that they run if they hit any of the buckets it'll tell you the operation that happens and what's really cool on the left is it builds this incident map and this is on the free think site canarytokens.org so one thing to mention is you know you can spend like five grand a year which is the best bang for the buck and get a hosted console or you can do this yourself for free and you can generate tokens galore and when you do it tells you everywhere geographically that this key is used the idea here is

like you can automate deployment using something like jam for intune you can put it on every single machine and if marketing all of a sudden fires an alert you probably have a good indication well their machine's been compromised but what a lot of bad actors will do is they're not going to run it on that machine they're going to grab the keys and use it and in the process continue to de-anonymize themselves because it's grabbing the ip grabbing the particular user agent like did they use photo 3 you know python or did they use some other sdk so you're just continuing to grab more and more information and build this forensics trail you know why you're deceiving them and

leading them along the path yeah and speaking of bang for the bucket this is where you can start to really layer things in so with amazon web services you know you've got these simple storage service buckets and over the years these things have been prone left and right you know you've had people accidentally storing credit card information personal health info on these buckets and security was optional with aws you know for initially you know your bucket policies weren't strict by default it would a lot of times it'd just be world writable and anonymous and people didn't realize that they were putting sensitive information and so given the nature of this zero trust world we live in

you know companies will get breach it's just a matter of when and how badly we know this is a fact this is the only sensible way to operate in our modern world so we just have to make sure when our preventative controls fail and they will we have the ability to detect things immediately so all you need to do is create a bucket you can name it something that blends into your existing buckets and the mere act of just listing an account will fire an alert and you can tie this in with the aws keys and you can give them permissions into a bogus account with s3 buckets and you can have them spin their wheels and

think that they're doing something that's valuable but it's really not and so in the process you go through you get more information on the user agent you know we learned here looks like they're on linux and they're running boto sdk in java even tells you here on the right the actual operation just the head just to hit the bucket ips same deal time even the the user right and so you um you can kind of customize that as well and make this look super convincing and one thing with canary that's great is you have almost an unlimited number of endpoints to send alerts to so you can email sms slack generic web hooks you know

splunk you name it you can send data to it even just generic syslog and i recommend you do because you know you should be sending this to your sim and building the forensics trail print area by itself it gathers this data but their whole pitch is they don't want you spending time in the console they they really do just say take five minutes deploy it and then don't look at it and most security companies you have to constantly babysit the product but this is one in which it really just works um yeah and we've also got slack tokens and i like to go meta with these and do things like dropping them in your slack space

so you can take your face back to my space no but the the idea here is you could put these in public slack channels so that if people are compromising a slack account they're going to look around they might search for xoxp or xoxb which indicates either a personal or bot token and they'll find it and they think they got something you can drop these in your code repositories drop them on people's machines possibilities are endless and they have no permissions but they're really appealing you know you ask any pen tester including um haroon who you know founded thinks if um he would pass over a set of api credentials no anyone that comes across

credentials is going to try to use them slack tokens are no exception and exhibit um you know exhibit pulls it all together here for us [Music] and then on the api front we also the latest token which is super cool it's a kubernetes or k8 because it's k eight letters get it and then s and the whole point of kubernetes is managing massive amounts of what are called containerized applications which is like the next iteration of virtual machines you know containers like a self um self-contained environment where you don't it's platform agnostic you just spin it up with something like docker or whatnot but we can actually create a kubernetes config file and similar to aws we can

drop these on people's machines automatically and fire alerts when they try to use it and similar to aws we also grab information on the user agent so we'll know what version of the cube cuddle cli tool they use what endpoint they hit all the all the really useful pieces of information and it looks just like a regular kubernetes config file behind the scenes though it's hitting you know a bogus endpoint that canary kind of like masquerades it's brilliant and i've seen this fired you know probably half a dozen times all right we're going to talk about some tokens to protect your telephones and your devices these are personally useful for everyone here now and so i'd

recommend you know creating the following so there's a really neat token canary released last year it's called uh so wireguard is a new virtual private network protocol it's faster than openvpn it uses some different cipher suites and it's just more reliable and you can create like a bogus wireguard profile with the idea that if your phone is compromised and it's really hard to know you know last year you had the pegasus spyware that was hitting political dissidence phones and you know it was really hard to detect unless you did deep forensics i mean happened to jeff bezos in 2020 allegedly when the saudi arabian government hacked his phone so like how would you even know

that your phone got prone well one thing you can do is use this token to create you know a vpn profile name it something that sounds appealing like you know corporate vpn or production whatever and then if somebody turns it on because it's not going to be you because it's useless it's going to fire an alert one important thing to notice about these tokens make sure you use a memo that isn't test make it actionable and and relevant so i like to automate this typically by using like variables of hostname of the machine and then username so something token on whatever but otherwise you're going to be like where's this token and you know because

there are companies that literally have hundreds of thousands of tokens and the more the better you get unlimited with with things so you should be deploying this everywhere and anywhere but i love this and you totally can pull out your phone right now and scan this code because you know bad guys hate it when you do this one weird trick qr qr codes are really versatile and there's a lot of things that you could do with this you could put this around your office and say here's the new wi-fi password but you know full well that you don't actually give out the wi-fi password because you probably push it out securely so anyone that's trying to get

it is probably just up to no good but one use case that i've seen that's really effective is if you have multi-factor you can drop an unread email in everyone's mailbox and make it look like it came from octa or duo and say hey you got a new device scan this bad boy and roll it and then when they do it's going to send them to some funny site and have like some lyrics from top gun or i don't even remember because it's randomized and where it goes you can rick roll them you can do whatever you want and at that point they're probably going to like try a little lightly you know it's all

about psychological warfare and as you're going along and playing mind games with them they're going to be more inclined to want to leave and on that note it's trippy but let's talk about mind games this is where it gets really fun this is all about the information warfare where you get to manipulate data to convince people that they're losing it so this data basically database is just asking me dump because databases are treasure troves of valuable bits you know everyone wants the database it's usually got credit card numbers it's got social security numbers we can make this thing look super convincing call it nft payments table or call it you know accounting backup whatever you know you can create a my sequel

token and you can drop these things everywhere you know you can put them in amazon web services and use them within your existing databases or use your imagination right and then once i go to open this you know database just running command there's all kinds of gibberish it looks legit but it's completely useless but then in the process you know we get another alert here we figure out oh there's the host name that's helpful so if they have a specific mysql client inversion it'll tell us that and again that starts to limit our our profile of they're using this certain version of this and this so we can start to you know um like pile it together and

and start building this profile who this person who people may be what if conspiracy keanu made the realization that conspiracy theories are actually conspiracy realities so this is a fun little trick right you create a google doc or a word doc name it something that everyone wants to open if they're up to no good like organizational structure go to market strategy ipo whatever and then you hit them with a truth bomb so you tell them something that they may not know maybe that leads them to you know walk away and question um you know everything that they're doing you know mk ultra is a highly legal cia program that you know likely created the unabomber among other things but you

could also drop something in about operation northwoods or write a whole story about the jfk assassination is if you know what happened just confuse them and below them at this point they're probably going to go away and this is a good point to mention there's another really neat service called bittrap that's experimenting with this whole idea around giving people bug bounties but differing amounts depending on how far along they go in the process so when you kind of use canary you know you can have certain checkpoints and say all right they popped my web server if you leave now they'll give you this you know five grand but if you keep going you're going to get less and less money and

also we're going to come for you and it's a pretty neat service i'm i don't think they're officially partnered with canary but it works works well just like anything you know it integrates perfectly so there's a lot of really neat things you can do along that line and it doesn't just stop with google docs we can spread some [ __ ] i just love puns but i'm fascinated with db cooper because the fbi closed the case in 2016. we still don't know what happened to him and you know maybe the person that reads this is going to go reopen the case and do something useful other than try to you know break into your network and

you know be a scumbag but you know you can drop stuff in here whatever you want it to be i really am curious what happened i mean he jumped out of the airplane with 200 grand he probably lost it all but uh you know they're still going to give 100 million dollar rewards so if anybody knows there's a there's a funny hbo documentary about people that claim that nowhere db cooper is but they yeah they don't to this day but that said hey use your imagination whatever you can think of that might be interesting and might educate them this is what you should do cool so now we've worked up to this point to a seven layer dip or

of deception inception and this is where it starts to get really fun and this is where you tie it all together so the idea here for example you got an adversary who pops say your your actual web server maybe is running apache struts it's vulnerable it has to be on the web right now or something you know there's so many situations you can think of where you can upgrade this because of that software dependency so let's just say you're stuck with this vulnerable server what you can do is wrap it in tokens you can do things like the netstat process which is one of the first commands in adversarial run to get situational awareness we can

actually wrap that in a dns token so when they run netstat it'll fire an alert and and then it'll start to build everything into your knee seam i'm such a dork security incident event management system and then that's part one right so they run netstat and then oh nmap happens to be there that's helpful well once they run nmap they're going to hit a windows server conveniently that's a file share let's just say there's a document on there that has some credentials that give you access to a bucket seven layers you're gathering all this different disparate pieces of information you're sending it to liam neeson and he will find you and he will kill you or he will at least

turn the information over to the authorities and you know this is just one example there's so many things you could do again this is why i love this is not only does it lean on your kind of like analytical brain but it requires you to be creative and think of different scenarios as a bad actor of what you may do you know what are you interested in if you have a background in offensive security what's your approach and you kind of think of it from the other side of well how am i going to trip myself up and you're kind of playing a cat and mouse game with yourself and there's bonus points if you

have a team that you can constantly like run these exercises with because you're just going to keep getting better at the end of the day and with that i wanted to thank you all so much and open it up for questions and comments [Applause]

it's all hand yes yeah absolutely i'm gonna share it out on on sketch and um also yeah you can hit me up on twitter i'm not very big in the social media world i'm pretty much a ghost however i created an account so that i could interact it's just at justin t varner send me a message i love this stuff i love security and i love coming up with solutions you know i think my generic title i gave myself a security philosopher so i just think about problems and solutions and i think we all kind of do that in some way but yeah i'd love to talk more about all of this stuff because it's super fun

what else bueller [Music] okay got a quite honest thank you guys you're the best