Honeypot Boo Boo: Better Breach Detection With Deception Inception

thank you dr b my name is uh justin varner appreciate you all being here i'm excited to talk to you all about deception technology 2022 really is the year of deception technology people and companies are trying to figure out a new way to do breach detection and alerting and monitoring because what we're doing isn't working every single year gets more expensive more breaches we need a new way of doing things and so i'm going to talk to you about some ideas for how we might do that well first let's talk about honey traps honey pots honey tokens what they are they are security mechanisms designed to force adversaries to announce their presence they can come in

the form of tokens and honeypots and i'll talk more about those in depth in the subsequent slides this is a component of a larger discipline known as deception technology the goal here is to be a trickster the more of a trickster you can be the more success you'll have thwarting your adversaries and in addition to doing that we need to reduce our mean time the detection and mitigate the blast radius of these catastrophic breaches that have been going on for the past 20 or so years also there's been a huge toll on the industry you know we depending on what report you read anywhere from three to four million unfilled security jobs from people that are burned out and

you know you can get a sense of it by just talking to laundry room viking and um this is the plight of a lot of people who are security analysts you know they thought it would be a good idea to just work and triage tickets all night except after a while you're going to be dead or you're going to be burned out and that's not what we need for retention and to build the next generation of security personnel so we need we need a new approach so why why deception technology why is this important well millions and millions of alerts have been ignored you know and breaches have happened if you look the past ten years even going back to

snowden with the nsa you had the most sophisticated spy agency they had no clue he was rummaging around and exfiltrating data you know until it was too late solar winds people now probably have you know post traumatic solar winds disorder or supply supply chain concerns everything i do now i'm worried about is there a third party component that's vulnerable have i done my due diligence you go back to last year with colonial pipelines you know they had physical effects gas was unavailable on the east coast of the united states for 10 days and it's going to get worse with the convergence of physical and virtual and [Music] if any of you remember from 2012 jimmy mcmillan

he ran a campaign in 2012 about the price of rent being too damn high he was right and if he saw the price of cyber crime now he would be even more disturbed so it is estimated to have cost six trillion dollars to the global economy uh last year about every 39 seconds a business is breached 2025 they're expecting this to cost 10.5 trillion which is you know half the united states gdp and the fbi said there were about 847 000 complaints filed last year which is absurd but until now there was zero chance of any solution let me tell you this is another alarming statistic this was from ibm's data breach report from 2021

212 days before even detecting a breach let alone remediating remediation was 279 days and if you look at some of the biggest breaches of all time as far as records you had cam4 it's interesting because there's 10.88 billion accounts but there's 7 billion humans this is a naughty streaming site so clearly people didn't want others to know that they use the platform yahoo many facets of their platform got breached yahoo sports fantasy stocks and then aadh aar that was india's attempt to do digital identity for everyone the problem is they tied all of this in their 401k to your retirement to your driver's license and i think the government said you know this can't be hacked i mean that's just

an invitation right so within three hours yeah everyone in india was affected by this it's just entirely too long and the effects you know are catastrophic so let's talk about canaries canary's in the code mine so i'm a i'm a partner of things canary i don't work there but i represent them because i've used their technology i've seen it prevent four or so catastrophic breaches i've seen pen testers fail in 13 minutes because of their stuff so i really believe in it and the whole idea goes back to the canary in the coal mine where you'd have a bird that was down in the mines and it would be alerting you of danger before the

miners got taken out by by any of the conditions in the mine so following along in that concept they have two kind of representations of their technology you've got birds which are traditional honey pots or they're not production honeybots i'll talk about that and then canary tokens now birds are designed to look just like real devices they can be everything from a dumb terminal to a smart fridge to a scada system and it looks real and you'll see in the following slide when you interrogate that system you'd be hard-pressed to tell the difference and then you've got these tokens tokens are free you can go to canarytokens.org right now and create as many as you want

these can be google docs dns aws i'm going to go through examples of all these and how you can use them give you some ideas for how to layer deception and create inception as i like to say and the whole point of this is we want an early warning breach system we want to know at the time that there's a breach not 212 days in the future when it's too late you know and that's why this is important bird is the word uh yeah and so things can air their company out of cape town south africa started by uh harun mir they've grown organically over the past seven years and i'm really excited to see them do well they're just the best

they also have the best support and if you ask them for swag you won't be disappointed it's the best swag ever all right let's talk i like memes and puns clearly as you can see so let's talk about the actual bird and the honey pots the idea here is you know windows active directory is used in pretty much any company that's over 10 years old this was before octa and identity as a service and if you're a legacy or even modern company you probably use active directory and azure active directory so one thing that you can do is you can create a device that looks just like a legitimate domain controller and a domain controller its

whole purpose is to manage active directory which is a way of centrally managing users machines passwords files you name it and anyone that loves pen testing or red teaming knows that if you attack active directory in some cases you can take over the company you can read email you can tap the network log into any computer it really is the crown jewels so the idea is we're going to make a domain controller that looks acts and chirps like the real deal this is what it looks like for those that are familiar with nmap it's a classic port and service enumeration tool and what you're looking at here is there's a port scan against known windows domain controller ports you've

got 21 which is ftp ssh you got iis so on and so forth when you do the interrogation to services you can see all of this looks legitimate if you were to go to this ip on port 80 you'd actually hit an iis web page and it would ask you for credentials and it would even capture those and the adversary is thinking they have something legitimate in reality it's totally bogus but we're building a forensic trail this this whole time and building up to that and even if you start looking down at port 389 and 88 you can actually create a legitimate domain like 80.thinks.com or 80. b sides so on and even um sql server

it'll show you like the version the build all of those things it's it's very convincing and you'll be hard-pressed to tell the difference between a real domain controller oh the formatting's a little messed up but uh yeah you don't need to uh light up your own christmas tree this year because this is what happens when you do fire your port skin not only do you get alerts that people are interrogating all the ports but if somebody tries to log into microsoft sql with say sasa which is common it's going to capture that credential it's going to capture any attempt to access these services if you have a file share like smb over 445 if they hit that it's also going to say

they access this file share you can even put documents within the file share and layer it and layer it to create more and more deception and for alerts you have multiple avenues for endpoints you can use slack pagerduty email sms this is the console they give you and even in this process people get tripped up because once they visit a site with their browser you're going to get all these browser header you're going to get information about the browser from fingerprinting it so you're going to get the ip the machine type you're slowly but surely gathering information that you can use forensically and even if you're not going to be able to de-anonymize this person you'll have enough data to

give you a third party and give them a head start yeah ted stevens you know back in the day alaska senator he said the internet's a series of tubes it's not a big truck true not a big truck it's mostly noise and in fact mostly gray noise so the reason i bring this up is because the idea of a honey honeypot now in the thinks world is built around production honeypots where they sit inside your network and they're designed to give you real-time alerting when you have issues and organizationally the outside bird is designed to be a traditional research honeypot that goes back 20 years or so and all it does is passively listen to ips and what you can do with that

is you can filter it through grey noise which is a meta thread intel servers and they and it says give me the ips we know what our bots what our cl content delivery networks um all these things we're going to leave just the stuff that you care about so you get 50 000 ip's hitting your left you might get a couple hundred to go investigate gonna save tons of time going to help those analysts with their burnout problem super cool company free service just kind of gives you a glimpse into you know just a bunch of ips i had that i was collecting on an outside bird most of these are malicious and from china

no big surprise there there's another great integration in ways that you can use your canaries to do more so rumble is a company started by h.d moore wrote metasploit awesome dude the whole idea with rumble is it's comprehensive asset inventory and what you can do with canary there's a native integration you can hover over an alert launch rumble and it'll create the asset automatically and give you more context you know like this is my home ip in richmond tells me my autonomous service number the location my isp um all of those things so rather than just have alerts come in you can say this is one of my assets maybe it shouldn't be communicating this

way if you if you have an asset that's firing off alerts for example like a log for j token that's probably an issue and you want to um you want to like know about that so also recommend yeah and it goes together like george foreman and grill tilapia so or countertop grills okay now that we've talked about honey pots and birds we're going to talk about tokens and tokens and this is where it gets really fun and the first type of token i'm going to talk about is the recon token and let me give you a couple examples of what you can do here so most security people know dns stands for deceived nosey strangers but

in reality it's a system that allows you to type in google.com yahoo and not worry about the ip most people don't want to memorize it unless you're weirdo like me but this is why it exists but we can use dns tokens in a pretty elegant way what we can do is we can deploy them on say dark network segments like entire vlans that shouldn't be used at all and we have dns entries mapped to them so if somebody does a reverse lookup of say you know vault dot whatever domain it'll reverse it'll hit our dns token as a cname and it'll you know hit the ip one thing to note is this is pretty obvious here where it

says canarytokens.com when you go and get the service you can use your own custom domain and make it look legitimate so it'll be you know um docs dot p sides stock code you know so on and so forth but they're free site they kind of this is what you get you know and even just from that even as a dns token it didn't get my ip it got the dns server's ip and i was using cloudflare but it even could still determine that i was in richmond so you're starting to build that forensic trail because once you know someone's in a location they're using a certain machine type and a browser like if any of you have used the service

steam from valve they're constantly sending hardware data and they're gathering information so you can see forensically where this can go you know it's just a matter of time before you figure out who this person is and they're gonna make a mistake okay so this is probably one of my favorite tokens for all the nozi o'donnells out there that are just poking around and let's say they do some sub domain enumeration and they're like this sounds cool vpn dot domain dot whatever you know or in this case i bought a uh fishing domain that i transferred to thanks thinks that i owe which is pretty pretty high value and i just wanted to see who was hitting it

as people from all over the place you know all over the united states um australia it creates like a map for you and it tells you like okay is this part of a larger campaign or is this target you know is this limited because this might help you with your threat intelligence service to say there is a sustained campaign against my organization and i want to know about it or it's just some board schmuck you know it's just hitting the site but this token gives you a wealth of information not only do you get geographic info from the ip you get the user agent so i already know this is chrome on a mac so again you're

shrinking down your pool of potential people how many people use a mac in dundee and chrome you can figure this out you know you even can figure out javascript there's a lot of attacks against javascript that will inadvertently spit out an ip you know the fbi did this with the tour browser in 2013 when they dion honest de-anonymized freedom hosting it was an application layer attack and javascript went nuts on windows browser and accidentally spit out the last ip of the exit node so these things can be useful and even knows that i'm on a mac intel this is my m1 so yeah how many m1s are there in dundee not that many because supply

chain dodge [Music] so this this token here is uh great this is a clone website token and why i love this is you can use this token to figure out anytime your legitimate domain has been cloned and redeployed a lot of times it's nothing some people will test on local hosts or you know they'll have some staging site but i've seen phishing domains purchased in this token deployed and we were able to contact the registrar and get this taken down by cease and desist and if nothing else it can start to tell you coupled with the web redirect token like yeah is there some legitimate campaign happening externally that you should know about and yeah this this is super easy all you

have to do is add a couple lines of javascript to your html page you can obfuscate it so it's not evident to people that are just investigating your source super helpful okay now apis application programming interfaces now but these really these tokens are designed to make you better alerting protection and investigation just ask bert macklin from parts and recreation he's on it okay so let's talk about aws amazon web services many people use it it's very attractive for adversaries api keys because you get a api keys with the right permissions you can do anything spin up permissions you know create databases uh rack up a huge bill unless accounting is on top of that you're not going to know no one's going

to know so what we can do is you can create a script they're out there can air utils i wrote a whoops i wrote a basic one for uh jamf and um in tune and what you can do is create this token it generates a unique one in every single machine with their memo and the idea here is like if susie q in marketing gets this key that fires she doesn't use aws right and so you're going to know something's up and what happens a lot of times people make the mistake the adversaries they'll take the token off the person's machine and run it on their own machine again further de-anonymizes them they don't even tell you the exact command

they're running in this case list buckets so you know all it takes is one mistake to know but after when you get good at this you're gonna get people making lots of mistakes and you're gonna have a compelling argument as to who this person may be there's another really great token that's good bang for the bucket so there's simple storage service buckets in aws and what you can do with this is you can use your aws api key give it permissions over a bogus bucket the beauty in this is any command that runs s3 like listing buckets in an account it's going to hit this token it's going to fire an alert so you can name it

something that's convincing or blends into your current bucket names you know aetna health just listing it is is going to tell you well someone's snooping around but you can add additional permissions if you have false positives or if you use like in terraform or something to deploy infrastructure you can say i only want to give certain people permissions to it but if anyone's accessing this directly this is probably an issue and this token's neat because it'll even show you the exact type of http request the user agent again this was using the aws photo 3 python sdk again on linux like if i continue to keep using tools on this mac m1 pretty soon it's going to you know it's going to be

known who i am you know it's like well there's this clown he's just you know trying to do pen testing and hitting everything every token he can come across uh and then similarly you get you get your slack alert you get your console alert any number of endpoints you can even send to a sim which i'll talk about the last stage super useful and then there's a lot of memes wrapped into this one let me explain so exhibit back in the day on pimp my ride had had a great way of layering together ideas like if you had if you like fish tanks you could put a fish tank in your fish tank in your car and you could

drive it to an aquarium but the idea is with slack tokens which are also very appealing like ap uh aws tokens you can drop slack tokens in slack and if somebody compromises a slack account they're gonna go find that token they're gonna use it and they're gonna probably go back to my space after that but no um but it looks so legitimate and you can also assign permissions bogus permissions over a fake slack space by default these have no permissions um but this is great you can also drop this in your code repositories you can put these on hosts ec2 metadata because people accidentally do that all the time put them in places where they're going to be

discovered and they'll be hard to use you know and if you have a problem with secrets management now like a lot of companies what i tell people is yes work on cleaning up the secrets but in the interim litter your known secrets with a bunch of these bogus secrets because you're going to get people if they're mucking around the trip over this so you at least have visibility but this is no substitution for good preventative controls you know like secrets management also the latest token they came out with which is great for people that use kubernetes or k-8s it's k eight letters s get it pretty cool and for the non-technical kubernetes is a way to run

a containerized application which is self contained and you can run it agnostic of the operating system you know in its entirety kubernetes is just a wonderful like just mess of everything and it'll allow you to run thousands of containers and it'll orchestrate and do dns for you and all these things so similarly this is an attractive api token you can drop this on every single machine in your you know config and if somebody takes it tries to use cube cuddle which is the command line tool to interact it's going to tell you that it's going to tell you the api endpoint and you continue in the ip you're just continuing to build this forensic trail

and who can't who can resist a kubernetes token or aws i mean literally like pen test company came in 13 minutes they failed the engagement then 17 minutes and the whole engagement went two weeks what was funny is at the end they compiled this report that was like 70 pages long and i gave them a report that was 90 pages long and said did you know that this was happening and their face was great it was it was priceless all right now there's some tokens here that are applicable to mobile devices and i recommend you all create them because they're valuable let me tell you more about it so wireguard is uh the latest and greatest type of

vpn it's way faster than openvpn it does a much better job with um active connections and not not dropping connections and it's easy to set up and what you can do is create this token you install the app and the idea is if somebody compromises your phone which has been happening from pegasus spyware to whatever has been floating around now you know you want to know that somebody's on your device how else would you know unless you did forensics with imazing or something like it's not easy to know that you've been pwned but if you do this token and you name it something convincing like you know corporate vpn if somebody triggers it it'll fire an alert i've seen this save

like probably half a dozen people that got this pegasus spyware this is the only way that they knew and then they blew up their phone and they got a rotary phone well that's what they should have done after that but is a great token it's free highly recommend everyone do it similarly you can also scan this this qr code it's not malicious but the idea it's just funny and this works in so many great ways what you can do is you could print out qr codes put them around your office and say here's the guest wi-fi password but you probably don't have guest wi-fi because if you're doing things right you're pushing out the password with jam for something so

the only person that's going to try to connect is you know someone that's just up to no good rummaging around that's one technique but one that i've seen very effective is you create this qr code as new device enrollment you know for octa or mfa hard to resist you put in their inbox you can even spoof it from octa your security team they'll scan it and it's going to rick roam or whatever you want to do you can send them any number of places this particular one just sends you to canary tokens with a randomly generated like movie quote or meme or whatever but bad guys hate it when you use this one weird trick that's for sure

okay now we're going to talk about psychological warfare and mind games yes that is an optical illusion if you stare at it long enough you're going to lose your mind no and this is really fun the idea here once you get to this phase your goal is to mess with your adversary get them to question their sanity and ultimately leave before you know before you do something to apprehend them so one of the types of information warfare you can do is you can create database tokens and this useless pile of nothing looks pretty convincing you know nft payments who's not interested in some nfts right looks like a database but if they interact with it or use it

they're just like nothing happens but again we've gotten valuable data we know where they are we'll know the commands they run if we get that far you can get super creative like this you could put um you could put a token inside of a token so you could put a dns token in here and lead them down a path and so you say hey this database makes a connection to let's say vault dot um things that i o they may might hit that thing and they're going to fire an alert on a dns token you can get super creative here there's no limit to what you can do your imagination is the only limit yeah what if i told you conspiracy

theories are actually conspiracy realities some of them are sometimes but the idea here is you can really start to engage on a level that is downright confusing so you might create a google doc or a word docker pdf and it has a appealing name like executive compensation plan or go to market strategy and someone's going to open it because they're going to be snooping around you can use that as an opportunity to drop a truth bomb on him you could tell him about roswell or aliens or um harp or you know globalists you know you can go on and on this case mk ultra you know which sounds like a conspiracy theory because it was but it

actually happened and at this point they might just reevaluate their life and maybe they'll pursue justice or finding the truth who knows but there's a chance that they'll leave and an interesting point about this there's this company called bittrap that's doing something really interesting where they will pay adversaries at different points bug bounty depending on when they leave so if you pop a server and we know about it from canary they'll say okay if you leave now five grand if you get further along four grand and the offer dwindles until they get so far along that um you're like you're screwed you know and so don't keep going further it's a cool concept thinks i think is partnered

with them it's not official but i like it i think it can just discourage people especially with information warfare and similarly you can spread some [ __ ] bullsheets and uh similar to the way that you would engage with google docs you can put all kinds of juicy financial bits in here i don't know if there's any db cooper fans out there he was the only person who successfully hijacked an aircraft and got away with it and to this day we don't know who he is there's theories around it but you know he he jumped off with 200k and he probably died in the process but if you can prove otherwise you get a hundred million

dollars it's a cold case though close it in 2016. but what if this adversary opens this sheet they get really interested in db cooper and that's their life mission you know that's the worst case worst case scenario but you can get super creative with this okay and then this builds to what i call better breach deception with inception deception inception that's a mouthful and the idea here this is one of many examples this is a seven layer dip or seven layer example of deception and it kind of works like this you've got a bad guy let's say that they pop your nginx or apache strut server but that server also happens to be a honeyderm you know it has some kind

of tokens on it well you can actually token processes you can token the netstat process the nmap process so they'll get on this host they'll fire they'll do situational awareness netstat nmap tcb dump netcat you name it all the while you're sending the lim nissim get it sim but yeah i thought that was pretty funny but at each point in the process you're getting more and more information so let's say they do nmap they find a windows server it's got 445 open cool it's a file share there happens to be a file share with a google doc that has credentials boom alert those credentials are aws they use aws for a fake bucket if and

all the while you letting down this path you force them to make multiple mistakes but by the time it's all said and done you're going to have a good likelihood of who this person is or you're going to call some some company to help you along the way um at that point you send liam neeson or you use your sim because as these alerts are happening you can send the full log to whatever you're using splunk what's up you know panther and it'll build this build this trail for you and you'll be like okay i think i know who this may be and yeah that's that's one of many things you can do literally possibilities are

endless and with that i wanted to thank you and open it up for questions

yeah it's a good question so it can either be like a bird which is like a honey pot so they talk about canary birds which are like here's this device it can be like a cloud device it can be a virtual machine it can be a physical device and you can figure it this bird the run sql server or jenkins or you name it that's a canary bird if somebody interacts with it it's going to fire an alert but then you've got the whole tokens thing that can be literally anything any piece of data so they have predefined tokens on their site they've got a google doc token they've got wireguard aws you create this token and

once they trip over it it's usually just like a url or um or some kind of pointer it's gonna fire an alert so like the dns token is a simple example it's a c name like dns record so if somebody queries the cname it resolves to this token once the token happens you know fires the alert their aws key is a little more sophisticated they actually use cloudtrail to monitor api calls in real time and you know within 10 minutes if somebody uses the key it'll fire fire the alert so it's really any any piece of data you can even do something really cool you can monitor the movement of data like sometimes your legal team

will send a document to someone and you hope that they don't forward it to someone else but you can token that document and see exactly where it goes so people are just sharing it you know in other unauthorized manner you can know about it so

it's a good question so this is not intended to replace like preventative controls it's meant to say okay someone's in your network assume breach right it's not a matter of if it's a matter of when and how badly once you've been breached you want to know as soon as possible and not 212 days when it's way too late you know people are gone um at that point but that's yeah still do your firewall or you know vpn's endpoint protection all those solid things for sure thank you [Music] oh man there's so many so many different techniques um try anything i think just so when i got ready for this uh pen test back in the day i just went all

out and i just put mine fields of tokens everywhere so i built a lot of automation with like jenkins and like i put at the time there were these ec2 metadata tokens that i deployed on all of our systems with chef and there were like hundreds and hundreds of systems and so you know they get on a machine like cool you know they do that they hit the metadata api fires an alert and then it happened to be tokens on there so like it's our lucky day it was just an epic fail on their side um but i'm kind of talking through working through an idea of a active directory token right now with thinx because

there's a lot of value in that the issue is they don't want to run agents they are very lightweight they make the pitch that you can be up and running in five minutes and they're absolutely true you know and so to make it work you almost have to tie in into azure and so they're exploring maybe using you know azure sentinel and doing similar to the aws api key yeah so many so many scenarios yeah hard to pick one oh yeah

[Music]

yeah so as long as it's part of whatever mechanism you use to deploy um say your pods or your containers like if it's part of your your flow like you know i've done this with git ops where you know a new container gets spun up and as part of the kind of user data or like the um the ammo you give it you can have it like fetch a token and it can deploy it on every machine and so and the neat thing is it'll because you can have unlimited you don't need to worry about expiring tokens like you can run some api calls to clean them up but it'll generate a fresh pair it'll tell you the time stamp so

you'll know that it's modern and you can get better with your like variables too because like sometimes there's not a lot of actionable data right if it just says uh this is pod dash something right you want to know like what's the host name and so you can pull in those variables and use that to construct the token which is really really helpful so thanks for the questions

great question and yes so i set up grey noise doesn't have a native integration with canary and rumble doesn't allow you to create assets however what i did was i built through soar i use this product called tines they're in dublin they're great and what happens is it fires a canary token it'll look at gray noise and say is this a known ip is it not if it's not known it goes to the flow chart goes to rumble creates the asset gives you context creates a jira ticket super easy story to do tons of like examples like that but yeah i'm happy to talk to you more after this about some specific ones where time

oh i'm on time okay i heard one more question so

i'm sorry they guarantee that well so the first rule of deception technologies you don't tell people use deception technology however yeah they've done a really good job of masquerading the identity of these things and they make the point that if somebody figures out that there's a canary at that point they're going to be more cautious no matter what they're like okay you've got honey pots in your environment where else are these minefields like you make it so difficult to not step on these things that they're going to trip up you know or they might just turn around because they're like i don't know what's real and what's not i just saw a document that said mk ultra

happening or whatever like you can get you can trip them out for sure but yeah awesome anything yeah thanks dr p appreciate it thank you