Easy Mode Deception Technology Deployments at Scale

[Music] my name is Sasha I am a security engineer on the detection and response team at remitly which is based in Seattle uh I have done a few different roles within the Security Org uh I started off doing user security and trust and safety and then I spent a few years doing vendor risk management and governance risk and compliance and then I've spent the last uh 3ish years in detection and response and today I'm going to talk about easy mode deception Technologies at scale this is something that I've been working on only for about the past 6 months so you'll see how much progress we've been able to make really quickly uh so first off what what are

honey tokens and what is deception technology uh deception technology is a type of tech that helps incident response teams detect analyze and defend against attackers by enticing them to interact with fake it assets deployed in our network uh there's two different types of deception technologies that I've been working with uh one is honey tokens and the other is honey pots similar but just a little different a honey token could be a set of credentials a word dock or a domain whereas a Honeypot is a fake infrastructure resource like an ec2 instance or container that's created and positioned within our Network potentially alongside legitimate Network infrastructure uh the there's a lot of benefits to deception technology uh one

is that when there deployed correctly they provide really high fidelity alerts um if they're not deployed correctly then you're going to have a ton of false positives and so we'll talk about later why it's really important to place them correctly in your environment uh they provide good risk mitigation for areas of the environment that don't necessarily have other detections deployed on them so if you are an early stage detection and response team uh or you don't have the budget for a full EDR tool but you want to have some coverage on your endpoints it's much easier to deploy some uh honey tokens on your endpoints than deploying a full EDR tool across your environment and that can give you some coverage for

potential Bad actors uh on your endpoints they also help you identify suspicious activity early on in the attack chain again if they're placed correctly uh it'll potentially alert you that an attacker is active in your infrastructure before some of your other detections go off uh they misdirect attackers towards useless information and waste their time which is always good and giving you some extra time to discover that they're there and stop them before they can cause any further damage they provide information about attackers when the tokens are used or when the resources are accessed so uh I'll show you later on there's some tokens that if they're used they'll provide valuable information about the attacker's infrastructure like their location or

their user agent uh and they also can show you what areas of your environment that attackers are interested in so if you deploy 10 different types of Honey tokens in on a resource and you see that one type of token is consistently accessed then you know that's probably the area that you need to shore up defenses on they also provide enhanced incident response capabilities for when the tokens are triggered because of that data that you get from uh about the attacker's infrastructure when they're uh triggered so they're so awesome why doesn't everybody use them uh because there are a lot of challenges with deploying deception Technologies at scale uh the first question is where do we deploy

them we probably know uh approximately what our Network topology and corporate environment looks like but from a Time perspective we can't deploy tokens on every single uh it asset that exists in the environment like think about a um a knowledge repository like Confluence uh you don't want to put a Canary token on every single Confluence page or every single Google doc because that's just going to take a lot of time and probably not have a good return on investment the second question is how do we deploy them we need an automated method that tracks deployment errors for the existing environment and has handling for when we add new assets to the environment this has to scale to the entire deployment

stack how do we make them these tokens need to be attractive to attackers but they also need to contain links back to our honey token dashboard for monitoring when they're triggered if we just make a random doc uh that says you know honey token. do it's probably not going to be accessed it needs to look like something valuable and then how do we respond when they go off so say you've reached the point where you get a token on all of your it assets and you have this centralized dashboard for monitoring when they go off but your analysts need to know what to do when those go off so we need playbooks we need action plans

for different severities of tokens uh and we need uh some sort of automation ideally for responding to different severities of tokens so enter in a tool called thst Canary uh think is awesome because they have some free resources that are available for deploying tokens uh last slide I talked about the four main challenges of Honey token so that was where to deploy how to deploy how to create them and how to respond to them and thst helps us with each of these problems thst makes creating a token as easy as point and click which I'll show you uh they also provide some open source scripts for deploying tokens onto large numbers of assets within the

environment and pointers for useful methods of deployment uh they have a bunch of different types of tokens that you can deploy in Mac Windows uh Linux and uh the ones that I'm going to show you are honey tokens not honey pots which they also provide so uh a fake container or uh ec2 instance that's also an option but those are paid uh paid canaries from than um you'll see how easy it is to to create a new token and test that the token is working and also some Intel that's gained against an attacker when they click on the token there's some cool customizations and flexibility that's available uh such as where do we place them what we name them

what the contents of them are and where they redirect to so here is me creating a new token so I'm just showing the different types of tokens that are available on the free uh Canary tokens website which is Canary tokens. org and one that I've found pretty simple to use is the QR code token uh because all I have to do is put in an email and name it and then it gives me a QR code that I can download and I can ship off to all of my endpoints and save it as something like MFA backup and now there's a canary on our end points so I want to show you what happens

when if I can get to the next slide what happens when uh this token is triggered so using my phone I just uh scan that QR code and then I have this email that I set up and the email showed up pretty immediately uh with some interesting data about who clicked that QR code and now I can use that to uh add to my knowledge repository about the attackers that are going after my end users and uh potentially add some IPS to my block list there's a bunch of different things that I can do with that

data so challenges there are a lot of challenges with deployment um the first question that we have to ask is what will be the impact of widespread deployment of a new technology on our existing security infrastructure so uh some considerations here if you do have an EDR you might get alerts uh from running the scripts that you need to call to deploy the tokens uh there's a Windows token in that alerts for sensitive commands being run that modifies registry keys so that will probably trigger an alert in your EDR and you need to make sure to have uh an allow list for your Canary tool before you run that script otherwise you're going to have a ton of alerts uh you

have to think about false positives and user confusion that could be created by deploying the tokens in a place that's too visible for end users so think putting a dot called Secrets onto end users desktops people are going to see that one day and be like what's this and probably open it and then now not only do you have a ton of false positives but you also have kind of defeated the purpose of the tokens because now everybody in your company knows that they exist uh probably the biggest challenge is uh at least with assessing impact is are you going to override WR actual user data causing harm to productivity so think about an AWS API key token which

is going to be really attractive to attackers and you probably want it you want to put it in a place where it's going to be seen and targeted so maybe the AWS config file but you don't want to overwrite someone's actual AWS config or you're going to cause a lot of problems um another consideration is costs so are you going to buy things or are you just going to use the free version uh either way you're going to have to have some Engineers that are going to be deploying and managing the tool and then if you're creating the tokens yourself you have to think about infrastructure cost um there could also be some concerns from uh stakeholders so it and

infra teams um to deploy these tokens across your environment you need a lot of access and there's not much that you can do to mitigate this except for to use uh a secure development cycle so make sure that every deployment and uh change to deployment settings for the tokens on endpoints is codified and uh requires consensus and ideally from A system that uh only allows pushes to production from trusted environments um so our practical steps and and best practices for efficiently deploying a single token to many devices uh across the environment the first thing that you should definitely do is start with a proof of concept so one token on each different type of device

that you're testing ideally in a test or pre-prod environment so maybe you put one token onto your test VM and then uh after that you can move on to your security team is a really good uh group of users to start with because those folks might already know that this project is happening and like I said you don't want too many people in the organization to know that you're deploying this technology or it's not going to be as useful um so you deploy to your smaller set of your smaller group but you don't give them too many details and see if there's any performance issues uh or if the token shows up in places that it shouldn't be

uh monitor your false positive rates as you deploy to this larger group um for S3 tokens there could be some automated processes that are accessing S3 buckets and you want to make sure that you're not creating an alert every time there's a robot action accessing the bucket you only want to know about attacker accessing the bucket uh for endpoint tokens uh we changed the visibility settings and put tokens in hidden folders to lessen the potential discoverability by end users and then we also had to think about which tokens make the most sense for different environments um we don't want to attempt to deploy the windows sensitive command token onto Max because that'll cause airs um another challenge that we had is

how do we do the deployment so uh do we use EDR or MDM uh to automate uh and streamline the deployment process for multiple end points so I started with EDR because uh on I'm on the security team so I already had access to that tool and I already knew how to use it uh and I knew that I would be able to use the tool to run scripts on my endpoints to deploy my tokens but uh anyone who's used EDR probably can guess that it wasn't the best solution because it's not really intended for that purpose this is really something that you should use MDM for uh because with MDM you have better State Management um with the EDR

I had no State Management at all and I tried to kind of hack something together but it didn't work uh and then I tried to manage State myself with a local database and that was also too complicated versus MDM I just had to get access and learn how to use the tools from the IT team and uh then all of a sudden I not only have good State Management and I still have the way to deploy the scripts automatically but I also have uh an automated way to decommission tokens from old devices and add all my tokens to devices the second that they're spun up in the environment um so we're not quite at this place yet but I know what the ideal

automated endpoint solution would look like and it would start with your engineer makes a change in their coding tool and they submit that to the code repository that another engineer has to plus one to that goes to your infrastructure tool where you have some automated function that's listening for new updates to uh the code repo that contains all of your token scripts that also contains your State Management uh database so you can say what devices are supposed to have the token and which ones are not and which devices are new and which ones uh have been decommissioned there's also error handling uh which we'll talk about after but so you go once you receive um the

update from the code repo that would send a push to the MDM tool which then deploys the script to whichever devices don't have the token already on them uh as managed in the State Management tool and then in MDM you push to the device the device goes back to the MDM and says yes I've received the token uh as expected or no your push didn't work and depending on that response you go back to your automation tool which uh has a module for error reporting and then from the error reporting you can send out alerts to your detection response team that something isn't going as intended once this entire State this is the automated uh deployment state but

even before you get to this point um if you're deploying tokens to endpoints you need to have a way to respond when those go off like I said earlier you don't want your analyst to be receiving alerts and have have no idea what to do with them so uh response you have a few steps in the response process that definitely need to happen so if you have a SIM uh you should send all of your alerts that come from think or wherever else you manage your tokens to the Sim there's some magic and context that gets added to all of the events so uh in an ideal world that context could tell you if it's a false positive or A true

positive and if it's a likely false positive it'll be lower severity than a true positive from your sim you go to your ticketing tool and you also have your playbooks uh which tell the responders how they should respond to the different types of token alerts and then depending on what context is added you might want to reach out to the end user to see if they accident L triggered the token themselves or if you suspect from that context that it's likely a true positive you can send uh command through MDM to lock the endpoint or uh decommission the infrastructure uh declare an incident some sort of automated response is always ideal so what's next um Canary tokens is

a really easy way to deploy deception technology but you will have scaling issues uh and so that's why I do recommend them as a way to start because they help solve some of those uh first round problems that you're going to have with deploying these tools and uh once you've deployed to a single host make sure that you deploy to a small group of hosts before you try to deploy to your entire environment don't deploy to the whole environment without monitoring for your false positive rate and make sure that before you turn on the tokens in production you have your response steps laid out whether that's manual playbooks or automated response actions thank you we

[Applause]

have we have a couple minutes any question [Music] questions hey there thanks for doing this talk um what do you one of the things that I worry about for something like this is you really want to hide all of your tooling and all of infra for an attacker so what what do you do so that they can't just easily find everything about where your Canary tokens are you mean hide it from end users but not like make it visible to the attacker but not to users sorry let me clarify so um let's imagine an attacker is inside your company they they've breached in some way like one of the things they might want to look for is oh what are they

doing by by the way of uh detection engineering like how yeah so what do you do to to kind of quell the fear that they're going to find your it repo with all of your in configuration code and everything yeah that is a really good question and definitely something that we've had to consider uh I mean we try to use code names wherever possible and not name anything Canary or Honeypot uh but at the end of the day if they're in the environment we just want to make sure that they find the token first before they find our DNR infra so it's like if they think that this looks juicier to Target then we might have a step we might have

some time before they get to a more important part of the network to kick them [Music] out hey so thank you for the talk um I think it's more of an art than a science I think that there's a lot it's it's a game that that's being played between both the adversary and the the people inside one of have you considered things like synthetic traffic to cause tokens to be in caches to be like to make it look like they're actually being used cuz if see people see files that are not being touched surrounded by files that are mhm right things like that um like are there more like adversarial type like not adversarial aggressive ways of kind of

making it juicy um to to draw people's attention yeah that's a really good idea and honestly not the step that I'm at yet um I would say we're still at the place of how do we just get these end points or these tokens on all the end points and in all of the places that we want and uh make them look at least attractive and then the next step is to make them even better which I think that would be I got another question for you um like if you want to do have a canic token for AWS awesome uh if you're actually you're looking at a situation and you've got like oh I've got 10 third

party API tokens and you know Canary to tokens doesn't support them what do you do then yeah I mean then you get to the place of trying to create your own tokens so I mean there are certain I think that Canary helps you learn about how tokens can be deployed and you can learn okay the way that uh the S3 token works for example is uh it uses the AWS uh S3 logs and then you create a second bucket and then you get an alert whenever there's a new event to that bucket because there's been an access event to the bucket that you're monitoring so once you know that once you know how it works under the hood

which you can see on their uh open- Source GitHub then you can try to deploy some of these things yourself I think that's just where you get uh more of the infrastructure cost that you won't have if you use things hey Sasha thanks so much for your talk today this was awesome um I'm curious what metrics or data you're collecting to show the effectiveness of the canary tokens as a detection mechanism versus your traditional you know threat detection patterns yeah totally um so we don't have a ton of metrics yet besides false positive rate and also the success rate of deployment but uh once they are fully deployed uh Beyond false positive rate I think it

would be really interesting to see if we do have metrics on uh security incidents how many times were the tokens involved in that incident versus were they just completely ignored and if we know that an point was compromised and the token was ignored then we're probably doing something wrong and we need to revisit it cool thank you thank you [Applause] [Music] everybody