Canary Tokens - why and how to implement them and related gotchas

[Music] and uh I was just in the workshop talking about how to make a good good presentation and they were like oh yeah your abstract will give you all the context your your audience needs and I didn't do that Canary tokens this is rushing ahead to go do that thing but why do you want them um yeah Canary tokens are really going to help you know when you're that data breach is coming because they are in your systems so that's why you want to be here listening to this talk uh this is primarily for folks who are defending or defending curious and there some little tidbits for folks who are attacking or attacking curious so I hope you enjoy we're going

to do some what why when how just so you know if you want to leave uh or stay uh and then the build versus by responsible moment of do by real want to build this or is that kind of a lie and I want to buy it I will build a couple of examples with you uh and talk about some more and actually there's a surprising amount of Lessons Learned content there's about maybe half because honestly this talk is for me like two three years ago when I started enthusiastically sprinking Canary tokens around and my God those can take a long time to come to roost in a certain respect and and I didn't know

a lot of things I needed to know so this is um the reason for this talk this is the talk I wanted to have and didn't have um although I didn't know I needed it and then we'll wrap up and maybe there'll be time for questions so Canary what not everyone's a native English speaker Canary and the co mine is uh the origin of this idea uh something dies to tell you there is a problem tricky tricky metaphor but um more or less uh that's what's going on uh something is going to tell you hey you might not have noticed there's a problem go do something about it this domain is kind of there's a lot of other stuff that

might seem like it's related to Canary tokens uh and it sort of is sort of isn't you can think of a canary token as like a digital trip wire like there's an API key it's not a real API key and and a and a tack is in your system finds it lying around and they use it and you get an alert um it's so it it's a subset of intrusion detection it's not everything of intrusion detection um sorry it's related and it's a subset of deception technology uh if that's a buzz word you're into but it isn't it isn't it's not honeypots it's not a decoy it's not some high interaction thing you want attackers to spend time and get lost in

um while you catch up with them uh it's not uh detection in your seam right that's that's that's not what that's about um it's not secret scanning it's not your EDR um cool so what is it then like tell me well so you got attackers they look like this uh they're in your system and the effect you want is they will trip over this Digital Trip wire and your helpful Canary uh will tell you like oh my God pay attention and um if you're in the earlier talk from uh Brian this is the your fail inspiration upcoming okay because guess what everybody has a canary already it's just a bad Canary you don't want this Canary it's a

motivating Canary okay so you you saw this image I think yesterday um great I blurred it all out right but solo wind at some point they got compromised and then Along Came the evil Canary uh someone out side of the org said you have a problem right that's your evil Canary it exists someone down the line is going to say oh by the way way back then you got breached and if that's what you're cool with cool stick with it but each of these actions that the threat actor took along the way were moments to set up a trip wire something to say oh by the way they got this far through your system now you can start acting or this far

this far this far right you're not guaranteed that they'll trip over one but each one is improving your chances and I mean I just picked solar winds as an example because there's a really good timeline graph that shows it right um and You' be natural to be skeptical at this point and say oh we are solar winds but here's industry data like some uh the what's the dwell time according to mandant uh sometimes it's like a week or less but so often it's a long time and the little asteris here this is skewed by what mandant actually called in to investigate and then also byy ransomware that's not necessarily what we're talking about here I think this this

quote it kind of blows my mind um they were not talking about Canary tokens but they're basically saying someone from Google talking about rust I think in Android they were like why are we always like making things harder for the attacker by spending lots of Defender time like that that's that economy is busted we for guaranteed spending defend a time to maybe cost attack a time what and I think the why this linked up for me is because I say uh Canary tokens are the cheapest highest signature noise detection you can do and possibly more important you pay a cost to set them up and and then you don't keep paying every day right you only you only pay the cost

once that Canary token gets tripped that's when you like have to then respond and do something um so they're very very very economic uh as Defenders um and here's a quote from a black hat speaker who said it very very cons concisely if you don't use these you are criminally negligent in your defensive posture so go do that but that's all very well to say everyone should do everything right uh here's an important thing you should go do that well should you uh I think this breaks down to like cost benefit like step one do you even have the bandwidth you know the people power to go do a thing so if you have dedicated security staff I would argue

you do um if something happened to decrease the cost like let's say hipper happened at your company and now you ingest audit logs you've done a part of that work already or you went from uh your customers being uh small businesses to lots and lots of Enterprise customers you're probably more of a Target you're in their supply chain um so those are kind of motivating like when in the life cycle of my company is it time to argue for doing this and as a meta comment I would say uh if you're feeling if you're looking at this and going you're telling me I could do this thing but I am drowning in work right I argue that your responsible

position is to not always be responding to stuff that's coming in and instead making prioritization decisions where you can say cool there's all this reactive work there's some proactive work we should be doing too and here's the cost benefit uh and if you can pull that off I mean you'll generally be a lot more successful in your career I say um cool so that's the why the what the when a little bit of the how uh so this is something I'm going to be referring to this diagram is going to come up again and again the component parts of uh a canary token look there's the canary token itself there's the thing it interacts with that the so the attacker

takes the canary token to use it with a cloud service that thing generates an audit log event that goes to an log aggregation system that can then tell you like wake you up page you say go respond you have a problem um yeah and so this is that this is that fix moment oh you can't read that okay it says fix um so cool there are solutions that exist for this why would I ever build one why not just buy one uh and uh like me you probably should this is the always probably the responsible answers like buy that thing right um and especially if it's free uh which which it is uh there are very good

free Solutions um but yeah it's it's kind of it's kind of some weeds like uh you can get all of this you can get all of it for free from Canary to .org and in fact you just go there and create Canary tokens put them in your environment um and they work they're robust you'll get an accurate uh notification brilliant like other people have put the work in so why would I say don't do that um it's not that they don't do this job very well they really do really really do um and in fact you can you can you can pay them um moderate amounts of dollars to get a very nicely curated solution and it is not a vender pitch I

promise kind of I promise uh but it is just brilliant but but where does that leave you you do then don't have any awareness of any of this stuff cuz um ask me how I know or don't ask me how I know but what do you do when on when one of these Canary tokens goes off great do we have an intruder I I don't know um if you're not familiar with all of these things um actually I'll take a step back if maybe you're not even covering the things that really matter to you right is your internal tool do you have a canary token for that I don't know is node package manager really important to you or GitHub do you

get those from from as a paid service probably not right so you're blind here um and then can you go do incident response can you go do anything once this happens you if you didn't do all of this yourself you probably don't have the logs you want to respond you maybe not retaining them for the amount of time you want and maybe you can't even query them properly right so the process of building them means you get familiar with what you have and what you don't so you might think oh well you know if the compromise Vector is through slack we'll go to the audit logs when it happens and then you'll find out well damn we're not

on the the paid plan that actually gives us AIT logs that's too late right that's the kind of detail you want to discover ahead of time one sec so next up we'll walk through building one I'm going to assert this is the learn moment um so I'm going to refer again to this same breakdown and for folks that just entered the room uh you've got a canary token that your adversary uses against a cloud service in this case AWS AWS uh as it sees your Canary token being used generates an audit log that you ingest and then you have alerting that then will you know wake you up uh get you to take action how how do you go about

doing that uh well I didn't go into a lot of detail here but basically you have all the pieces you need to go create a user uh attach a deny policy to that user and then create an access key and then what you can do is you can leave lying around a AWS credentials file that when the attacker does something with it they find this file innocent looking file uh maybe they'll try and say hey what S3 buckets have we got and they'll get some sort of you know 401 access denied message uh which maybe will get them suspicious but you know by that point it doesn't really matter right then then you've got your audit

log so what does that audit log piece look like um well it this you know a anyone really like AWS and how intuitive it is right okay so I'm basically going to refer you to the documentation at this point your aw API calls go to something called cloud trail that you can set up to go to um S3 that you can then stream to other tools that can be something specific for logs like elastic spunks hum logic Etc or something more security specialized maybe so I mean if you're already using one of these things that's probably the way to go and then you know how you then alert is very specific to those products but I'll will show you what we use that

work I'm going to show you this piece so we use something called log scale and don't worry you can't see any of this detail in fact a lot of it is is fuzzed out and doesn't show up interesting anyway so there's the query so like hey when showing up in your AWS logs is a user identity username of the super plausible username that you created uh this is something you defined uh you know like internal backups prod or something like that uh yeah when that shows up in the logs so here's an example of it showing up in the logs hey the username is this event Time Event Source all that kind of stuff the operation that was performed

so you could you you refine your query go yep it's good I like it and then you go up and do something pretty much the same to set up your alert uh all I've added here is uh this is just particular Syntax for log scale uh I think it's very important to add a runbook link so that you along with your alert you know like oh what is this about right so Acme com run Bookers wherever your uh docs are you Pro you might find that your log aggregator has like some way of reducing the noise so this would say uh I'm going to be throttle notifications sorry I'll back up like if someone an adversary is there doing the

AWS operations running some script they might make thousands of calls you don't want thousands of alerts so this is this would be saying for each Source IP address give me maximum one alert an hour um little details like that uh help your sanity and then when that fires it looks something like this like hey this is the title of the the alert go here to look at um the data and by the way what the hell is going on this is your runbook for context setting so to tie that all together the as far as the attacker what they see is some credentials that look so tempting you're just going to find out what that can do

and you then get disappointed um and then you on the blue team can freak out because you get something like this and you go oh my God I was going to put this all on a repo so that you could reproduce it and then I discovered that someone did that for me already it's called GG Canary it's a unique Google search term there's terraform for it and everything so go nuts so that's all well and good this is like aws's uh it's like a front door service that you can just go use right it's uh they've provided they've thought ahead that you might want audit logs what if you have an organization that hasn't thought about

that uh who everyone un familiar with npm cool I'm going to explain anyway uh node package manager that's someone said dumpster fire I think I heard that's off topic um um node package manager it's like um where does your JavaScript source code go get its dependencies from um and good news they do not do audit logging Go Go Fish right we we serve files we don't do audit logging um but then if you're an Enterprise that uses uh JavaScript internally you're going to have an mpm registry and it's very natural to go to mpm JS org and say let's have a private organizational repair with you please and they go great here it is no audit logging um so then

as the defender if you're worried about your supply chain like oh my God if if an adversary gets in here and pushes code to our priv private repos will deploy that everywhere including to millions of customers um yeah then you'll be very interested in let's have a canary for that please um that's actually very just just kind of gets repetitively boring maybe in a good way uh it's the same flow um mechanically under the hood it's it's uh a bit different but we're going to have a fake npm token and our adversary doesn't know that it's fake necessarily and a fake npm registry um here's where I very quickly actually lazily reproduced what I did I went to

my favorite llm and I said kind of make me a fake npm registry please um I actually didn't I broke it down a little bit like hey list me the routes cool that looks good now make those regex expressions and 404 when it's not one of those etc etc um so it generated me some code I I don't expect you to read this uh this maybe so basically the point that I was interested in is when it's a real route that an npm registry might respond to and you gave me a real Canary token that I issued that's the moment to panic that's the moment to go oh my god let's ingest that puppy so that we can

wake people up um and what's just really nice about this whole flow is the Advent of llms means you can just invent plausible uh alternatives for existing interfaces well-known ones npm is well known um and it's just very fast like that process literally take me like 40 minutes and you know your mileage may vary right a few hours but it's still it's very fast to stand up something that uh looks like an npm registry oh yeah I was going to make this next slide look nice and I didn't oh Jesus okay um sorry um placeholder so what does this look like you'll end up with a npm RC that's just one way of sharing credentials um and the format there is

SL slash the repository address and then the orth toen um and you'll see that when you try and use this legitimately like npm who am I with this registry obviously you wouldn't use Local Host right um then you'll just get like oh 401 um and uh then on your Lambda that you have running you'll get hey got this request for dash SL who am I oh my God that's a valid token right so I was that was all of that by the way I covered npm uh the reason I picked those two is they're quite different one is there's a service that totes exist and provides you audit logs and you can work with that and the other is oh my god there

isn't anything what do I do now right it's the kind of two polar ends of the spectrum and there are tons and tons of other things you can do and I was going to promise that I'd publish a repo with lots of reference examples and instead I'm going to make you do a bit of work and ask me for them and then I will uh reverse engineer things from work and share them um so go ahead and email me there and I will share that at the end again if you want um so this is then the meat of the rest of the talk which is the stuff I really really really wish I knew before I

started this um which is it's is a tough sell it's it's kind of like telling your kid oh you know when you're get older you I don't know whatever will matter to you and your kid is like n you don't know um it's a little bit like that uh it's I can imagine you might just go it's okay I'm going to go use some free Canary tokens and it'll all be good and I'll be like great and in a couple of years when they go off you can come back and go I wish I knew some stuff so if you can suspend disbelief and uh take notes or or or I don't mean take notes um yeah

this is this is your act moment this is where you can go cool I'm kind of sold on Canary tokens uh I'm I'm going to try and reproduce some of these Lessons Learned so probably the funniest one is bad practice is good practice right and if you can imagine plausible bad practice you should go do that thing because an ad it's it's so common that adversary become oh they did this bad thing yes right uh so imagine you didn't put your secrets in a password manager yes um imagine you just put things on the internet um instead of inside of your VPN U VPC cool infrastructure misconfigurations Shadow it like inattentive code and data manage it's

all totally plausible um and you're going to catch an attacker faster that way um total example from uh yesterday this is Tiny bigger so I don't know if folks remember from the uh breaking the build uh there was an example of GitHub actions where you can put your secrets in Nars yeah just go put some Canary I mean almost everything in this list is a totally plausible Canary token right um and an adversary goes great I'm going to stick in your GitHub action something that prints your EnV and sends it to my C2 and then they will go trip those things and you will catch them uh excellent um cool but how many of these tokens and how

often uh I guess spoiler I me a couple actually three years ago right I put like one token in a widely accessible Place H how good is that um well if we fast forward to when bad stuff happens what does that tell you like uh something everyone could access got used so an improvement on that is uh you know if you if you think of this in terms of laptops it could be Services you have running right if you have a unique one per machine then all of a sudden you know oh Not only was this token used from somewhere scary but it came from this host so you have you have more breadcrumbs for investigation uh

super important um this starts to get more work though you ready uh how often you going to replace that thing cuz if it's like oh sometime in the last three years someone took it from this one place that that's a long investigation window you have to go look at right so your work is increasing the further you're going to the to your upfront work is increasing the further you go this way um but the downstream work is decreasing right if you replace a token every day and it's Unique per host then you know and you have an incident you're like oh on this day something happened on this host let's go look real close that is very different

to oh let's go rehydrate from Cold Storage uh logs from the last three years oh that's that's that's painful um

so please hold

so uh should you uh alert on unauthenticated access so imagine I set up this uh fake npm registry you might be wondering why why you only alerting when a valid Canary token's used why not just alert anytime anyone accesses it um partly yeah maybe I don't want to go into the full details short answer is don't you don't want noise you don't want any noise at all um you want if you have noise you're going to put it in slack and maybe you'll get to it sometime if you don't have noise you'll tie that stuff to your pager Duty and wake yourself up right that's the difference um but I guess to be properly fair I'm coming from a cloud native

environment uh so in a cloud native environment don't do this uh but hey what the no no no no no come back oh no it's showing I just couldn't see it okay uh yeah it's a bit washed out Cloud don't do this corporate if you got a corporate network uh like nmap makes sense that's a thing that's sensible and so the idea that you be in a subnet in map find a thing and then go explore uh that is something you might want to alert on right uh in cloud cloud native environment no your your attacker is following breadcrumbs they are not Network scanning right [Music] um yeah I mean even even if you the

reason I put yes question mark is uh you you you stick something here and say oh alert when I get interacted with you're going to start going well we do actually run a scanner so let's let's allow list that IP address you it just starts to get annoying and fiddly and really what you want is you want to be catching an intruder that is following breadcrumbs uh I I think uh just listening for someone who's stumbling around is um maybe not the best

plan yeah

uh I think that's the highest signal um like if if someone finds a canary token and they go over here to use it that's that's showing you like steps in a chain uh versus someone's like stumbled into a thing I yeah totally not relevant here um arguably relevant here um but this is also not my domain expertise I'm I'm this I'm over here um yeah you could totally stand up I don't know something juicy like a Jenkins server or whatever fits to your environment um and leave it lying around and see who stumbles into it um you got to be ready for more noise someone might legitimately stumble into it whereas if they stumble into it and

have a canary token like ah yeah that's that's pretty suspicious right um I so I guess in part it's uh how much signal how much noise would you want to tolerate and uh I think again if you can be so confident you have a problem you will wake yourself up that's a very different response model U and appealing to me um so messed up my animations that's cool how do you actually get these things places um can be surprisingly fiddly actually uh if you want to get things on the laptops if if you have a mobile um mobile device management solution like jam Pro or something that can work pretty well actually uh production systems get more fiddly uh I had

originally put something in here that a buddy of mine was like no you're wrong I'd said like hey in your cicd just go stick it in like your base image or or something like that or an artifact at the time you're building it and they quite rightly said no cuz then you're not going to know what environment you're in you know um so really you want things to be as dynamic as possible you if you can get down to the granulation granularity of this instance that I started running in this kubernetes uh container or this ec2 node or whatever that specifically got injected at the time we provisioned it with this Canary token that's a lot more

uh granular uh yeah and so it might vary maybe it's anible terraform Docker environment variables you know whatever your uh plausible delivery mechanism of choice go ahead so the question is how do you coordinate with your Ops teams folks putting all these damn things everywhere um yeah I wondered about putting a slide for this and I didn't um so thanks for asking the question uh you can't do this without letting people know it's happening uh so so the game The Balancing Act is for me you don't want to have lots of artifacts saying let's put the canaries over here like that's easy to discover right someone compromises your slack and starts searching for Canary and finds you know

links to the meeting notes that list all the hosts are going to put canaries on yeah um but maybe more the point you're getting to when you can confirm is like mechanically how do you get it there yeah you got to inform and you probably don't need Super tons of people to be involved um you can weave your things into the existing configuration that might be the easiest way to go uh or you can try and like separate it and have like parallel stuff going on but then it's syncing it up can be a bit tricky so you know your mileage may vary depends I guess how how good your relationship is with your your Ops folks

the better it is uh you know the easier it's going to go I guess so you distribut your tokens that's all good uh let's say you're distributing distributing them to Services uh or actually no one better you have that fake npm repos repository do I put that like on my Edge like accessible to the internet or do I have that like further inside like somewhere restricted that you already had to say SSO into AWS to access um whatever is most plausible um um you got to know that the further it is inside your kind of conceptual kind of layers of Defense the later in the stage of compromise that you're going to get that uh that trip wire trigger um so

really if you're looking where all the places you have canaries if you've got the you know the early stages really well covered then think about having some at the inner stages right um but it's actually I want to draw a little bit on the anybody see the what I learned about scooty from orienting a data CER data center talk H I like that a lot so the the Rings they had there I think they had more rings I I didn't screenshot their their talk um so conceptually you know you've got inside the ring somewhere and completely outside the ring which is uh your internet exposed right um but the subtle detail here is you you might get kind of

this twisted up between where logically it is in your defense in depth uh and where it is in practice because you can have something that's totally out there on the internet but if the only way you get there is from a canic token that's already inside one one ring then you you know that when that thing fires they were in this layer and and then I guess the next following question is why didn't you just make the mpm repository be inside then um you know it's if you make things more convenient for the attacker that that's be helpful too um and you know it's plausibly bad practice right some things you just can't fake like you can't fake uh AWS I IM or it's

S3 Services uh but some things you totally can and uh I generally think you can make it quick enough to do to go fake something and that's actually the safer option and you should do that unless you're really really deeply uh an expert in all these different systems and chances are you aren't because you don't know what you don't know um you could get that wrong you could you could be granting actually permissions to things you didn't realize maybe there are uh API endpoints that that do things that just don't require any permissions at all and you thought you denied everything and turns out you were wrong um or maybe uh there's metadata associated with that

uh that credential to say AWS uh and you didn't know you were exposing it like how many people in the room knew that your AWS access key gave away your account ID some okay so right you're not necessarily aware what you're giving away um and actually for attackers in the audience did you know that open Canary when they Sorry Canary tokens .org when they issue an AWS account ID you can in introspect it sorry when they issue an access key you can introspect it and go oh that's on the known list of like 10 accounts they use that's a canary token I'm not going to exercise it right so little details that uh you get to

sidestep I guess if you uh build your own uh but then comes with danger um if you're going to self host something like you're going to take some uh open source software or and go install it and then be consuming its logs I kind of feel like you're heading into a world of pain because that's now something you have to maintain right you got to patch that thing uh you got to keep the OS up to date um and you know maybe it's like a jumping off point to go attack the rest of your infrastructure right uh you can you can try and do some egress uh uh filtering things like that but tricky tricky uh

has costs so uh my favorite is faking services and I probably say this CU I come from a background of software engineering but um yeah llms can can really do good work for you here um but just because you can uh fake uh like the API Surface or Surface or the web UI surface for something don't don't go nuts right um I did one for Jenkins I just did a w get recursive to give all the static assets on the login page and then implemented something for the password Handler and that's everything I needed I didn't need anything else than that um so yeah keep it light um and then I guess you got the operational

implications right uh because you're not going to hear from your uh your fake service when it's Canary token gets used until it actually gets used it could be months it could be years right uh you need to set this thing up and forget about it and how do you do that uh you know you need to keep your your little application running forever right uh you need to know if somehow your hostage running on just fell over or stopped working and one really good way to do that is make a special Canary token whose per purpose is only to tell you if it works still and then you can routinely uh uh exercise it and then

alert if you don't um get something triggered I think this next one is maybe a tough ask but like know your attack paths every everyone knows their attack paths right there there are lots of blocks with arrows uh going this way and over here is the bad stuff this is the stuff the attacker wants out of your system and this is where they start out uh I did I thought I had an animation to drop a whole bunch of evil canaries over here just to make the point of if you do nothing you have this covered already in a bad way maybe I'll talk a little bit about how you know what your attack paths are but just conceptually you

wanted to you really want to drop a few of these around knowing what the different paths are through your system I like oh yeah this path to this bad outcome has like two canaries that are trip wires this one also has two this one's only got one you know that's the best we can do that one's got three that was easy to do um you you really want that kind of perspective that you're not just thinking about One path through your system uh it's actually a bunch of different paths and I'm probably understating just how hard it is to get to that information because even this fuzzed out picture of mine from work it's really honestly I get uncomfortable

because it's just a slice it's just a known slice like for all I know there's another path comes over here right and then there's only one Canary in the way um so I guess this is also this is also a defense in depth idea right each of these canaries is is a chance of catching the attacker before you make the news so you can do that yourself through adversarial emulation which is a tough ask unless you have a very large dedicated security team that is very good at advocating for their time um possibly much more realistic is uh you can get a pen testing company uh that will do an assumed Beach you could say

hey let's assume they compromised they got remote Cod code execution on a particular service in our infrastructure or on they got malware on an employee laptop go nuts really really go nuts for these outcomes and and tell us what you find um and if they're any good they will find crazy crazy stuff that will make you have sleeping problems this is my last main point Canary tokens live forever like they they will probably Outlast you and you like if your cycle time between jobs is like 2 3 years it could be like after you set up the canary token it goes off right be the person who's going to be kind to Future you right um You Want to

make discoverable what needs to be known you don't want a colleague going huh that's a funny thing H who knows I don't know what that is right uh you you I have literally put in alerts um like if you get an email notification like this is not a drill start your incident response now right uh yeah no seriously um because then you know oh oh if you're going the runbook way that's I definitely put in bold this is not a drill or I'll say like this is still a work in progress we are tuning this right you want to set that context up and and be really kind to you at 3:00 a.m. responding to a page right you you

don't want to be having to remember all of that stuff cuz it might also not be you so yeah uh making your alerts linked to your docs is really good um and then you can you know make sure any people who should have access to it can and then that's also the jumping off point to stuff you don't want to be broadcasting like hey here's a spreadsheet of every single token we put in every environment at every time in in place right or here's how we went and did this these are the repos that have the canary tokens right that's uh not stuff you want to be broadcasting but is stuff you want to know but thanks folks

this is everything bad canaries are bad for you good canaries are good for you and uh everything breaks down to that kind of pattern I occasionally blog and that's a place you can go read about it this is a zero not an O um and you can guilt me into uh making available material and I might possibly regret this but I am sincere if you want to go do Canary tokens uh at your org uh I'll I'll give you an hour of my time because like I wish I'd had that hour of my time when I was thinking about it so yeah don't be shy I can only say no cool um I'm happy to take questions that's all I

had are you using any sort of automation to make manageability easier like I saw you had blog scale humo are using like lookup tables to help man manage that you know because a lot of times like I use Canary tokens often for spear fishing put them into the credential Harvester to try to get some attribution what's behind the TDS but it's a lot to manage when you have like a hundred of these you know long-term living because then you have an employee who then gets hired six months later who actually ends up with that same name you're like oh cool so I'm just kind of wondering where automation maybe has helped you yeah I think you're saying we have an inventory

problem is that what you're saying yeah yeah no there's no lie that is a challenge um in my case I have uh I'm mostly focused on the the thing that keeps me upnight the most which is the employee laptop gets popped um so for me having a j script that goes calls a call it calls a service I'm running and that's essentially backed by something that looks like a spreadsheet right um and so in in my case I don't get like very clear information like anyone can call the user whatever they want they can name their machine whatever they want um so I pull out what's there and I have the serial number and that doesn't

necessarily line up to what's in the HR System for what role they have right so I I have this the gem stuff runs he goes hey I found this person I get a notification to basically say you you have a new employee and then I go go um some some things I autogenerate some some things I've just generated like hundreds of them I just got a column full of not yet used or placed anywhere uh tokens uh others I just the it just autog grates them for me um and some I just manually have to go and create a whole load of tokens all at once um but I think the possibly the answer to your

question is uh I have automation that tries to do as much as it can it tells me to what extent to get involved uh I fill in some information um and then the next time it calls home it goes okay cool this is what I'm putting everywhere um but yeah it's it's it's an inventory tracking program yeah it's you know it's it's not fun but it it's I'm I'm motivated because i' I've seen what the opposite is like thank you very much everybody cheers [Music] a [Music]