Getting People Out Your Automation by Adam Winnington at BSides Toronto 2020

we've got another great talk coming up uh we've got adam weinington who's going to be talking about how to get people out of your automation adam please take it away thanks max all right so this is not your classic security talk what we want to do is talk about device configuration i know it's not very sexy but it's a lot of what we do day to day as security people we know one of the biggest problems we have is device configuration it doesn't get updated when the server gets updated or when the server gets commissioned if we can free up the time that we have that's spent maintaining device configurations to follow up on other

security programs you can do something boring but productive like patching or doing some of the other fun things that have been discussed in the other talks today so you have something on your you've got programs that will actually work the way you want them to that you have software that will configure itself based on the correct inputs right it will take care of everything the way it's supposed to and it does it the right way you've written this software or you bought the software it works perfectly in your lab you're very happy with yourself all good considering that the os project considers the ranked security misconfiguration as number six of the top ten this is a great accomplishment how

awesome are you and programs you purchased or wrote but not so fast those changes should still be going through some humans if you skip this step you run the risk of getting fired we don't want you to get fired so what i want to do for the rest of this talk is talk to you about how we can work with the change advisory board or the cab as i'm going to call it for the rest of this meeting right and not around it right the cab it's there for a very good reason it's there to keep your company making money and keeping them out of the front page of the paper so we don't want to have that's that oh

poop moment where all of a sudden you're opening them your front your front page of reddit as a front page of slash dot and there's your company because you've been part of a data breach or the recent azure you can't log into your o365 breach or aws's dns problem or pick your problem you don't want that so we want to be upfront with cab and show them that operating your devices with your software whether it's homegrown or whether it's store-bought is actually a way to make the overall business risk better right your security posture improves you're more agile whatever buzzards you want to throw at them throw it at them to make them happy the

first thing we we talk to them is pretty self-explanatory scripts are not humans because cavs are so concerned about the risks involved taking the human parts of things so your scripts can't be blackmailed your scripts can't have a fight with the family your scripts can't be sick your scripts can't have gone out partying the night before and come back a little bit hung over and not paying 100 attention to this to the slides sorry to the pro to the config that they're typing right once we let cabs know that your programs are not human this lets us set up a separate pipeline so hopefully if you're not the first person to set up a separate cab

approval pipeline for programs you can go through and work on that right so somebody else has done it for you please follow their work take advantage of the fact somebody's laid the groundwork and blaze that trail follow through and see where they left off if you're the first person to talk to your cab about the fact that this is a not a pro a person but it's a program be the one to place that trail show them that your program is optimized for doing the one thing that this change is going to cover that one thing is something that you have tried and tested over and over again and you have evidence to show them

you don't want to show them things that fail all the time you want to show them that it has failed and you've corrected for it you've gone over your corner cases you've talked to your subject matter experts you've talked to the rest of your team you have a program that does exactly what we need it to do if it's something as simple as logging into all of the switches and changing the description of the ports to match what the cdp neighbor says great tell them that right but look at it and explain to them how awesome it is that this program does this repeatedly it does it on all the models of switches that the company owns

and we're ready to do it now the soft state programs don't change right if the script you have is homegrown of course it can be changed to do other things oops right once it does other things but that is a process we show cab that this is all source controlled everything goes into your bitbucket or to your github or to your gitlab or to wherever you guys store your code making a feature request has a jira ticket next to it going through having branches testing branches signing things off into master doing the appropriate tagging so that way we know that the version of code that is running in production has been thoroughly tested and approved makes cab

very happy now if you're going for bonus points you can also show cab that if you're using any third party modules you have a way to scan those and make sure that they are doing well now once you get approval from cab that you're able to run the program at least once you start telling them about how these things are gonna actually play out in real life so what are we gonna do we're going to run the program at the same time you would normally be running that change yourself if you were typing at the keyboard you're going to run the program at the same time now ideally you're just having coffee sitting at the keyboard making

sure that everything is fine but if that means you have to be up at sunday 2 a.m during a full moon that's when you're going to be up making sure your program does the right thing and most importantly if something goes wrong you are there to catch it live right so we're the cab is already used to accepting these kinds of risks the fact that you're there to catch it just in case should make them much smoother now the other thing we do we start by explaining to cab that we're going to start with a very small blast radius the smallest number of devices we're going to start with is hopefully one right and that's hopefully going to be

the one that affects you personally or is closest to you so we're able to work with one device make whatever change we need to make demonstrate that it works in production and get things working and then slowly start to ramp up once you get one device working you're then going to be able to roll it up to two devices once you have two simultaneous devices working you can roll it to five simultaneous devices then 10 or 100 or whatever units you like to do right so if your company only has 100 switches maybe you build up right first night you do one then two then fives and ten then fifty right you have to get it

right so that way everybody is comfortable with the fact that you're getting it right at each time maybe you start with one the first switch you do is one model the seconds which you'd use from a different vendor the third switch that you'd use from the third vendor build up that confidence with your cap work with them so that way it always goes the way it should because sometimes it doesn't always go the way that it should so what i suggest is your initial deployments to get cab used to the fact that you do have automation is you make some small changes to an existing flow if you have a load balancer or if you

have something very similar that you can just tweak a little bit adding a member to a load balancer group that's a great first script for somebody who's doing cab work right it adding a member to a group on your firewall rules that's also great for script what we want to do is make sure that we're just doing small little tweaks right i'm actually of the opinion that putting a whole new load balancing system in right you can script out for sure but that should actually go through cab on its own right you want to create a new website you want to create a brand new vip you want to move your dns records in that should go through cab as its own

change sure you can automate that but that goes through cab but modifying that is something that can be completely automated then what you can do is explain to the server operators that when they turn off that server so that way they're able to go through the script to do their backups or to do their patching it will automatically be withdrawn from the load balancing group so your end users don't see it the patch can go through and be tested and then when they're happy it can be added back again automatically to the load balancer group that means you don't have to be up all night you'll be up on it the first time but in the long run this is pretty good

now the scary thing is you always have to have a rollback plan you want to show cab that if something goes wrong you are prepared now there are two types of things that can go wrong that i've found the first one is you didn't test enough in your corner cases and in production there's some weird corner case that you just hadn't planned for and that's fine once we figure out what that corner case is it will blow up in production you're there to catch it you get as much information now you're able to catch for that specific corner case you've got production fixed you show them that it will never happen again because now you've added it to your

testing routine and away you go the second thing though that is much more dangerous is your sources of data are wrong so your sources of data say something that you are totally clear to do but what that actually does is winds up somehow impacting the company from their ability to make money right i don't know about you guys i really like making money as a company until i get paid so if the script breaks that you want to explain to cab that hey not only can i revert my changes but so can anybody else on my team so if i go to bed at three in the morning because that's when my change window closes

and somebody detects that there's a problem at six in the morning they can call anybody on my team and have those people on the team be able to execute the rollback steps that are required to get them going to show cab how easy that is i've actually in the past made a video of the change being made on the device and the rollback procedure and how easy it was so that way cab can be really happy with the fact that yes this is a very complicated change but really we're executing two commands go and restore right once we have cabs buy off on that life is good now as darth vader would say a fighter lack of test disturbing we

want to have tests now when you're changing people's devices you hopefully if you're lucky you have a test lab if you're really unlucky your test lab is also your production environment but we do what we can but what we can do is within our scripts within our programs we can test so if your job for that change is to go through and make sure that let's say you you get you grant somebody access to an external website maybe what you do is before you start your script you have a check curl tries to connect to that external website make sure that it fails then you run your program then you run your check again curl goes to the same external website

and it passes if it doesn't that should kick up an error to somebody and that way they're able to look at it and make sure that everything goes smooth once it's been restored now these type of tests and this type of evidence doesn't have to be within your device it can be with it sorry within your script it can be within the device itself so what we can do is we can rely on the device's internal audit logs to show that something has gone well so you can take a look at the audit logs themselves to validate that if you were deleting something it has been deleted and removed now we don't want to be sneaky about

this when we're trying to create a new pipeline for changes through cab for automated things we want to be way more communicative than normal what we want to do is we want to send our usual emails out to our team but we want to make sure that our automated scripts don't step on somebody else's changes that actually makes them very angry and they'll start putting roadblocks in your way so that way you don't get cabs approval as easily as you should so we're gonna over communicate communicate to your team but when your work is done go through and communicate to the people who might also have changes that same night say hey what i've done has been completed here's

some evidence if you need it or you can always call me and we can talk about the changes if you think that my changes from my script have somehow impacted your change right so having that evidence is golden now i've talked about sources of truth right and this is surprisingly tricky in most changes what we have is tribal knowledge and the way the wind is blowing dictates how the actual change is executed is this is this device part of the web server group is this device part of this application right if we're looking and relying strictly on tribal knowledge then yeah you guys can do the right thing but what we want to do is find

somebody who's got an authoritative list the odds of one person having one giant authoritative list are pretty low so what i found is that i can usually pin down the authoritative list on the websites is good this authoritative list for each application is good the authoritative list from different people come from different spaces now what we can do is we can show cab that we're making decisions based on people who have those authoritative lists now hopefully those aren't excel files but there's something that's api-able but if that's what they are that's what they are you've got a list it's been signed off on then we can start making interesting decisions now with those golden pieces of truth

we're now able to make some logical decisions if this server is in this ldap group then i can do this if this person is in the seldop group then i can do this making sure that your script is able to make decisions based on sources of truth help things go smoother when somebody goes through and says hey your program made a mistake you go oh great my program made a mistake awesome let's fix that first of all and then let's find out what went wrong right usually what happens is we find out that something went wrong because our sources of truth aren't quite as golden as we had hoped so that makes things a little more

awkward but in the long run it will make your scripts run better and better and better so wrapping up i know this is nice and short right we want to go through have lots of testing right programs are less risky because they don't change programs are less risky because they do throw testing for you right we have a rollback plan we're able to make modifications but we make modifications through a process that's been approved right we're able to we're able to monitor our changes to make sure that they go through at the right time and you're there to make sure they don't go crazy we're going to get as much evidence as we can in our scripts

and we're going to make sure that we notify people of the change and the results of our change so thanks to the besides crew for letting me talk thanks for you guys for listening i don't know if anybody's got any questions but there we go yeah thanks thanks a lot for the talk adam uh there is a question that came in and that is typically you can detect that change does not meet functional requirements within the same change window how do you detect within the change window that the change has or has not created new security vulnerability within your production system let's see this is why we add small changes right so what we're doing is we

will be increasing or decreasing stuff based on uh based on the request of the flow so ideally we're adding a new server to a flow that comes from that golden source of truth so we may be increasing the security vulnerabilities because somebody asked us to right adding a new server to to the load balancing group does inherently add another server that's internet accessible but we can if somebody's really concerned about that we can say that before we make our changes we can do a vulnerability scan on those servers before they get added to the to the golden list all right so that's one way you can do that to avoid making anything horribly horribly go

wrong yep great and uh all this new advances and uh infrastructure as code and devops pipeline stuff there's there's lots of tooling and stuff to be added there which uh meshes in with this um one more question came in just while i was stalling that is uh typically you could detect to change sorry i think that's raising the same question my mistake yeah sorry my timer says i'm two minutes over so i tried to rush it yeah no worries um cool you well you did a great job um yeah i think uh this talk and the next talk mesh really well um so we appreciate this and and uh yeah by all means join into the discord uh people loved your

gift game you're yeah i spent a lot of time trying to find the right memes there we go that's the one thing everybody will take away we'll be able to steal gifts well personally i'm all for getting people out of the way to to get stuff done and be more helpful that's the hard part man when things scale up doing it once a year tuesdays doesn't work right in small organizations and it really doesn't work in giant organizations so yeah all good uh so thanks again for for coming on to our b-sides and being part of our grand experiment uh i'll let you get in the discord and ask the other questions and get the next

speaker ready which is at 3 30.