Boston BSides - Breaking Out of the Silo - Julian Defronzo & Justin Pagano

drinking out of the silo the Inca rods through the automation explain exactly what we mean by that a little bit first we're just going to Google quick introductions so but in philly and I'm Astrid engineer rabbit 7 and a lot of my day to day is spending time doing definitely floor tuck belt and you know while I am honey new on this computer to charge its part long as though the ringer I'm Justin without on the information simulated rapid7 specifically overseeing the operations and engineering punctuation so let's dive right in going back to the title and slide what we need by silent silent automation broad automation silent automation we all know how to apologize miss okay think about a tool that does a

specific and sort of categorically within instead of tasks on floor abilities fire one it's automatic most tasks for you vulnerability scanners automatically discovering assets discovering the corner abilities advertising them and reporter Rob automation on the other like a mousetrap you're taking disparate systems that all do and specialize in different things and tying them together and tying them is together in the context of a broader workflow process we're going to dive into some use cases later on that sort of mean generic operating definitions so why are you talking about broad automation what sort of problems that we're trying to solve with it there's a pretty well-documented shortage of security staff in the US across the globe the u.s. monomers valve

from insolence 209,000 unfilled but needed security positions across the balloons about a million according to cisco and and flip side to that problem is you know we as defenders and security practitioners definitely getting better at our jobs in this case when I are getting better responding to incidents but attackers are getting better faster and that's a problem and that's sort of a gap amina focus on the things i hope you look back in the industry kind of see that major focus in the 1990s alkenes but it's focused on meeting prevention so you had tools like firewall maybe ids IPS and either a prime example of styling automation you can automate within their own world context of the firewall you can automate

pools and policy deployments same with ids example heavy and maybe you could change some response activities for that but you couldn't really do much more so maybe a firewall talked or 90s but not much more be on the security list and then moving to late 2000s we kind of had this to detection so we had on favorite tool to sim and you know it gave us access to all the data all of our event sources and you know we can build rules and do all this fun stuff and but still limited in the kind of detection days and then moving into the 2010 to come to move back toward prevention again but now it has a new name club next generation so

you had your next generation EV so fire I'd sandboxing and experience of firewalls backup application firewalls and we're still kind of limited in the context and not really focusing on a little skraelings that just takes us to broad automation and you know sort of going back to what Julian was talking about before in the context of the security lifecycle the trends in the industry focused on prevention a lot of tooling around prevention a lot of money going into prevention we have security practitioners today we know we talked about how sort of short sight of that one now we need to focus on detection touring a lot of money back but in this life cycle that we're all familiar when

it seems like the last stage hasn't gotten a lot of love in terms of tooling and automation and budget and if this is a sort of the second aspect of brought on you were talking about automating full lifecycle whatever we can for any given process and feed it into your detection tools and then take that out of your detection tool and manually run some process to correctly respond to and before we move on just one of issues and caveats we're not saying we're going to automate everything one hundred percent we're not saying this is a silver bullet we're not saying people are doing this today what we are saying is this is a basic foundational building

block that not to me in part of every security program whenever we have the opportunity to put it there this is something we really should have been focusing on years ago so that we could free up the time that our security personnel have day to day so they can focus on the things that humans are good at and creative thought engaging in dynamic processes that are predictable repeatable and also writing more code to further refine your automated processes so before we go on tiny strategies that we have we have to cover some basic assumptions any tools that you have or plan on getting we're going to have that you can get I capabilities so that you

can interact with them in a automated fashion they have to be open and rich enough to do a minimal job they're not going to everything that you want right off the box so you're going to have to you know kind of tie different systems together to do one thing to example this at rapid7 is we have no bunch of different data sources for our users and as great team you what is it going to consolidate all of that so we integrated our single sign-on provider KPI if you join us provider and other tools and vegetable together to this blending systems that we have one single source for our users and then we start using that brother another assumption is

programming skills to be able to interrupt the API is until these tools you have to go to program you know you ought to be a bleep row grammar you know rapid7 our team from big on passions that we can program it to be taught just the way of thinking is what we're big on and the last assumption is it's time and you stuff problem issues as easy as applying a tool of solid copy you're going to have to tie different tools together you're going to have to you know figure all that out it takes time a very handy oh it's worth it in the long run to be able to have all your tools tied

together in an automated fashion so so the core features of strategy behind brought on mission again I'm talking about automating repeatable processes that you might be doing man today we're all starting about automating across again from disparate systems and tools but also across different students whether those are teams within your security organization or between security and I teeny and DevOps an HR marketing I'm sure you can think of some processing on a date in a mutiny phases where you have certain rules and security team partnering with certain roles and adnan street if any of those steps in that workflow are repeated and predictable why not abated and then another caveat some of you might be

thinking well try going on me of everything it's kind of scary or talking about automating certain actions may not be the changes to certain systems yes that can be a little scary but let's extend that idea and think about why not how to make full backs of those automatic chambers again for talking about predictable actions some changes we should know and be able to predict what you will match to those changes should be too quick a problem that might be agreed by one of these automated steps again want to highlight that we're not going to be able to automate everything if you do have a workflow though where you can automate ninety ninety-five percent of it but your organization

isn't comfortable with automating certain steps especially the steps that make changes to systems throw in some manual steps and push button approval stages to to fit your organization's needs and we'll have some in there next to use cases we have some ideas for manual steps that we see that probably will come on yeah probably reasonable to have a feel just in case so use cases vulnerability management and patch management show of hands how many people have dealt with this or deal with this today it's a pretty common use case absolutely necessary to any organization and for these use cases we have this legend down here Green stages are automated orange are manual and and this

is in our eyes sort of a average current state and a lot of companies you have your vulnerability scanner it's automating that stage you're manually submitting tickets and change requests you might have a patch management tool where you know system administrators queuing up patches to apply to systems at a later date then that that person will send out a notification to the company or potentially impacted users say hey patches are coming down your system is going to reboot in 24 hours just so you know patch management tool applies those patches long skin re scans to validate that the patches took and then someone goes in and closes out those two tickets this is clunky this is going to take a

lot of time and it's not the most exciting thing to do every day or every week why not automate as much as possible there there are plenty of opportunities especially with modern tools modern ticketing tools and vulnerability management tools to tie these things together so that any time you can predict these vulnerabilities from my vulnerability scanning tool they pair up with these patches in my patch management tool let's get them stage let's get them rolled out in two weeks between these times we know what our maintenance window is we can add that into this automated workflow and we have minimal human intervention if this is scaring you though I might scare you a little more if you want to get a little

more creative here and you're thinking well my organization we can't tolerate having workstations out there that remain unpatched for two weeks we have users who aren't connecting to the corporate network very often and you know the last time we saw them they were missing the past two months of Microsoft Patch Tuesday updates if you have notifications going out to those users saying hey you need to work with IT to get patched in two weeks or your network access gets revoke you can tie this workflow into your network management systems or firewalls to automatically quarantine that assets so it can't interact with any systems on your corporate network until it gets patched and until your vulnerability scanner

tool validates that the patches talk I might be freaking some of you out let's say you don't want to automate everything you you don't want those patches to just go out willy-nilly throw in a manual step where an IT administrator someone from the security team used to go in and push approve on that change request in order for that workflow to proceed going forward and again any of these stages you can just throw in manual steps or leave any stage entirely manual if your organization deems it too risky to make these changes on the fly like that and I just want to highlight to you know we were talking about tying together for different types of systems it's very

doable stuff we do and are doing more of it rapid7 every day and it really can help your team's focus on more important things to help mature your security program and to deal with alerts that are coming in on a day to day basis this really is a time saver so in other use cases I mean much simpler than what justin has covered but it's a firewall configuration management so that goes from defining your policies and standards and rules configuring those firewalls and you know hopefully auditing those policies and the rules and making sure they stick to the standard and then reacting to any deviations you know all of this is very manual across lot of teams you'll have a

team may be defining the policies and standards another team you know actually writing the rules and then maybe have another team outing policies and rules and then they're going to respond to you know having another team maybe fix those deviations or alert or some kind of process around that and you know you're covering a lot of tools a lot of teams and you can kind of see this is pretty hectic and you could definitely have some room for automation so here let's say we have a policy in place so this is your defining polishing standards for your like web servers who have orch 80 and 443 open to public addresses so it's not RFC 1918 addresses and you drop

those policies and then hopefully you have some kind of a movie tool or something a little bit easier more automated than actually going to the firewall on writing the rule and command line and then you have an automated process of going through scanning those rules and policies and making sure it's all standard but then you come up with oh there's now this rule that has 422 open to some random public IP addresses that you don't own and fall now you're stuck like what do we do so you have automated process that goes through alerts or information security team and maybe hopefully automatically they're coming out that rule track down through your config management repo team who put

that win and following up with that and you doing that process and then you have ability to go back so during this process you find that oh this we needed this for some troubleshooting probably a bad for this use case because I don't a white one 22 ok but you have the ability to roll back and then put that roll back in place there's simple fashion versus having a bunch of tickets open and manually following that up and you have any questions so far so another use case as I'm sorry good so your automation tied to keep your product such that if you were to like swap out your firewall we have to change a certain amount of

automation you know is there like the kind of life cover like a plug-in layer that allows you to something like that has now so at least on our team we try to when we write services for those types of tasks we kind of make a generic enough that like oh I want to block something this and then we'd write plugins like you're saying for this particular tool and then when we're looking to purchase either replace a tool or a new tool we look for stuff that has similar API capabilities so that you know we're not stuck and like now we replaced our firewall but we can't actually block anything through an API so that's kind of always in our

mindset to be able to do that yeah so another use case as defenders is responding to fishing alerts and stuff it's pretty common we're all doing it and it's it's crazy because you hopefully maybe have an automated tool that where to maybe you have your users send you a phishing email study for you to analyze you have a bunch of tabs open for different analysis tools virustotal malware some kind of sandboxing to see if there's any kind of malicious attachments and then urls go to Europe riri to get a screenshot to see what that website looks like then you have a message trace to see who else got that email and then track that down notify

any users who may have opened the email and see if they interacted and then you have to follow up with that once you deem that you know there's a bunch of malicious domains you have to reach out to abuse context so they can shut that domain down and then you also have to update your prevention tools for any of indicators compromise so whether it be your firewall AV spam filters what have you and you know a lot is his manual when I first started rapid7 this was a big chunk of our time initially coming up with you know an SOP for it was all manual and so we switched to a significantly automated fashion and it's

going to kick us off into thinking broader and automation so we tied in our you know our web api so we can automatically pull in web phishing emails and I have great services that analyze the links the attachments submitted to virustotal get the results back we've created a service that takes screenshots of websites and they kind of do automatically instead of opening a tab and your query you can start tracing messages automatically assuming again that there's an API for it and you get a feel for who else got this kind of message based on the subject and the from and you know you can maybe if you have a confidence level okay I'm eighty percent sure that this message is

malicious you can automatically delete that email from the mailbox or on the flipside have a push button so if you have like a GUI representation and you don't feel comfortable automatically deleting an email like push this button because you know we're seventy percent sure or eighty percent sure and then we can go from there automatically emailing users to say oh we notice this phishing email and you've got it please let us know if you interact with it in any way and again automatically notifying abuse contacts and updating your firewalls and your any of your tools for any indicators of compromise that you found during your analysis and by automating all this you can start focusing on you

know what kinds of emails are phishing emails am I getting who's being targeted who's you know how can we use this information to maybe even fish our own company for better campaigns and you're not focused on like trying to always react to a phishing email you're kind of taking a proactive stance and you know that's what we're trying to fight so we're trying to do rapid7 so next we have some bonus screenshots and I'm apologize if they don't come up with kind of just did it this morning but uh we have this QE front end with people so I rapid said that we have users submit phishing emails that they see I think it's suspicious or fishing and then we

have our services automatically parse those and it's represented in a GUI this is the first one where it kind of comes up as a ticket and a workflow and in this next one which will be pretty bad but it's you know we if you can see they have some screenshots of any websites and URLs we automatically pull virustotal and get the results for that and it's all represented in a single page so that we're not having you know six tabs open for all these things and yeah so it's you know he's kind of do analysis from there and we found that it saves us a bunch of time other than again like having all this

running around like a chicken with its head cut off which is a lot of times what defenders do so so before I go into the next use case I just want to highlight quickly about the fishing analysis use case how that whole workflow again we automated to a great extent the whole security lifecycle for that workflow we have prevention tools already in place spam and phishing filters a lot of phishing emails get past them so we have detection mechanisms in place that we've automated and that detection mechanism and that were close to fold our employees they will manually notify us they'll send us phishing emails to a certain mailbox but then this tool we built will

automatically notify us when an employee has reported a phishing email that got passed our filters we automate all the analysis and then in the final stages we go in and we automate correction we add the sending email address of that phishing email to our spam and phishing filter so it'll get blocked again and we're adding those indications of compromise to some of our other tools like a file hashes and URLs to web proxies and things like that so we're automating that correction phase so that it feeds back into the beginning prevention and it just continues iterating on itself and it's in effect so anyway Next use case access management we've been dealing with this forever but for us we're trying to come

up with flexible and still secure ways to provision access especially to critical production systems so you know we had this problem where we're trying to both allow people who need access to production servers to get access when they need it without giving them persistent access without just having their account on there until the end of time we felt that was too risky and so you know we've been we've been taking this process which is probably automated to this extent it of companies and and we took a step further and so that a user an administrator or an engineer can log into a web portal they go to request access which is pretty standard in a

ticketing system but in this use case they specify I need access to these five servers for just an hour and they click a button and their account gets automatically provision just on those systems and after provisioning occurs and they log in for the first time their activities being monitored at a granular level and that activity audit log when the user is done on that system gets sent off to a security analyst and they just check out to make sure everything's kosher in this case we're talking about critical production systems we really want to be careful about even for authorized users what changes they're making on these servers and then yeah problem with some of those type systems

is that a person never locks off um so about applications yeah so we could throw in a step there to kill sessions you know that that's a great point to you weren't really yeah Whitney javi out that that won't break anything because you know you have to kind of do the information gathering and say okay these users will need to be logged in and definitely can think of a good use case why that would happen but again like you would say okay these users to login we don't want to kill their sessions because they're made in practices yeah and just another thought to before you even automatically kill the session send out a few automated

notifications we soon you're still logged in do you still need to be logged in after five notification attempts all right we're filling your access welcome good question and for the last slide you know we've gone over a lot of technical use cases and for for some teams we think that when we're talking about automation we're focusing a lot on automating the workflows of a security operation center or a technical security team but really we should be thinking broader if I can across the entire security organization so if you have a governance and compliance team that is getting out a new policy to a company like an acceptable use policy automate that workflow so the document from your

document management system gets pushed out to all applicable employees you're sending reminders automatically you're tracking how many reminders haven't sent so that after 10 reminders let's say if an employee hasn't acknowledged or signed than your policy you escalate to their manager maybe you escalate to the security team again and after 20-30 reminders if they still haven't signed it quarantine the Machine revoke their network access once you get acknowledgments and signatures move them from the quarantine and track that all in real time as those acknowledgments come in it's yes sir with the automation a good film sending the daily reminders how would you say they acknowledge the rate tends to be how frequently do you find yourself

we're here the party places for us it's been pretty great I haven't looked at our resellers for a most recent campaign I recent campaign it was pretty successful I don't like we made me add a few stragglers that had her reason maybe two I definitely escalate their managers but we didn't have a policy in place for warranty we don't want to just quarantine someone's box without like having that notice before like that if you don't sign it we're going to quarantine your box so something that we could definitely work into the future but yeah again we're tying together a lot of disparate systems to free up our security resources time to focus on better things and that's it in summation

automated like that yes of course any questions we have an answer yes so across there's always a full list within many was for that last case you mean so for the vulnerability management use case for example and for others really the systems we're building our I mean itís fair to say there are full stack you know there's the web layer we have a database layer in that database is tracking different approvals and they normal is its North at having the database helps normalize all that data to so we're not just stuck in the data format of the ITSM tool where that approval is clicked and it helps us to maintain that record as well a case

changes are made in the original system we can have our own source of record to double check things if that makes sense yeah and also like we create when we're creating the service we define the business rules of these are the types of people we want for author for the escalations and for their approvals and then I mentioned the user service that we kind of built and based on those business rules it will call it the service will pull in okay i need the IT director it'll pull in that and we can start working off those work clothes so nothing's really hard coded as far as the person but maybe the role was hard-coded that we need to contact this

person you use any kind of common rules framework across all of these different workflows or are these kinds of kind of separate programs right now it's pretty pretty separate like we are just getting like growing our team growing all the services that were coming up with and peach kind of person on our team is kind of come up with their own thing and we're starting to unify things a lot better by having you know one single repository for our users and all are the singular positive for our assets we're not defining all that information multiple times so like that's kind of starting the head yeah well that's a conversation we actually had pretty recently again you're having a business

rules repository of sources because and again to like when when you start automating these processes you're not going to automate everything from start to finish you'll do it piecemeal and then once you get one workflow automated and then a separate workflow automated you can start focusing on that sort of central business rules policy system and repository for it and and that just speaks back to the idea of once you start automating things more broadly like this you're able to iterate on that automation and mature your security program much more easily than you otherwise would have if everything we're still just manual so it seems like the moment your focus is you're saying automate it and

the extent that you're not comfortable with the the automations effective person function stickers previous data and make clear exemption is also have you gone past that point where on those processes that you have fully automate you take it as proof sets out all that you actually have done anomaly detection rather rules to kind of figure out when the automated furthers what your name percent when is going wrong what you need to do work rates on it and what lets you learn that that's a good point and we will definitely start figuring that out as far as like working that into our services like we haven't we're just getting like started and like these all these ideas and once we start having

more mature automated process that's definitely something that we need to like start looking out for so can you everybody is kind of like Robbins just so we understand you know the size of your organization number of users and people information what we've done three of us so far built a good number of tools so we have another person focusing on just one workflow and they're off on their own for kind of not on the

so rapid sevens roughly like 700 800 people and our security team always the engineering operations side is for before for four of us and you know we have it we'd which is Justin and we have a few other engineers that are actually you know maybe seventy percent of time focusing on taking manual steps and you know into an automated process and and then as we start doing the other against thirty percent is manual we start working that as a project okay how can we automate this and then for a good amount of time we were all to an extent sharing the workload on automating a certain process um but now we're starting to sort of crystallized into

specialized roles and center one person's focusing on this over here the others there but in terms of the operations side especially for these processes that are generating alerts that we need to respond to we're all sort of sharing the responsibility around that did that answer your question are you doing up in your Python I saw the Python um most of its in Ruby some of its in Python like we has we hire new like team members we don't really care like what language you're writing and as usually six the Python Ruby because that's usually where our expertise lies but as long as the service is working and like you're able to support it and documentation is relatively good it's

not really all that matters so I'm really looking forward to refactoring everything into Phil's yeah I here go is pretty pretty popular so we're probably rewrite everything so yeah

fishing lake tables yeah so there other zone that you don't have as big as you guys have expertise

to to commercialize it yeah yeah we just work on the security team we don't yeah we don't know our heads are down all day we have no idea what there's phantom there's command what you'll see people walking around a breacher I mean there's a lot of like energy security orchestration the products are coming on the more I think you really have a way whether the cost of those is worth the first right except icon screw himself yep yeah can you recommend just like a good starting point if you're looking for something automated process where Jan first um for us the fishing analysis workflow was a great starting point and I'm pretty sure a lot of companies deal

with that and I guess I'm biased but I would recommend that it's it's straightforward enough you know you can set up a mailbox that employees can send all that into and then you can start to pull out those emails and test parsing them and then move to the next step integrate with some sort of link analysis system file analysis system and you just iterate on that slowly and then all of a sudden it all ties together and they're like how did I find the time to do all yeah we were fortunate for being able to have the mail service that supports API is so like that's where but we saw that capability like oh well we

can easily automate this and we also started like independently developing services because a lot of times if you want to look up a domain see if it's malicious we will get a bunch of tabs outside of the fishing use case so we built some services to do that and then we start incorporating that into the fishing workflow so that everything seemed fine work seamlessly and on up I I guess I had to answer that question with my own bias just because they're probably so many other processes out there we haven't thought about automating yet and that's kind of the point of the talk sort of bring this idea to the forefront of everyone's mind so when you go to back to work next week

and you engage in that day-to-day process you've been doing for the past few years you stop and think wait a second I do this over and over again isn't there a system I could plug into or tie two systems together just to get this one step done and then think about the other steps in that process yeah specifically regarding your fishing flow do you see like how you have treat handle incident response within a success others correction of the UN dismayed cultures and stuff like that a lot of people play from Atlanta it's actually executed malicious he may consider as part of that yeah and we we didn't have stages on there for that just to keep it simple but you know you

can imagine that if you have some kind of ids IPS tool or web proxy where you can see okay I got this phishing email from an employee it's definitely malicious and there's a link in there did anyone else visit that link you correlate that across your proxy tools what have you then you can narrow that down to all the users that actually visited the link you can see if anything was actually returned it was is it a dead link was it just a 40 for something or knows the 200 and it looks like a file gets dropped if if you're brave enough and you know like okay that machine is screwed you can add that

quarantine step to the workflow if you have a remote imaging system could add that to the workflow say okay we're going to quarantine the box we're going to do a little more analysis to see what happened and then once we're confident that we can just blow this away and reissue something let's remotely wipe it a reimage it pick it up later give them a loaner laptop or something like that that's that's definitely something to consider for that work for them to add into it yeah

you