Boston BSides - Becoming a Multiheaded Hydra - Jen Andre

I'm Jen Andre I'm the CEO of a company called command which is a startup based out of Cambridge here in Boston my background is I would say it's an RD and insecurities products um I'm also an entrepreneur so before command I was co-founder of a company called Brett sac which does cloud-based security monitoring which is also based out of here in Boston before that I worked rd at Mandaeans and for that I did several roles at symantec that semantics MSSP I started off working the song and then I move more to like the product engineering side so I have for better or worse a background in security and security products some of the other stuff I do is I helped organize a local

Boston security meetup group here called hack secure we're just getting started but we're starting to hold monthly meetups we do things like lightning toxin so it's basically designed for practitioners to come and talk about what they're working on so I'm really excited about some of the stuff we I'm coming up we've organized the CTF over at rapid7 um and we should have some fun stuff happening there anyway so I realized when I was writing this talk not everyone knew what a Hydra was so I figured I should put some slides here to explain it and thank you for pointing that out um so a Hydra for those of you that don't know is a mythical creature

it's definitely not real but it's a creature and the story of this creature is if you cut off one of its heads to will grow back so there's a whole there's a whole myth all mythological story about it but that's the essential core point of it um and I wanted to talk about how we security teams ah should aspire to some of those traits so the first train I want to talk about is resiliency so um I like this picture here because you see me all of you are familiar with the pirate a and the many attempts to shut off while sharing our report sharing their but I think resiliency is one key point that we can we can take from this

mythological myth you cut off one of its heads to girls back and we a security team should aspire to that short of resiliency scalability is another thing obviously having multiple heads is a useful thing the guy this picture here looks like he's kind of struggling to deal with what's going on here and um it can kind of feel like this as a defender but if you're if you're an attacker and you have to avoid all these different traps you can kind of turn the tables on an attacker as well and I'm going to talk about some of those use cases later on so the key point about this is that we want these traits across not just

like the systems that we're trying to secure and the stuff we're trying to secure but also in our security team as well we should be resilient and we should be scalable so as the organization grows that we are protecting we should also scale to that organization we should be resilient to changes within the environment we're protecting and for most of us we have to be just because we don't take a snapshot of a company or a network at a certain state and protect that we have to always be thinking about the next big thing so ways we fail at both of those things we own the policy in the process around things that don't help execute and I'll

talk about more specific use cases but it's generally the philosophy that we give out these policies or to find these processes to these other organizations several organizations and teams that were working with and we say you need to be doing this versus you need to be doing this and here's a way we can help you get that done I think another way where security teams can fail is we don't scale ourselves so if we are the gating factor for security decisions if we put a process in place or a policy in place that says you know we need to have some sort of uh you know but a vulnerability management process but you guys are you know we'd ever prove

everything or um at for any login that comes in or any access control we have to approve the access control we become the gating factor and it's hard to scale ourselves and also if we hold all the knowledge and expertise about security and every time a decision has to be made they have to come to us then I think we also fail so this concept of the human body net um how many of you have been or seen a security at scale conference how about Facebook cool yeah there's one in Boston recently and I like these conferences because you just kind of see some interesting ways these new technology companies are solving problems in security so this term actually came from

yoga Monica who I think he's at daughter security now but back when he gave this talk he was at square security and this is actually a really good talk to watch because he talks about some of the stuff they did it square which is obviously a payment processing company um you know a technology company that he had to do in order to protect that business and he didn't have a huge security team he didn't have a huge security budget so he ended up having to do some really innovative things and one concept he introduces in this talk is the idea of being a human botnet so rather than you know security taking all of the tasks

that's pretty traditionally owns he decided to introduce some processes and some tools to start scaling that out to the rest of the organization and that's actually kind of the core of this talk there's going to be some you know talk of automation and different tools to accomplish that but really the philosophy is like how do we as security team scale ourselves so we can be resilient and we can be flexible so the first use case I want to talk about is around security monitoring so when you think about how security teams have traditionally done security monitoring you have all these different tools and systems generating alarms they're looking at a bunch of different stuff the cool thing is is like there's

there's actually more I think data collection and more technology today we have all these new different types of cool endpoints that you all these interesting on collection and alerting of things and it's really great except that you need someone to actually monitor these alerts you're gonna you're going to do some level filtering things out but um you know this is how I felt like this is what private why I left the stock because after you know six months of doing that wreak havoc I don't want to do with that triage ever get in my life I want to work on the systems that actually detect these threats and try to improve that so what have I told you

that you didn't have to do this anymore like this this wasn't the only way to do security monitoring so I don't know how many of you saw um I guess three or four weeks ago Ryan Hoover was a straight edge here at slac close to this blog post and I thought was really interesting and it very much ties into this whole talking getting but it's all around how they the team at slac scaled out their security monitoring they don't have a huge you know sock with a bunch of security and us triaging all these alerts what they did was very interesting so oh my i failed on animation but you get the point so the

way the process works is they have monitoring on various servers and in points that you security monitoring right and what happens is that every time there's an alert that happens the Lord doesn't immediately go to some security event manager that queue for security analyst to triage what they do instead for certain possible Ertz is they actually determine who could potentially triage alert so on this particular use case if someone logs into a server right like an application server and runs a command that slap being suspicious on what they do is they look up who's the user that's actually logged in at the time what's a slack username and then they triage via slack so what they'll do is they'll send

a direct message to that person and say hey did you actually run this command that we saw was suspicious that person can reply yes or no they can acknowledge or not and then the acknowledgement itself is actually um confirmed to be a two-factor system they have a kind of an app on their phone where someone has to confirm that they actually knowledge learn and that's cool and the cool thing about it is like it's not really that hard to implement when you think about it so what about some of the pieces that you need to actually accomplish this you need the endpoint monitoring the cool thing is is there's a lot of open source tools that actually accomplish this

stuff and by the way like in their particular use case they were doing some very specific behavioral monitoring this could also apply to application wondering well there's no reason why you couldn't scale those alerts or even certain classes and network alerts to you know more sophisticated members of your operations and engineering teams um you need a means of determining user identity ownership so again like if you're using open source stuff you can use a configuration management database if you have some other internal system you need that as well you can use that as well um you know need to determine who read the first factor notification capability so in this particular example they're using slack you could easily use hipchat

jabber IRC anything else to first get in touch with the user and try to confirm whether or not that security alarm was something that needs to be investigated and then you need that to factor confirmation so an example they used I think they had built an app if you actually go and look about blog it's like a custom app that they built it does a post trip edification but again like you could use twilio or something else some other you maybe the email as another two factor authentication mechanism um so I just talked about that and then then this is sometimes the toughest part you need the technology to glue it all together um to be honest

like this isn't actually that much code to put together and in order to prove that to you I'm gonna show you a demo so pray to the demo gods this works actually first before I do the demo let me talk about the architecture that I came up with to replicate this so you see here like I have this example server that I'm to set up using a vagrant virtual machine I've installed system dig foul Scott code because it's open source and it's free you can tell I just like it didn't take me weeks to write this because this was released this week I literally just launched it but like if you want a more commercial solution like

actually the last copy at co-founded threat sac has a very nice turnkey solution and you know for what a support there's other commercial solutions that you can use again like it's not the the actual monitoring system isn't that important it's more about demonstrating the process here and then I have a little agent that I wrote that basically looks for the alerts on that box and then what it'll do is every time there's an alert it will forward it to our deputy bought server and what's that server do it actually connects to slack and it will do well before I even does that it does certain things like it looks up who does the user account

belong to so if it's jai andhra and just logged in it's going to be me and got a look at my phone number and then it will try to notify me on slack I have to confirm that that behavior was me and if I confirm it it's going to try to two-factor message me on to Leo so let's see if I can actually get this demo to work so some of the tools i'm using here just to go over it again these are all free or open source or like open platforms there's nothing here that's you know like customer at all ah this tool here is just a tool that I had to use to mirror my messages it's super

sketchy so after this talk and probably gonna delete it because it's like connected to my android phone at all these weird ways ah let's see if I can make sure i'm logged in here cool those are all my text messages if anyone here is my phone number please don't text me cuz it will show up here in the notifications I'm looking over you commanders um oh and before i get started to show us a little bit of some of the other stuff here um this is actually the repo with the source code again this is very much proof of concept don't deploy this in a production environment it's going to require a lot more validation to actually use

um just to show a little bit about what I'm actually going to learn on I've deployed the Cystic Falco tool with two rules one rule which is to detect what I call privilege escalation and all this does actually let me just zoom in here so you can see this is that good so the privilege escalation rule will just look for people basically running on sudo so if you look at a studio user whoops yeah the user exact binders here corresponds to a pseudo execution if I can find it yep exactly so it's looking for anyone running pseudo or su on this particular machine and um if it's not the agent self because I don't want to report on

my own alerts i'll create this endless loop of stuff that's really annoying to deal with probably more annoying for twilio slack than me but annoying and then I want to look for anything this is here just the system call exec CDE which is a process starting off very simple um right let's go down here I can't actually zoom in here for some reason but if you can see this here the second alert I have is on network activity and again this I think actually just shipped with Falco and modified it just a tiny bit to filter anything from the agent itself because of that infinite loop issue and all I'm looking for is any inbound or outbound FPS

activity from the IP family that's not the agent and um actually has a process named sometimes them for some reason some events come in without a process name that's it so this is the system tool if you're interested in it again I will look at you were looking at other tools there's other open source OS query threats that's a very good turnkey commercial one it doesn't really matter so for the second factory notification i am using twilio twilio is just basically an API for sending text messages and creating voice messages um they have like a nice little rest api and it's very easy to use so i used it because i have to do this demo very fast um what

I've created here is a slack slack group i called it security bot demo and once i start the service you'll see basically a bot come up called deputy you'll see here i also created a channel called security so what happens if i don't actually confirm the second factor authentication or I say this whole you know this thing happened but it wasn't me is that the security alert will get escalated by getting posted to the security channel so the security team can deal with it now again you probably want a more robust process for actually escalating security events alright so now that that's done let's get started with the demo again try not to message me or at least keep it PG if you do so

um the first thing I'm going to do is actually log into my vagrant server credit a user J Andre to do so hopefully this will work cut off um oh so this tool here is just I'm just running basically a proxy so that my agent can connect and post messages to HBS IP like I just wanted to make it more consistent for my demo environment that's not required in a production architecture and here is my security BOTS and I'm using docker basically to start it up so that's what I'm doing right now going to start up my security BOTS and what it's going to do it's going to create an ACP server it's going to listen for two

posts so um alerts is where my agent is going to be posting alerts to when it sees them and confirm is where the twilio call back api is going to post the confirmation after I've applied via text message any questions so far now pray to the devil God's this works so first of all i'm going to pretend i just logged in this box you know on the system admin i'm going to do something that's going to be triggered by our security monitoring system and because i have one of those privilege escalation lips configured it should this should work actually let me just um ah so you can see here that i got a message saying

a suspicious commanders r and so again what happened here was the agent which is constantly looking for message from cystic valco got an alert and posted a message to my slot channel say hey ah you logged in and you ran sudo LS on this particular box now my job here if this is you know this is me i'm just going to confirm yes i ran that and then it says acknowledging and what it's going to do is confirm via my phone and because i don't pay for twilio i'm using the demo or the trial account can be a little bit slow but hopefully a message will come through actually confirming in a second and if not you can see previous examples

of me testing this service out come on twilio

oh yeah so it came in on my phone and then i refreshed right what was coming in so you didn't see the notification but you did see the message um sudo LS was ran on this box at this time now I can confirm um that I actually ran it but like just for the sake of this demo I want to pretend that like hey like someone had actually stolen my slack credentials and um tried to basically confirm the security alert happen for me so I'm just going to hit no to reject and then what you see is the escalation process happening where this where we see it actually in the security channel now again like this probably isn't the

most for both rebuff escalation process you're probably going to want this to go when total an actual ticketing system or event queue afford instant handler to handle but you can see the benefits of this is like I don't have to deal with every single potential security alert that comes in I can actually fan out or crowdsource some of these security alerts to more sophisticated members of the engineering team and off steam and reduce the burden of alert triage on the security team any questions give us a thumbs up again all the source code is up on github it's very much proof of concept like some of the things like the actual looking up my phone number is

just hard coded to environment variables you don't want to use this you want an actual system where you're looking at people's credentials and everything like that so cool now back to this talk

so that's just one example of how through a combination of both technology process and I think security culture we'll talk about a little bit later on we can actually start crowdsourcing some of the tasks that were previously owned by security to um you know other members of the organization second use case and this actually comes from the top that I reps in the first part of the slide on the human body is around authentication and access control so we have this problem like everyone you know every company tries to have some sort of single sign-on solution but it turns out like that rarely ever works especially now that we have all these different services that you know teams like to

sign up for news and that makes the big pain in the ass for the security team when someone reads the company or even switches teams like who actually has access to what and if you see this image on the right is actually from a friend of mine this was a list of his credentials that he had to have at his company and i will say it was a security company so even though yet even though even seen it several single sign-on solutions like the thing i like here is like some other password that is SSO but not you have all these different passwords for things and like someone um when someone leaves the company whoops

the usual process is you get you get this checklist right and someone goes through a disables on this access which is not super scalable and if the security team is a gating factor at all that that's time taken away from you guys to do other things that you probably should be doing so let me get the story again cool so that's absolutely the wrong solution security should not gate access control for things although that is a solution i've seen some teams actually try because i want to be able to wear the compliance auditor comes and say yeah here's who have access to what you know when someone leaves the company we have a process in the sack

gets implemented and also like they don't want to have potential flying around in case they get stolen so the next case study I want to talk about is again from that talk it's from square and I thought this is a really cool solution um so what square did is they built this portal called doorman and what doorman is is an app store of access control so I have all the differences things that people would need access to and what they do is if you want access to like Amazon you go in and you request access to Amazon and instead of the notification going the security who has to approve it it actually goes to the person's manager

because what they figured it was like your manager was the best person the person was best equipped to actually answer should this person have access to it or not that said that brings up a couple questions what happens if your manager just approves everything like he doesn't want to deal with a she's like yes yes yes you know Bob ah who's basically a administrative assistant has to have access to everything and what happens if people just abused the system and you know get access to all the things they don't need what happens when people leave so one cool thing they actually built in this system was basically access exploration like automatic access desperation so if you

they create an account for you on like an application server or somewhere else and you never use it what happens to that account it will actually get deleted over time so things aged out so even if you know someone leaves the company and they go through that checklist I'm assuming this is Matthew automates the provisioning but that's how you still had a checklist and someone forgot to remove you from an account eventually it would still lose access because you're no longer long that you can't get into the VPN you can't log into that one the server so I thought that was a cool concept again by using by crowdsourcing the work to on the managers instead of

making the security team to gating factor they actually are crowdsourcing the work that to do so again this is all cool stuff but there are some actual challenge is to implementing this like you know not every organization can do this and I wanted to point some of that out so it's gonna be challenging diplomat if you have a highly siloed culture where people you know a responsible for their thing they don't want to take any security task because you know they're good I'd say that's securities job you need to have a very collaborative culture where you all try to solve problems together to be able to accomplish this I think if you're outsourcing your IT and operations it's

also going to be very hard because again like there's a contact in place and you're paying this person do this it's going to be very hard for them to do additional work outside of the scope of that contract if you don't have development skills in your security team this is also going to be challenging because again a lot of the system relies on some level of automation being able to figure this stuff out and scripting it and to be honest that's a hard thing to hire for these days everyone wants to have a dead on their screen 18 but not everyone can and again the final challenges you don't have the technology in place to do the things you want to

accomplish like security monitoring the Joe API access to the things you want to instrument that stuff's that sucks challenging you can't really get around that very easily that's there are some organizations where it can be successful I think you have a strong operation engineering shared culture if you're doing DevOps where people are starting that you were they take responsibility for these things and there's a high level of communication you could you can probably introduce this concept you succeed and same sort of thing good cross functional relationships you have an assist or some sort of security leadership that you know crosses boundaries and it can be a champion for you etc and your security team already

have some Deb capabilities or the business is willing to dedicate you know some programmers to a project like this in the name of you know security in conclusion security should not be monkeys in my opinion we should be the experts and we should think of creative ways to reduce you know alert fatigue for the security by crowdsourcing these tasks where possible instead of just providing a policy or guideline we should provide some level of automation um to actually make it easier for the business to adopt our security policies and whoops we should bring on experts what experts are needed and that's what I really think you know in the future the security team is going

to be we're going to be the experts we're going to be that the knowledge but our job is also to educate on the rest of the organization it and you know deputize them on specific security task wherever we can and we should not be the sole owner of security and the benefits of this is that you know bring it back to the hugger thing it's hard to kill you know if the whole security team goes on vacation or gets food poisoning security monitoring and these cash should not stop and that's what we want to accomplish here thank you any questions so there is a paramount to chat up stuff in there yes are there other like chat

apps applications that you think are more relevant to security than just kind of general like the security monitoring use case yeah like you could definitely I saw someone beautiful on post where I think they integrated with my signal sciences laughs so you can actually I think run scams or like few laughs lobster do something but I'm like oh I've also seen someone else for someone else like you could set a command to the bond and have it run at em up scan or you know even have it go fetch logs so based off of a like a security alert that comes in maybe I need more information to determine hey that was me or not maybe I was drinking or something

I mean I needed the other logs just for contacts so maybe you could have it you can send a command to the bot and have it fetch other logs around the same time so you can actually use the bottom more than just like a little bit initial notification you can actually use it as part of the incident handling and investigation and even projecting part of response process

think I'm early