Buy & Build: Complementing Security Operations w/ In House Software Development

so hi my name's David this is bye-bye up but also built complimenting security operations with in-house development this is a talk you may or may not have known what's going to happen right now it's certainly not the talk I expected to be giving in this current moment because back in August I was told that this was going to be an alternate so of course as the responsible individual that I am I didn't write it they told me to be repaired and so I was prepared to forget about it and then as of last Thursday they come in it's like hey we watch to speak on Sunday you know you have your talk prepared right so I've

spent the last 48 hours kind of putting together this talk so I'd like to read welcome you to buy but also built talk written in 48 hours this is uh so as I was putting this together I was kind of thinking how I wanted to present my points in kind of a concise manner to kind of get across sort of the bigger ideas I wanted to highlight and something that my friends and I kind of enjoyed doing in our free time is giving hot takes on things so my definition of a hot take is sort of a sum it's expressing an idea respectfully and sort of a kind of direct manner and then being able to follow that up with lively

discussion on random topics so we've talked about things like oh is a hot dog sandwich was the best way to create s'mores that sort of thing surprisingly Google has a definition on hot takes I prefer that that they draw attention to a topic instead of attracting attention to something so I would like to rename my top again to hot takes somewhat related to by but also built a talk run in 48 hours so some disclaimers ahead of time nothing I talk about in this talk is reflective of my prior and future employers purely anecdotal things that have been kind bubbling up in my mind over the past couple years as I've been working in InfoSec and I thing I say about the

security products like take it with a grain of salt I still love them they serve a very good purpose some of them are great others they're okay but in the end like they in general you do need security products you can't just build everything yourself so if that said who's this crazy guy up here my name is David I'm an undergraduate senior at George Mason University I'm studying cyber security Bachelors of Science it's kind of a new degree a few years old generally we focus on like looking at systems from different perspectives like oh it's taken networking cores like a cryptography course I'm kind of deep diving house security plays a role into that so

that's how I've got my introduction into security with school outside of school I'm the president of student and CUNY and technology we basically build open-source software for the GMU community by the GME community generally involving web apps so that's how I got my like interest in building things is from that organization I've also interned for every summer that I've been in school the best to have been on InfoSec team at a large media company and generally kind of like my quick pitch as to my interests is I'm very interested in building out systems that mitigate different vectors based on prior mistakes that we've made as an organization and then automating a solution to that at scale to the point

where you can't really prevent something from happening again but you can get it to the point where you're like 90% sure that you'll be able to detect it if that vector comes back again so I'd like to give three hot takes in this talk and in the second one I'm gonna focus on two case studies related to building your own solution in lieu of buying one and then there's being a Q&A in the ad where you can roast me for anything that I say so how take one security is an unsolvable problem I think this is kind of something that's been like widely acknowledged but really stated by people we're never ever going to solve security it's gonna be a

constantly evolving thing based on you know new vectors that come out new techniques new ways of mitigation and I feel like that sets a very important baseline for discussion discussing new approaches to kind of handling the problem that can never be solved because security is an unsolvable problem though it's generally seen in traditionally in organizations as sort of attacks you know civil xxxx they just have to give some amount of money security teams so that can mostly avoid something bad happening you know the sort of drill over the past decade go a decade or so and this is just an anecdotal point I believe that the amount of resources going into InfoSec teams has greatly

increased and that's mostly due to high-profile breaches such as like your targets or your effort Equifax or you're so nice and so because of those increased resources you end up with teams having more options on what to do whereas traditionally you just try and get as much information as possible try to you know use traditional methods to detect things now we can try and be a little experimental but as the old saying goes more money more problems the general easy I don't say easy but like the quickest way to kind of ramp up security operations is to buy more security products because you have like little problems in your organization that you could address if you true

through more people at it but you know there's this really neat product that might come in and not solve your problem but get maybe 70% of the way there which is generally good enough and then when you purchase these products you generally try and min/max them to work as intended and maybe even more and this kind of works to a degree and I'd like to kind of dive in on that mid maxing so the way I've seen organizations go when they purchase these products is you generally fall into this cycle where you're rejecting it for like existing infrastructure to support a new product so you've got another damn agent to deploy onto every machine in your entire

network or some like SSH key needs to be placed on every single server oh wow some some SSH key needs to be added to every single server and your entire date of the environment and nobody adds it let's say you buy a new box and need to reroute all of your network traffic to that box so it can you know decrypt it and then re-equipped it and go out the other side and nobody can configure it correctly so you're spending like it should be like a couple of weeks but it ends up being a couple months trying to get it to work in the first place and then you end up stretching the product to do things they may not have been

designed to do so I've been in situations where we've had to like go into admin interfaces and scrape like little bits of data in like an HTML table rip that out and the in folks who like parse that into useful information because it's not exposed anywhere or you've got like undocumented or deprecated API calls that you're using in prod because of course you are and then if those who you know don't work or your kind of throw as much resources as you can at it you know calling support or purchasing professional services to come in and pretty much begged them dag and new functionality or make you think that it works the way it's supposed to

do in my opinion this cycle leads to a dangerous time waste sure it can work in some situations but in general you're just spinning your wheels every time you add something else to and you're really only getting 70% better if can max it all the way and there's also a non-trivial risk that you don't even use the product I think every organization has something that they purchased for thousands of dollars and that they've never used it just in general like it can work but there's a dangerous kind of slippery slope you can go down where it doesn't work so I like to kind of jump off with that too my second hot take which is that buying

security products will never fully address your organization's security problems and I'd like to bring up a couple of examples to kind of hone this n so incident response to a previously unknown vector I've seen security products fail when this happens so let's say some O'Day comes out or Brian Krebs puts an article out about your organization and you need to very quickly figure out what's been affected how long is you know those acts that's being been affected how can you resolve you know this incident that's just all of a sudden come up generally the security products will like follow their knees like they don't work for very rapid things I've seen them work well in

like long term sort of things but you're really only getting 70% if you have legacy like systems or strange architecture implementations like for example I've seen some subnetting setups where every floor of a building is its own subnet so when people walk up and down the floors they're given new IP addresses every single time and so you just end up with like one person owning a hundred devices just because they keep going up and down the floor we're like IOT vending machines because of course you have IRT vending machines those things products don't really work well on because they assume a generally perfect setup and then also that answering of questions so like oh you

know do we have any windows 98 machines still around the answer is always yes I've seen just you never get full coverage with it with any of the products that you buy you get like like I said 70% and then the kind of edge cases or niche problems they don't really do a job of addressing and so this is where I'd like to present the idea of solving your own problems per organization you're going to have weird edge case niche scenarios and the only people that can really solve those are the people who know them that know those problems the best and generally that ends up being you you can try and call consulting people to come in or I don't

know buy more products but in the end you're the only person who knows why this niche problem exists and maybe a way to attack it and the way you kind of solve your own problem is you want to complement the existing workflow so that they do not become a disruption it's very easy just like oh I'm gonna build 100 tools and it's gonna cover everything that our products don't address when you deploy them and then you instantly forget about them because they don't integrate well with what's currently exist they don't complement what's there they just RuPt what's going on because people have to like oh what was that like what was the URL to go

that you know new service or what's the API call I need to make you really need to work in 10 the Operations team needs to work in tandem with whoever is building out a new security service internally and it's an it's an order by our basis on where you find that balance between functionality and disruption I can't really tell you how to do it it's really up to how your organization is currently set up or managed so I'd like to do two case studies on organizations who have implemented solving their own solutions and what to do thing is that these organizations or at least the high-profile ones that I'm familiar with have been done by younger tech focus

companies who have a lot of talent in building out large-scale applications and so they've kind of started to merge those type of minds with their security operations teams so I'm gonna talk about Netflix with stethoscope which is more of a compliance sort of project and an airbnb of binary alert which is more of an Operations sort of project so if Netflix they had a problem where they purchased these endpoint protection Suites to put on everybody's laptops you know just get a franchise iris that sort of thing and the problem is these Suites generally yell at users when they do something wrong and users ignore those Yelling's I mean if you see like 50 dialog boxes

come up saying oh you got to delete this file or you got to move it people are just gonna hit cancel because they don't have time for that and there's really it ends up being information overload these programs that I've seen in general there's no feedback loop to kind of encourage them to keep using the products so you have it on everyone's machines but they're effectively not being utilized in a proper manner so their solution to this was to pretty much pull out as much data as possible from those Suites that they purchased and build a kind of friendlier interface for users to actually use the products in the first place there's a quote here

I pulled from their blog post where they talked about how designing this interface with focused actionable information led to a more secure state within their system then there's an image there the general idea is that like whenever you get a laptop from Netflix as a new hire stethoscope is loaded on to there and all of the products that they purchased would get added into the interface and instead of yelling at you when something's wrong it would teach you how you can actually get that done correctly kind of a show don't tell sort of mindset and so I'd like to highlight this because it complemented their existing products by ensuring that they get used by their users in the manner that they

should be used so sort of correcting a faults with the products that they purchased in a traditional organization you might go to the AV vendors be like hey can you change your boxes to be different but they don't do that so by able to being able to roll this out within a couple of months they were able to get to a more secure state Airbnb they had a different problem where they needed a very specific tool to scan binaries against yarn against yard rules they've developed internally or open-source er rules specifically needed to be private and low-cost something that could just scale out horizontally for whatever they needed fire Steudle does this but it's a public

service and due to certain business requirements that they don't really state in their blog post they couldn't send it out to the Internet you can probably take a guess as to why so they built it they utilize pretty much a bunch of AWS services tied them all together and spun up a security service API that people can submit binaries to and scans them against ER rules and was able to integrate that workflow the results from that of the results from that program into their security operations response so in the diagram they just say they just use SNS alerts but I think this is a really good example of sort of a plug-and-play solution where you have one or two

people build out very quickly they're being able to rapidly iterate and deploy an application that integrates with their security operations team such that it can get used at scale and I just wanted to throw in like I think lambdas is probably like one of the biggest things to happen to security operations teams because even very rapidly build out a service like this for these nice edge cases that you know may not be offered by traditional vendors so three big takeaways from these two case studies high ownership of the software that you build leads to effective software design so you don't have to add in lots of features you don't need to factor in lots of you know

weird user stories you have one thing you needed to do and you needed to do it really well and so because you know exactly what you need to build and you have the high ownership of it you can deploy it and it'll hopefully integrate well with your security workflow a lot of people don't realize though that there's a higher upfront engineering cost when you're building these things obviously you having to dedicate time and resources away from responding to incidents to building out these tools but I believe they reap in long-term benefits and I'm gonna return to this point later and something interesting about both these projects is that they open source and I'd like to make it two

point five here is that open source is the only way to ensure the long-term success of an in-house developed application I'm not going to go into a Richard Stallman diatribe here but I just really want to highlight the benefits of open sourcing you're in the house application and functionally the collaboration aspect is probably the biggest thing that you gain from open sourcing your application you're not the only one of this problem no matter how weird you think your IOT vending machines are someone else hasn't even were set up of IOT vending machines and you benefit from any functionality that people either add to your repo or if they fork your repo you can kind of pull

in any changes that you need it also encourages you to not cut corners or be negligent with your code I've seen a lot of applications be built in-house deployed and I completely forgotten about or there's like hard-coded keys everywhere and it's just a mess by open sourcing it it forces you to build an effective program that can not only be used by you but be picked up and plopped into other people's organizations and I think this is a benefit that now people realize when you take it open-source there's also an idea of strategic altruism which is the term I made up by is something that is becoming increasingly popular companies where they open source all the things to gain

sort of goodwill from the people so Microsoft is in the process of doing that right now where they open source pretty much everything that they do to gain kind of developer trust and stuff so the organization will be highlighted more in a sense if you start open sourcing all the things that's just something I want to add so last I take it is necessary for operations teams to begin taking ownership of the issues not covered by their security products in my opinion it's dangerous if you ignore the 30% that isn't covered by your vendors or you try to stretch the vendor the vendored products to cover maybe not what they don't originally cover it's not a sustainable solution it's going to

bite you in the butt at some point and I've seen it happen at organizations and I feel like taking the time to engineer a few things here and there is in the long run going to be a benefit for you and ensuring that it complements your existing workflow again is better than not having it at all you can't engineer everything yourself obviously no one's gonna rebuild AV no one's gonna be building their own woth you're gonna be buying those but you should know when those products fail you and where you can kind of build in a complement to cover up whatever 30% they don't cover in the first place so in real life at scale implementations

with this IC devstack ops being the biggest sort of movement in people building their own in-house applications it's in my opinion the most mature subset they kind of take it to the extreme where everything is done in the cloud everything is done on lambdas but I think it's a good kind of reference point for what I'm speaking on there's this great book that just came out called securing DevOps I really recommend checking it out if you're interested in this topic they basically step through building out you know piecemeal security services security service api's and how they integrate in with your workflow there's a talk right after this one on Def suck-ups so just stick around if you

interested another example I can point to is I was recently talk with an organization and they spoke on how they do 40-40-20 time with regard to their security operations where 40% of your time is responding to incidents you know resolving issues that have come up your kind of standard practice operations day-to-day workflow but then the other 40% of your time is spent automating what you just spent 40 percent of your time doing so that they don't happen again I think that's a really interesting way it's about in time and then 20 percent is how you double you get up between the 40 40 so if you need more time to respond to incidents you plop it there fee if you

have more time to automate what you were just doing you can put it there sort of a flexible way of dividing up your time so that and all of your resources are going to building stuff and not all of your resources are going to responding to instance all the time I think if you have balance it's all roses and sunshine though as I mentioned a few times then you should be asking yourself a few questions like do you have the resources to build out an adequate solution right now manpower is probably the biggest thing if you're on a team with like four or five people it's probably not the smartest idea to dedicate two of them

away to build out some service really quickly when you you know needs to be continually running your operations you know on a day to day flow so maybe you want to bring on new hire maybe you want to reach out to another team see if they can help you out also compute compute spamming less than issue of organizations whom like abs and stuff but it's still in some some cases like oh you know I need to manage my infrastructure to deploy things on to like do I have time to manage that compute would this with the development of this application result in no effect on your operations like are you just like spinning your wheels like

will this only help like one or two times or will this happen or will this help every single day if you end up building things over and over that have no effect then you're just wasting time and then can you ensure that this will be maintained over time adding new features fixing bugs that sort of thing open source is a great way to ensure this but if you want to keep it internal you should have like one or two people who are pretty knowledgeable about the project being able to pass that knowledge log to others and don't repeat yourself don't rebuild what you're already using only add-on to things that you've already purchased or you know

practices that you're already doing the biggest thing here is just don't waste time so that's well really have like I said I wrote this in 48 hours I'd like to open the floor at any questions but she might have and thank you for coming out

yes yeah I think the biggest thing I've noticed when vendors have come to my team for example trying to sell products is just like over-promising every like the world and moon you know well our product will cover everything you should always have a healthy dose of skepticism in general as long as it covers 70% of your use case I would still recommend buying because you do have some sort of guarantee some sort of contract with them that it'll work in some sort of manner that will help you what I'm really advocating with building yourself is covering up the holes that they ignore or they don't do as well I don't believe that building everything out

yourself is either saying sustainable solution it's sort of a balance that you need to maintain and it's really up to you and your organization on where you draw the balance there yeah yeah oh yeah I believe it was Julian yeah Julian V the hint ju li en v eh ent this like just came out it's it's great I love it yeah yeah

[Music] ya know I think part of it is like a feel where you know that something like your current practice for handling a vector that has come in is inefficient like everybody on your team kind of agrees that this is stupid we should do something about it but also like I said you need to know when it's a good time to start investing resources so it's a very bored by org basis like I said and I think it's nice to have like one or two people on your team who are traditional software developers who have come into security I've seen roles on on teams of like people who are software developers be like code review people or

like that sort of thing you can kind of move those resources around to hey can you build this out in a week prototype it maybe get it deployed and then we can iterate on it as we need to but really it's very situational I the only thing I could say is I would never stop what you're doing to build something out it should happen in parallel okay I'll leave you all to the rest of your afternoon thank you [Applause]