From Cockroaches to Marble Floors: What Happens when You Turn On the Lights?

thank you everyone we're lining back up outside I really appreciate it please give your respect to the next speakers they're going to introduce yourself real soon right now right now if everybody can just take a seat we're gonna start late so we're gonna end a little late about five minutes late just letting you guys know take it away what's up do you guys like the Encore we took an encore you guys had to leave to pretend like you know I [Laughter] mean I hate spoiler alert you're not going to like it no very very happy you're here are we uh kids ago yeah thank you for all coming back how do we project sorry Leeson showing and we can begin we can

upload this to tethered you did a pretty good job now you did you did I do good composure it's a sign of a respectable human being you did you send up slide stuffing you know we can send two slides out that's fine oh oh I see something I see something

maybe all right let me start talking you can start trying to fix Oh beautiful look at this look at this life is good yeah okay now we can start yeah thank you thank you again for coming for coming in and out welcome to from cockroaches to marble floors my name is Daniel Tobin and this is Paul carrion you go through a quick agenda here intro background how did we get here talk about our cockroaches and then what we did to get to those marble floors cool so I actually was one of Daniels mentees let's call it and I'm currently working with a company called primer I spent a lot of time more on the dev side

and Gantt open here he ran the developed security compliance mighty organizations at a fin tech company that we worked at together and he's with us I will now yep just joined sorrow as security lead focusing on data access to security so these are some of the things that we learned working together Fintan company the fact is we've seen all these these have been I think these are three five years old but like where are we going to get enough people actually working in security so like what are the different options like up to three million people there potentially projecting we'll see how that actually plays out but what are some of the the ways that we can expand

the security umbrella so a couple of the big biggest things lately have been this everyone's talking about devstack ops I'm so excited working with like the the DevOps teams working with the engineering teams shifting left whatever you whatever you want to call it and the other thing is our boot camps one of the things the camps they're great you can learn cybersecurity in 24 weeks I guess they are self selecting and it's it's a good start but we need to expand expand out more so one of the things that I've been focused on for my entire career is something that I've been calling in props and in fraps is taking these 3 separate silos of the structure of

engineering and IT as well so I actually spoke at dirty VidCon in 2015 about this like deaths a cops are starting to become a thing then but still even in the past like 5 years like we barely see anything about the IT portion side if anything I've seen more on the security side just being like no we're just gonna take it i we're done I just I can't deal with it like whatever but I the way we've really approached it is that like for each of these areas like you have people on the front lines that you want to engage like they see what's happening like they're getting the tickets in like whether it's like changing your

kubernetes infrastructure or working with some of your high high impact employees like I need how like logging in so like my Windows machine and I downloaded this thing because there's too many like security like guard rails in place that I can't like do my job so what we focus on what I've been focusing on for my entire career is like let's expand security out well like we can't just sit like here and just tell everyone no like we we need to like work with the like on the front lines that are working with like working with the code directly working with like your endpoints they know your endpoints they can tell you like if you talk to

them about ransomware like they're like yeah it's a good possibility that we might be like these are the five people that are most likely to actually get ransomware because person a hasn't restarted their computer in like seven months person B barely uses their computer etc etc but the people on the front lines they actually know what's happening so why like let's go out and talk to those people so thinking about this like this this worked really well for for for us together so if we're just talking about like different operations like the other team that we worked with and we really started engaged more with the QA side like they're on the front lines of like the obligation like

they're depending on your company like their entire like day-to-day is like I have code I need to check the code does it have bugs well why can't we just talk about all the bugs like can we just talk about like security bugs like authentication logs up authorization bugs like they're already looking for bugs you have people that are that's their entire job is to find bugs like so we started to engage like the the QA team to take a more holistic approach to this so some of the the things that started to transfer like from from security like bringing in security we already had a pull request process we would flag certain things for the

security team to review they can't review everything but if they we've like something oh we're changing like a 10 occation here let's make sure that we actually get like the security team to double-check that so we implemented going from LD TM looks good to me for engineering pull request to L GTS for for security pull request all the way into like LG tqi like let's make sure that there github working with the rest of the the tools that the rest of our teams are are working on for years we've been saying in security oh we need security advocates well what about just quality advocates like we're all trying to find bugs together like making sure that you

find people on your engineering teams that are excited about tests that are excited about the process of like running through like CI CD and then bringing that back into like the automation pipeline itself making sure that's like the infrastructure is code so what we've seen like really the devil soccer ops movement and overall like the big gains that we've seen have been like infrastructure as code or being able to have like this openness being at having everyone be able to contribute be able to see was it FLE he said sunlight is the best disinfectant making sure that we have this infrastructure security already operates as consultants there's the most people security I've like ever seen and I think they said about 2,000

people here is that like what 1,800 different companies you are already working like across teams so working with hourly QA teams to like have that idea of a consultant the fact is like again there's not enough of us as is we can't do everything we need to leverage tools to like to scale yourself where or possible what are your engineering teams do doing to to scale scale themselves are they using like Jenkins to be like a cron job that runs everything like what are the ways that we can to work with those same tools so that we're all working on the same problems from from the from the base level and then again diversity really is the key to winning

one of the things that we we've really found is just like finding the goats in the organization Paul loves talking about but without goats we're both on hashtag Rico 2020 at this point like who are the people that you can go and talk to that are excited about all sorts of different things like if you can talk to them in a way that makes sense on like what they're trying to do like they mentioned earlier today being able to like have the same language being able to like people do want to do the right thing they don't necessarily know what the right thing is so being able to like go and talk to them and being like this is

how I see it from the security side but this is how it actually relates to your day to day job fact is we're all hunting bugs like they are security bugs I think we mostly talk about that just so that people actually listen to us but at the end of the day we're all we're all hunting bugs like QA is focused on like 9 classes of bugs and we've taken one class of bugs and like expanded that so that people like actually try to listen to us but trying to go back out and being like we're all hunting bugs like that's what we're actually trying to do we're trying to prevent bugs we're trying to put out

quality software for a great like end user experience the fact is the end user doesn't care if it's a security bug like a feature bug a lucky UX bug like they're trying to do something trying to make your software work like they just want it to work the thing that the really that we really thought about though is like let's take a holistic look like I am certain that for a lot of these companies that have had major bugs found a large financial company had struts out there I very much doubt if you looked at the entire like bug system for that that was the only button like if you really focus on this as like

something this is a quality problem that we need to Sall it's not a security problem we need to solve we need to focus on the quality from from the base level like look across your your your JIRA your your bug tracking system like if you look across like all the bugs like where where are the bugs are they do you see them like across the across the spectrum is one team dealing doing better if you focus just then on the security bug like building those those charts those reports can you find the trends well I go through do some like basic reporting figure out the metrics like where are we as a company in quality like overall

like is it just a security problem or do we actually have to focus on the entire company on more on like security or bugs as a baseline again we need to remove the gates it's it's not working it hasn't worked like we need to stop being just sitting there and like saying no like it hasn't hasn't worked one of the things that I see especially now of like what I'm worried about like in in security going forward is what we've seen with like the DevOps movement what you saw with the DevOps movement is that what happened was that you had sysaid mins that put up a ton of gates you don't have sentence in many companies

anymore they are gone they're like you can't necessarily give a new job at like most of your your startups like here like in the valley no one's trying to say that you should be assist admin and read from the voh manual to like tell your users like they're they're stupid like we need to like remove the gates being able to like work across the aisle working across like the entire entire company I really think that if we if all we do is talk about security bugs like where we're creating a false dichotomy where we're saying this this is my area and if that's what you want it to be it is area and I hope you have enough people

and process in place that you feel not overrun from now being that just your area okay the rest of the company is now running away and like doing their own thing and you're left in your own area again the the real thing that we've seen from like DevOps from devstack Ops is infrastructure as code having this out there as as openness encouraging other people to contribute back it's not just a sysadmin like and like engineers throwing that stuff like back and forth it's everyone actually working together and encouraging people to actually contribute back like you now have a sense of ownership across like these these aisles so I'll let Paul talk about some of the things that we actually did

yeah so we got some coverage of some of the philosophy that I think that we came to and working together wonder shift gears here and talk a little bit more about how we actually put this into practice and then some of the best practices that we would recommend you investigating yourselves so I'm a really big fan of alignment with a set of goals or metrics when you're working with your people or with your colleagues and I think that the you tell people to work on or tell them that are important you know humans are gamers we will game those metrics we're gonna go after those sorts of things and they they can say a lot so this is an example of the three

teams or three of the teams when Dan and I kind of got into this and I was running on the QA side you can see at a high level there is a real emphasis on agile type metrics which are based around having let's say units of work that you can consistently produce it is interesting to reflect on this excuse me and and also some qualification or quantification rather of like the amount of exposure that you have to badness right so if you're an InfoSec it would be a weighted score of severity times a number of bugs so how much exposure to badness weighted by how bad that thing was right I think these are easier metrics to measure but they

also aren't in your control I think that that's a really critical problem so when we thought about how we were going to set up new goals we thought a lot about trying to align everyone together around a very similar set of goals and trying to to build highly leverageable tooling so everyone really felt across those teams like they were charging towards the same thing and obviously we find the people that are excited about InfoSec are still going to gravitate towards and do things like code review from the perspective of making sure that security bugs don't get out there right people that are on the DevOps side are gonna gravitate towards deployment problems but when you start to tell people hey we

can work together and we're gonna be able to move so much faster and get out behind the eight-ball it really frees up people's time to be able to do the things that they love to do and in sort of the the the dev sock ops world they're talking about education you can't educate people unless you're out there and doing it so this is somewhat controversial I think but we focused everyone around the build pipeline health and so looking at mean time to resolution and mean time to failure really downplaying that number of bugs and bug severity we were still tracking that internally because ultimately that's what we want to pull back down but in kind of a stoic sense we wanted

to control the things that we can control right we were making investments we wanted to make an investment that led us when we for example discovered a new problem so a new software defect of any kind or we decided we had a new idea that we wanted to race out to win a customer we wanted to make sure that we could go from ideation to production as quickly as possible and we found that that gave us true agility right it led us it let our people go out and ourselves to go out and really engage with people about what was important and move quickly in that regard and we found that we spent a lot

less time doing menial stuff that we didn't need to be doing ditto with the mean time to failure when you have frequent incidents outages whatever else frequent fire drills you're not spending time on the stuff that you want to spend time on people get burned out but all you're not spending time on making things better the the second goal is codification and I think this is a really interesting one we made a goal to try to produce more of the tacit rules that we had about how our systems work when we were off in our ivory tower saying no you don't get to have that machine no that doesn't get to go out because of violates compliance

rules hey write those down and I can tell you it was scary as to be able to put some of these rules out there and have people come fight you over it but they didn't they wanted to have a dialogue with us they wanted to tell us this isn't good because of the following reasons people were incredibly reasonable I think the bulk of people like Larkin said want to do the right thing and we found that really was able to create a dialogue with other people that was incredibly useful and profitable so we started we started quantifying the number of rules that we were putting into code I'm that really useful sure this is kind of the classic

us everything we saw that we were getting the sort of things you see in the Google essary book where you have a sub linear growth of your operational teams relative to the output or size of your engineering organization and I thought that to me signified that what we were doing was actually producing something that was higher leverage and I was really really happy with that obviously this isn't a real number but you know we can talk about specifics if you want later let's get this one real quick let's talk about the sort of best practices that we got into because I will tell you this was a pretty successful endeavor on our part what sort of things can you think about

doing where you're able to get these pretty large gains everyone across these kind of operational organizations the IT the DevOps the info set QA can do together one really big one was end-to-end tests I'm a huge fan of the kind of behavioral ask stuff particularly Cypress and some of the some of the the assertion libraries and luck that they use that gives you a really good way of describing this is what my application is supposed to be doing it's amazing when I go to consulting clients how few of them are able to tell you in any level of detail what their applications supposed to be doing or you can ask a random Deb hey what is this page for no it also

provides a really great jumping-off points for doing additional things for example I've seen people hook up fuzz testing to endpoints that you hit during the point of say a customer into n test and when there's a failure it's much easier to communicate to the rest of the organization that hey this thing is broken and by the way it affects the login page well that gives it that gives it a little bit of umph we found also the environment provisioning and setup was a great way to do something high leverage the faster people could spin up an environment to test something in the low lower friction that was the more likely it was someone was going to do that she the investment

that we put in across the board there we've got really amazing returns there are some other places load testing tools where we're big ones data data access controls you know so Cyril where Dan just started working that's an example of something not only from a security perspective have I been exploring that for looking at trying to produce things like data exfiltration risk but also being able to look at the performance of different parts of my application and you know which functions are touching which parts of my data stores we also obviously logging monitoring pager duty those sorts of things have really high rates of return we we also found that utilizing things like burp suite QA

people do a lot of the stuff that you do in burp suite as a pen tester they do it using really really crappy terrible tools and people's eyes burst wide open particularly if they're excited about this stuff when they're suddenly given a way to do their job so much better and that that was a really great thing I think another another arena that we spent some time looking at is sort of code based meta analysis there's a wonderful book called your code is a crime scene and one of the big things that I learned there is that areas where you see there there's sort of a by modality right there areas of your code that never

change those are areas that oftentimes when they're breakages are horrible because no one's kind of kept up with the joneses there there are other there's another modality where you see lots and lots of frequent change and that tends to be an area where you're going to have a lot more churn and that churn again like with the the process comment that Dan was making when there's churn you're gonna have deployment problems you're going to have logic problems you're going to have security bugs dependency problems whatever else it's gonna be there so those are good places to kind of start to we mentioned the code review process oh oh I got my burp right there oops let me

highlight one more thing because we're running out of time here but that's the reduction of attack and failure surfaces I think that's a really big thing that I think is InfoSec practitioners we think a lot about how do you reduce the number of dependencies how do you minimize the amount of code doing different sorts of things people in QA love you when they come to you and say hey guess what you have to test a whole bunch less stuff or there's a whole bunch less things that are going to go wrong with your code that's awesome when you're able to consolidate things down into best practices where you really have a freeway or a paved road like netflix

calls it maybe there's a little bit of churn to begin with but you've really reduced the amount of stuff that people have to worry about similarly with using open source tools right if you use open source oftentimes people are maintaining something and you don't have to put your eyes on it as much we heard we heard a great talk yesterday about auditing open source dependencies I still think open sources is in Adana you know I think I think open source is a fantastic way to go because often times you're talking about no visibility into code that your teams are writing internally versus some visibility and some auditing of things that are external that you can at least

compartmentalize so I found that to be a really powerful thing to try to wrap my head around let's see and I guess as a final piece of homework I always like talks the best what they say like go out and do something good tomorrow with your team and so that's that's what I'm going to attempt to do here so if you're thinking about getting started in something like this if some of the other dev SEC Ops talks that have been here they've been universally wonderful I think they're a great concept if they've kind of inspired you the thing that I think with with us and then with with consulting clients I've experienced is that instrumenting for a more data

different approach for managing an info SEC Pro program or for managing operational engineering programs in general that's really giving you a good place to start it also is a great way to communicate to management as to why something is important so one one great thing for example with focusing on they build pipeline health as we were able to take improvements that we're making for example as an InfoSec team and say hey we're actually shipping product faster people loved us for that right I think there's also something to be said for looking at the dev psych ops movement the visibility that's there whether you like everything that's being said there or not they have some very

nice branding and some nice ideas that are battle tested and will resonate with people so I definitely would suggest that you go look at that and use that internally if you're trying to sell it obviously talking to QA is huge I wouldn't be here if this guy hasn't talked to QA right that a lot of people that are in QA see it as a means to get into something else that's more interesting I think that's really sad but that's the reality of our business most of the good QA people I know are trying to become devs or security analysts they love it when someone from Memphis that comes and talks to them because it's an opportunity for them to

grow and to show what they know and for people that give a about what they care about so absolutely go talk to folks I guarantee that you'll find a diamond in the rough for sure and it may not even be all that rough and finally it's really easy to do a quick audit for existing tools that you can leverage externally I mentioned end-to-end tests but there are other things like for example burp suite or Cypress tests or you know even even just reutilizing the CI CD to run things effectively on a cron that can be incredibly powerful and getting to push this stuff over the line and seeing an impact when you're using internal knowledge and you're not only are you

creating those sorts of relationships where you can leverage your way up but you're also getting a lot for free with a very very low amount of risk so I encourage you to take a look at some of these things and I guess that that is that is all for us but definitely please come and ask us questions or hit us up online we love talking about this stuff and thank you so much for your kind attention really appreciate it you may go thank you you