DevOps - Security's Big Opportunity

all right so that was a good talk let's let's give a next round of applause for Peter chestnut DevOps Security's big opportunity thank you everyone all right so I always have this problem I speak I speak at a lot of conferences I was in Dusseldorf Germany yesterday so if I'm a little tired that's probably why one raised our hands one one hand see is the problem people's arms are rusty I try again when I ask questions is good to get like response back is everyone there all right all right so a little bit about myself longtime developer so I came to InfoSec through vera code so 25 years of developer a lot of experience in

application security all right this is my 12th year at vera code yeah anyone heard of our code anyone a customer code nobody oh great room full of prospects love it all right I've done a lot of development stuff what I'm here to talk to you about is helping you understand how developers think and how to get them to work with you instead of against you won't have that problem yes we're developers working against you I have done a crap-ton doing waterfall to DevOps monolith the micro services I understand the architecture is the different methodologies and I consult with the largest companies of the world on how to roll out successful apps tech programs so please feel free to ask

questions reason that I am so passionate about applications security we're in the DHS 90% of the breaches occur in some form or fashion through software now it's not necessarily the software that we create it can be the software that we buy you know do you look at your software supply chain before something gets installed into your networks I don't know that you do if we take a look at what we protect and how we protect it oftentimes when I go into a prospect they talk about securing the important stuff all right well that wouldn't have protected you in the case of target they didn't get in through the main target website or any of that stuff it was

social engineering small shop that did their HVAC that went to their billing system that I mean their network now I get everywhere so really it's not about securing just the important things open source so you know great current thing now I'm gonna challenge a couple of speakers that have already been up here and talking about the fact that Equifax chose not to patch or as knew it and did nothing I will assert to you with pretty pretty good certainty that they didn't even know that they were vulnerable right in the case of struts that's an easy one I had a good conversation with some people at lunch today it's like alright so struts you could probably go around and

ask you developers do you use struts to use struts and figure out whether or not you even have it in your enterprise but what if it's something like Apache Commons do you use Apache Commons I don't know I use JBoss the answer to that question is yes if you're using that or if you're using strut it gets pulled in but way down deep it's not a first-class citizen of something that you import into your project cuz I have to have Apache Commons alright so who has an abstract program no really nobody I've got one guy two three come on they see the arms still aren't working for okay well alright who wants it aspect for round you don't have any of your

company really so it does it look like this or does it look more like this because more often than not you know it's the sea snow I am the director of no I will tell you you cannot ship your application because it has stuff wrong with us we are not enabler as we stay in there at the end when they're ready to go out the door you on a Friday afternoon at five o'clock is like I want to get home like nope you can't leave yet I still got some more work for you to do now typically in companies that have an application security program I'm going to see a couple of different outcomes right if I have an app set

program and I have rules and regulations for what can be and cannot be inside of an application oftentimes I see this right there's a gate but it's either they don't have to follow the process because there's no mandate or it's the escalating phone calls wanted to make an exception so I'm gonna call my boss who calls the air boss eventually we have the CIO and CEO says we have to ship this because I own the amount of money that we make in here and I need this feature to enable sales so it becomes a PL problem and not really a security problem or if we do have a gate it's usually this developers ignore the

fact that they're going to have to do security vulnerability testing they get to the end we do our pen testing who does pen testing other applications in their enterprise ok they get they're like oh sorry can't go they were hoping the gate wasn't real with some figment of their imagination now the reason that I'm here is because the times have changed for how we build and ship software and it's important for you to understand that because the stuff that you do today the things like pen testing while still relevant and I'll show you where are not the be-all end-all we can't do it at the end anymore so if you look at team sizes and release timelines

for waterfall anyone doing waterfall it'd be proud it's okay been doing it since the 50s why shouldn't we continue to do it exactly we're talking one to four releases a year it's really slow and I got a big team doing it too now this is the nice thing about this for security is it moves so slow that we can kind of keep up with them almost kind of right now if we start moving into agile agile we're talking about doing one or two releases a month or we're really starting to cook now how do i do pen testing if I'm releasing twice a month I don't know if you have an in-house maybe you could do

it but that would just encount that allow you to do the testing never mind any remediations or mitigations that you would have to do on that and it's a very small team moving really quickly sometimes multiple teams combining into a single application if we get to DevOps now we are haulin ass we're talking about multiple times a day and hundreds of times a year there is no way to pen test everything that goes out the door if you're using DevOps who here is doing DevOps in their in their shops some people may be kind of alright good this future stuff so let's explain a little bit how we got here so this is what waterfall looks like I've got my silos

of responsibility and those very much follow the way the built we build the software right someone tells me what I'm going to build then I go design it and I coat it and then I test it and then I test it some more and then I go out the door with it right now the problem with this is I lose fidelity as soon as I do that handoff from the time that the person says here's what we're going to build so the time that it gets to the people at operations or the people in security I don't know why I built it I don't know who I built it for I don't know what it's supposed to do that is a

huge gap that makes these people just have to be reactive similarly with the people that build the application so our developers they know the choices they made they know the trade-offs they made but by the time that gets down to operations they know they went the way it was intended to run but operations know how it actually runs in a production of our wood production data and the problem there is their information never makes it back so they go crisscross so I don't understand the trade-offs they made or why they made them I'm just angry because the thing shows up and it's on fire all the time and I have to kick off these scripts

every day because you have resource leaks and I have to make sure the process stays up it just it sucks so enter agile now in agile the first thing that we did was we started to combine some concerns which is what we're going to get to with DevOps really we started to combine the concerns of development and quality in the old days in waterfall we'd say the developers say I am done I coated it here you go we throw it over the wall we go off and do something else quality people will come back and say yeah not really here's all the crap that's broken well the stuff that I broke let's say worst case I did once a

year two deployments maybe January first I wrote the problem Virginia way second I wrote the problem I'm finding it on December 30th how many decisions were made based upon that one bug along that entire here that are now gonna have to get revisited and retested so it's really not a very good run way to run an airline so we started to say alright we'll combine them and they'll say you know what you're responsible equality is not done until it's tested so I built it we test it and then we say done and it goes out the door but we still have this problem right so we've still had losing business intent we're still losing application knowledge and we're not

getting any feedback on how the thing actually performs unless we get to call into some incident where our productions down and we've got to go in and try to figure out what happened so after DevOps or in this case def seconds because it actually have security in there high card security love you guys because we're all in the room when we talk about functionality because we're all in the room we talk about testing and requirements and operations we have really good continuity so we can start to give each other feedback in the room in that team about how things are running now let's think about this from a technology point of view in waterfall again once a year or maybe twice a year

I'm doing deployments if a test takes me two days and it is grueling and sucky like this now some people choose to do this for fun that's not me if it if it sucks it takes me two whole days to do it it's just there are tears in my eyes when I'm doing this thing but it's gonna take me three weeks to automate that that means my pay at my payback is not four years I might not even be working at the company why would I do it you know what I'll just deal with it I'll grit and grin and bear it and we'll get through it and we'll do it again next year when you start to shrink

those timeframes and now we're doing a release once a month or twice a month things need to be repeatable now so I have to think about putting some automation in and saying when I run the test it better run the same way every time because I have to trust that machinery in order to make these releases happen when we get the dev ops all bets are off everything is automated everything if you can find a way to automate it you have to because it will just slow you down okay I'm gonna make the presumption because you guys aren't good at raising your hands that someone wants me explain this agile process so this is typically

thank you this is typically the way DevOps is rung it's run on an agile methodology whether that be agile scrum or agile Kanban this is agile scrum here so the guy on the left with his with his the little light on his head is our product owner that is a role inside of a scrum team his job is to look into the future and decide what the most important thing is for us to build and he's gonna build a list of those now dissimilar from what we would do in waterfall would say I need to know everything we're gonna build for an entire year for this I only need a couple of weeks all right or maybe one

week I need to know what the most important things are for us to do now things that will customers will actually buy so I'm gonna prioritize that list now we're going to go into between the prioritization and the planning we're gonna go into what's called a grooming session now write that one down because this is the one that you guys want to start thinking about attending inside of a grooming session we're going to talk about the story we're going to talk about what the use case is who we're building it for why we're building it and we're going to make sure that as a team we understand it so let's say these these first three rows here this is my

scrum team we're going to do something call it we're gonna test for understanding and we do this through something called planning poker so there's actual physical decks of cards you can actually get it on your phone as well you go into this meeting we talked about a feature and we're gonna talk about how big this is how long do we think that's going to take us so we do this in what's called story points their virtual some people tie it to hours or days except early you're not really supposed to musical say this is some understood quantity of complexity and so we're gonna say is it bigger or smaller and by how much so what I'm holding in

my hand is a 1 a 2 would be twice as hard a 1/2 would be half is half as hard a 20 would be really long so on three we're going to talk about two stars I say on three everyone throw their card they were gonna pick up a card so I'm gonna get a half from him I'm gonna get a 5 here a 20 and a 13 I'm like when you don't understand jack about this story right clearly there's something missing so why do you think it's so easy why do you think it's so hard talk about it together to make sure we come to some common understanding until we throw something like maybe maybe it's

a five or an eight this is a Fibonacci ish sequence so we're gonna get to that understand and say alright this is a five all right now we're gonna have some understanding of how fast our team can move and where I have this what's called velocity how many story points per sprint to do we get done so in our planning session we're gonna as a team say we are committing to do the first three stories let's say so that means that's all we're going to do that entire sprints think about those three things so every single day we go into what's called a stand up and a stand up is five to ten minutes you answer three

questions what I do yesterday what am i doing today and is there anything blocking me these are run by what's called a scrum master now they are not the boss of the team they are a servant leader their job is to unstick people so if he says I'm really stuck on this I need an answer from security on a question that I gave them my job is to make sure he gets that answer other things that we'll uncover is maybe he's finished early and he's running long hey can you combine with him and go work with him and help bring that story because what we want to do as a team is get to everything that we committed to

finish the work that we set out to do we'll track that and then at the end we'll do a demo to our product owner who will either accept or reject the work and we'll do a retrospective so it's usually what went well what didn't go well and how do we improve and when I was a scrum master it was what stuck with what didn't suck and how can we suck less the next time now in this process security is usually not involved which is why I'm calling out these ceremonies because this is where you guys are all right I hope they do it right I what are they building over there and what you end up fighting for

is budget you come in at the end after your pen test or after your vulnerability assessment say here's 50 things I need you to fix them all shipping tomorrow or we're shipping next week can I just fix the important ones anyone heard that one before how about just the important ones you will never get all 50 maybe you get five maybe you get none maybe a ten but you're fighting for that budget with the team and what happens to the rest of it goes to the bottom of backlog never to be seen or heard from again right so it gets buried because we've surfaced it too late we found risk in the project too late for the business to

do anything about it alright now let's talk about DevOps this is my favorite quote this is the one that I leveraged from Nathan Harvey from chef this talks about the entire organization a high-velocity organization is not just about a team it's not just about dev and ops working together that's not what it is if I am shipping software twice a day let's take the case of my guy that's running support so here's my support guy over here I shipped a new feature yesterday because I shipped multiple times a day phone rings picks it up says hello can I help you yeah this feature I'm trying now doesn't work he's like what new feature how does the team

communicate to the business to say we release this new feature are the sales guys out in the field that are out there to communicate about it and say hey here's what's coming up here's what we're doing here's what we ship yesterday how do we do that as an organization how does the security team become aware of all the things that we're doing to make sure that they're involved in the beginning stages so when I talk about a DevOps team when I go to a customers usually this it's about release management glorified release engineers they write a bunch of scripting they automate stuff so that's DevOps for us okay that is not DevOps because remember that last picture where

I talked about everyone being in the same room it has to be this has to be the person that writes it is responsible for monitoring it and responsible for operating it if they don't then they won't ever change what they do if they write crappy software and they write crappy logging and they don't have any monitoring and they can't go and debug a problem when they get woke up at 2:00 in the morning instead of the ops guy maybe they're gonna think a little differently about their behavior when they're writing their software this is why DevOps is awesome it drives the right behaviors all right so we're security usually you guys know where it is yep yeah that's right there it's it's

most of the time that's where I find it it can't be security is whose job everyone's we got to do it all the time we've got to think about it we have to help people get this accomplished all right so here's my strategy again I talked to a lot of companies and I know and have seen things that have and have not worked so first of all we're gonna start with relationships and accountability most of the time when I talk to people their problem isn't their tools they have tools that can find stuff what they can't get is developers to actually do anything about it if you don't fix that problem you might as well stop we have a

customer that for five years five years scanned everything fixed nothing why are you spending this money it's a waste of time and a waste of energy if you can't get your developers to fix what you find you're just negligent training and remediation coaching so to the previous talk here talking about what do our developers know about security and it's usually dis and it's not that it's it's not their fault when they went to college so I went to college back in the late 80s and early 90s they didn't have that in college barely today can you find courses on it and barely ever are they required we are not teaching people how to write secure

software so guess who has to do it we have to do it our company has to bear that cost now you have this huge work force nobody knows jack so we have to teach him jack and this for us so oftentimes this is this is the way security does releases white-knuckle all the way through we gotta test everything I got to get it all done I got that know everything about everything we can't do that if we're in a world where if our company doesn't start releasing faster we're going to not be a company anymore especially in a financial industry there are a lot of upstarts that are coming in with Greenfield I got a new

package we're gonna build it all in right from the start I got CI CD pipelines that shoot me right out the door so I can do that stuff really fast guess what they start leapfrogging those companies that are doing waterfall so when I talk to companies they're a little bit of panicked I have to get the DevOps because I'm losing business someone's coming in and eating my lunch there's this new market segment we want to go address I can't do it for nine months how do I keep living how do I keep making money as a company okay relationships who knows their peer and development show hands excellent a couple you have to all right

actually let me ask this or any developers in the audience yeah okay do you know your security peer oh you're both all right that's easy you talk to yourself every day if it's those developers if it's those security people where are we getting if it's Jim or Bob or Jane or Susan over in security now I'm starting to get somewhere I have some kind of personal relationship do we understand how they're gold now in the in the world where we start we get to DevOps once we started to release from once a year where operations could say yeah I can I can watch the software as it goes and I can figure out how everyone said by the

time you kids say I'll be fine to them shipping twice a month where the thing is always on fire when they install it they have to go back and try to figure it out because the developers are gold to release fast that's all their gold to do get it out there and make it quick in the operations guys I want everything I want to look like a mirror those two things are in conflict they are not gold the same but we throw them on the same team and they start to figure it out right they start to work together such that both their goals are met they can't be in that conflict so you have to

understand how your developers are gold to be able to understand the things that they have to live with and the choices they have to make they're usually giving you that stiff arm not because they couldn't care about security but because they're not paid to care it's an important point here if I'm not paid if part of my bonus at the end of the year has nothing to do with application security when you call up you think I'm gonna answer when you send me that PDF or spreadsheet with all the vulnerabilities you want me to fix you think I'm gonna go do anything with it mm-hmm you got to understand what they struggle with so part of the reason for

walking through that agile methodology and talking to you about DevOps and how stuff is done you need to understand how the sausage is made I was in Germany yesterday which is a perfect place to talk about sausage you need to meet with them lunch go to some of their meetings you don't have to go to all of them but go once in a while show that you actually give a crap about how software is made and what their team actually does and you've got to build some sympathy for each other for the goals that you have because by vpa one of my first companies every time there would be some contention in the room we'd start

arguing and things would get a little heated so if I stop pull out a business card okay what does it say in that business card oh look it's all the same company guess what we're all on the same team here okay because we're gold differently does not mean at the end of the day that doesn't feed up to something that is a goal for the company to make money or whatever it may be there is some common goal that we're all working for all right so this is the big one this is the most important thing in any application security program I would argue in any kind of program anything Tim you want to get someone to do something so I will

disagree with my former speaker here about making them accountable you want them to want to be accountable you want them to take accountability voluntarily right I am responsible for the software that I write that doesn't just mean functional aspects that means non functional aspects I'm responsible for the outcome of its insecure it's on me I need to fix it usually it's a security team problem they have to go figure this problem out it's not my problem they're putting compensating controls in tuning glass and all the other crap which doesn't fix the heart of the problem that I keep making the problem worse I think the problem doesn't tend to get better that way so we need to have that accountability it

has to be shared so we have to find some alignment in our goals that means the seaso needs to work with the CIO and say you know what security is important for development so me as CIO will say I'm willing to be measured on it it guess what that means for all the people that work for they're gonna start to get measured on it we're gonna start to do reporting on it easy to say hey Bob why is Lisa's team doing a much better job on security than yours you better go look into that and it starts to filter down the organization it starts to create the right outcomes and the right behaviors has to be in their goals they

have to be part of what they get measured on at the end of the year you will not give me effort out of them they're gonna work towards their bonus they're gonna work towards their salary and again measured and reported agree on metrics or agreed on reporting transparently not some you know hidden document that only goes to the CIO or only goes to the development leaders it starts with the application team their management they're made all the way up to the CIO who reports then to the CEO and then the seaso and the CIO are sitting in the same room reporting to the CEO saying here's how we're doing on security as a team all right we have

this thing at Varick old called the state of software security we have a software as a service which means we have access to lots of data we take a look at that data and try to find correlations between what we do and the outcomes that we see so first and foremost is training as I said before developers were never and to do software security we have to teach them one of the ways that we can teach them is on-demand learning you learning right we have measured companies that have eLearning versus companies that don't need six times difference that is huge six times difference in how fast they're able to remediate their stuff if they have access to this now we can do this

also through instructor-led so my favorite saying is I'm gonna fill a room with pizzas and then the developers come because developers love free food and I'm gonna get to talk to him over there lunch for an hour so what am I going to talk to them about well what if we used the scanning tools that we have to create some kind of heat map and some kind of thing that says hey you know what sequel injection again is our highest problem people keep creating them so when I have the pizzas and the developers in the same room I'm gonna talk to them just about sequel injection not about you know how to do secure coding it's like who cares that stuff is

important but not critical let's talk about something that they're doing today that's hurting the business now as a byproduct of that I can take that and now measure the effectiveness because if I'm if I'm doing my measurements regularly of my application security I should see sequel injection start to go down because I've trained them how to do it either that or I find out that I've not a very good trainer I need to find some other way but I can measure the effectiveness of that now if you do that if something that matters to them you will get their attention and if you feed them they will come alright so for all of you in the room who's read the

Phoenix project all right a couple of hands that's the first thing if you're gonna start learning about DevOps read that it's an easy read it's a novel it's a story that kind of goes through this fictitious company and the problems that they had you can read it in a weekend if you wanted to the DevOps handbook actually has security that's pretty cool first time you see that this has chapter on chapter about different aspects case studies on what the effectiveness of DevOps can be inside of your environment it is a good read you this is not something you have to read cover to cover I have you don't have to read it from front to back if

you jump around inside the chapter so you guys could start with the Security chapter if you wanted to all right so next we have to help them fix what they find so if I say you have an XSS and they said huh the other sequel injection what the hell is that they don't understand because they weren't trained so oftentimes this is portrayed as we think they're stupid look they're stupid they just haven't been taught about this stuff there's lots of things that we don't know about their world as well I can fortunately get to bridge both worlds but we need to be there to help them fix it I'm going to talk to you a

little bit about how we can scale that kind of a program but we need to provide them something whether it's us ourselves some outsource agency that provides that kind of service but what we don't want is them wrapping around and doing Google searches and maybe finding something that's good maybe not finding something that's good and what we've measured that we have a consulting service that does this it's called a readout call where we walk them through this a this person who's called an application security consultant they are experts in security and experts in the language that that application was written in they speak developer so they can walk through those vulnerabilities help them understand why

that is actually a thing because usually what you get back is that it's a false positive that can never happen so we help they explain to them why that is a thing and why and how you would fix it so we see a two and a half times improvement and how fast they're able to fix these things all right so how do we scale typically 100 to one are worse for developers to security people or at least application security people or if you got a security person it's like this much of their job so this comes up a lot in InfoSec conferences these days and this is about extending your reach about creating a force multiplier right creating the eyes

and ears inside of those teams so as we talk about having let's see it's some of our largest customers five eight thousand applications oh my god that's mind-bending in teams hundreds of application teams how do you possibly go to all of those Sarah boys how do you go to all those grooming sessions to make sure that they're talking about secured you cannot it is physically impossible we just talked about the gap in cybersecurity Todd you can't even hire enough people if you wanted to you have to build it into the people that are already on the team so what we have done and what we've seen other companies do is we get volunteers from each team voluntold is okay well we

get volunteers at least one you'd like at least one two would be great because I don't like bus factors of one as an engineering former engineering manager and they get some specialized training now you sell this to people by saying hey look we don't have to go hire people we can't hire we're using resources we already have and to sell it to the developers you say you are now more important not only to this company but in your next job if you can put down cybersecurity expert or application security specialist or something like that you are now more employable win-win so we're gonna give them some specialized training and talk to them about CIA I talked to about threat

modeling we're gonna give them some grooming guidelines now these usually appear in the form of some simple checklist so we're gonna start simple and build on that so the longer they're in the program the more training they get the more they understand so we're gonna give them some guidelines for instance if this touches crypto or this touches authentication or authorization pen test write some simple stuff other things will use the automated tools because they can find it better faster cheaper earlier than we ever could we're going to teach them how to do secure code reviews not for everything but some certain easy things for them to go and spot so that way when they're in

their grooming sessions they say hey you know what I see you're writing sequel in that I want to do make sure that we do a secure coding assessment of that before you were allowed to merge that so it becomes one of the things and their definition of done so actually that's an important thing part of what happens in an agile team which again migrates the DevOps because it we run it on agile is the concept of definition of done so if I have to build something I have a story says build this clicker there's a checklist that I have to go through to say did I do all of these things that I test it for functionality

did I add new unit tests did I do security testing on it that becomes part of what they are expected to do before they can say it's finished and ready to go to a customer so we add that in there so we can add that in our grooming sessions through these security people basic security controls and guess what we get to do some fun stuff with them so we'll get together on a weekday at lunch one brings their laptops let's do a CTF how fun is that developers actually like doing that stuff and understanding the other side so if I'm going to be the guy that's attacking maybe I think differently about defending in critical

part here is they need to know their limits they are not infinitely powerful I have a longer form talk on on building security champions programs if you're interested to talk to me but there's a there's a point at which they say ah I'm over my skis please security expert come in and helped me all right holistically now we have to fill this gap with tools if we are running at a pace of multiple releases a day you need something to catch the easy stuff I don't want to pay a pen tester to go find my sequel injection oh my god why the hell would I do that those people are far too valuable and can find really cool things and

hard-to-find things I do not want them finding my sequel injections would you agree is it do you enjoy that yeah but that's the easy part of the job right so what we're gonna do is we're gonna start doing static analysis the second they start writing code day one in their IDE watching the mess a code and pointing out the things that they're doing wrong we're also gonna start looking at open source risk so we talked a lot about that today I want to know what they have in their software I need to create a software Bill of Materials before I go to production because and we'll go back to the my security champion the other thing

that they are responsible for is that part of my incident response team so CVE thus and such gets created we get together and say are we vulnerable I don't have to do that work security champions please report back to me by the end of today on if you use this library and if you do what you're gonna do about it and when you gonna have it patched darn they're part of their job is to be part of that instrument or sports team you can't do it as one person that I would argue that if Equifax had this they would have been patching long before they were exploit if you don't know what you're shipping you are screwed because like I said it's

easy to understand if I have struts hard to understand if I've got some particular version of Apache Commons which ends up if you look at it in over 80,000 libraries inside of the frameworks it gets pulled in everywhere which means it's in millions of applications you've all probably but how would your not so I'm looking for that from the very early stages of coding now as soon as I have something that can run again to save on costs and and to make myself run faster I'm going to use dynamic analysis which is chief automated pen test it can't do what a pen tester can do it can't but that's okay you can find things like hey it

misconfigured data you didn't do this or you're leaking that information that's the super easy to find you can find it quickly and cheaply if I'm gonna pay a pen tester I want them to find the hard stuff so let's find that with dynamic analysis and by the way we go all the way into production with dynamic analysis why because typically especially if we're running old-school on our own hardware we've got a quality lab and we have a staging lab and then we have our production environment are those maintained by the same people do I get patch management on my QA systems I doubt it do i configure everything the same well maybe at first but when a problem

happens in production they op the ops guy goes in he makes a configuration change do you think it comes back and make sure that QA has changed and stagings changed doubt it who has time its until you get to that infrastructure is code where if you touch it you got to change it in the code you cannot change it on the machine because it will just be overwritten that's the place that we're going to but until we get there we need to start looking for this stuff because the configuration just diverges and then we have this thing called rasp anyone heard of rasp right so this is a smart agent that sits inside of the the running

application and watches things happen so it can find easy stuff like sequel injection or cross-site scripting now let's say that my application security program is just starting out I could protect myself while I go back and fix them I don't want to leave them that way forever but it's a mitigating control if something can protect me a little bit and then if I miss something maybe it's the air bag that saves the application okay security team big important job was starting all the way over here modeling security grooming talking about secure design helping those teams in the very early stages before they make the mistake because after they make it after I've knit the whole damn sweater and I

got to come back up to here because I dropped a stitch I got unknit the whole thing and then reknit it that is incredibly expensive so our CTO his dad used to work for GE jet engines build jet engines right when do you think the right time is to figure out that a fan blade is is gonna break is it when it's in the test stand and I rev it up and the thing comes apart and shatters and loses millions of dollars well what if I could have X rated when it was coming in the door and I make sure that that faulty part never makes it in DevOps is all about testing as early as possible

to find these mistakes and fail fast so that way that stuff doesn't happen or at least it's very unlikely and if it does that I'm going to add a new test and that new test will hopefully save me the next time so we're there to help them big important part in the middle here and it's big on purpose we've got to help them fix what they find if we have an application security program that only does scanning and identifies vulnerabilities and that's what our job is to is to just get the list who cares what are we there for we've got to reduce the risk at the end we could do our red teaming and our pen

testing that stuff is all still valuable and can lead us feedback loops back to the development teams to make them better so if I have red blue team I can say here's threats that we see or here's threats that we've figured out we can add that nor threat models we can make our development teams better over time all right so right size testing anyone heard the term CI CD pipeline the straight man alright so this is an actual pipeline for a micro service from a dev ops team that I built and my instructions were that for them were when this gets checked in I want this to be a completely lights-out deploy it was a greenfield project we're starting from

scratch let's build it right the first time I want to build tests that I trust and those will take me all the way to production so here's my backlog like we talked about my developer comes in now we're gonna do our standard building tests which they do now because they were agile analysis take the test before you take the test it's no different than taking your cissp or your CSS LP right you take all the practice tests you can because by the time you get into the test you're gonna you're gonna blow right through it right they need to know that when they scan something it says it's clean then it's gonna be clean later because we're in a retest right

that's what our assurance testing is for but let's take care of the problems here and by the way when they start doing this they're gonna start to educate themselves so eventually I've proven this because I have developers are boring for me into 10 years it comes out of here secure isn't that the best world that we could be in where they actually write it securely the first time I don't have to find it I don't have to track it and I have to chase them and beg them for for a budget they just don't do it again that's before check it now quick pause good manual testing pen testing is important I believe in it but I only

believe in it for certain categories of stuff only in cash but in manpower and in calendar time DevOps if you're leasing twice a day and you start measuring testing calendars those things don't work together so we're gonna pull those things aside before we do the final check-in so this is release candidate when he gets in there I'm done with it if you'd identify your grooming session that has to pentesting fine put in an environment say hey pen tester go at it I don't care how long it takes it will wait I'll do other things now he goes and does the pen test come back to me or similar results we worked through them I fixed them we're both satisfied now I

check-in don't put people in the rest of this process what I'm about to show you no people should be should play here all right so now on every check-in because this is very small low surface area micro-service my CI server is going to come up we're gonna build and we're gonna run some tests though this is this typical static analysis stuff of lint plus code complex the plus code coverage unit tests I'm gonna introduce static analysis in here because that's my insurance test did they follow the procedure or not and I get my result of this is the CI portion the continuous integration portion of this pipeline it's like sniff test stuff I could do

without the code actually running it doesn't even have to be functionally complete I can run some tests to say whether you're doing well or not if they're not and I choose to I can break the build and synchronize the backlog another key step here and you give them spreadsheets and you give them PDFs and you say go to this other management system that has all security vulnerabilities your stuff will never make it to the top it has to appear as a bug in their backlog for them to do work when I was in charge of a scrum team and someone says hey are you doing this and like well tell me what the number is in

JIRA if it's not in JIRA it's not getting done I promise you that if I don't have a work item for it it's not going to happen all right let's presume developers did a great job they followed all the right process now we're going to go into operational environments we're going to with our multi-tier @qa the staging may be UAT and then our production environment I'm gonna run more testing on it so I'll run some automated regression testing I'm also gonna run dynamic analysis against it in certain places this works very well so this happens to be a single page application written in angular very low surface area runs very quickly in other places that

might not be the case maybe you need to do some more pointed scripting against your dynamic analysis to make sure that that works again I'm going to get that pass/fail we move on to the next stage or we break and we move all the way back so this is the CVE continuous deployment or continuous delivery portion of this sometimes this can go all the way through to production in this case it does I trust these tests and it just goes there are other places where you have your cab or you have other places that you need to go for approvals and there's button pushing and stuff that can still happen but automate as much as possible all right so

wrapping up in conclusion you got to learn this stuff if your company's going to DevOps and you don't understand it no one's going to talk to you you have to be able to speak developer okay get into their world and be able to go to them and talk to them about their ceremonies talk to them about creating a security champions program build the relationships that you need and you need to work at the very highest levels in the very lowest levels you can't cold call in to an individual contributor so I need you to fix this you need the highest levels of your company to agree that this is a problem that we are going

to address is a top concern for the company these are boardroom discussions today you don't think Equifax we lost the CEO the CIO and the C so all within a week you don't think the board's talking about these you don't think the C Suites talking about this get it on their roadmap it's a great opportunity and get it part of the yearly goals you have to train them they're not trained but you can't throw them out because who you gonna hire even the newest college graduates might have this much I was talking to someone yesterday she just graduated from college I said did you get any security training she goes yeah I get some Network training

network security it's like well it doesn't help me for a sec they're not trained on it you'd have to do it and you have to adjust to the speed of DevOps so you can still do your pen testing I'm not here to say you can't I'm here to say it is important let's do it at the right time and for the right things let's flush out all the easy crap that we can find through other means and automated tools that are very low cost and very quick and flush those out first so that way I'm not paying this guy crap a little money to go find a sequel injection it's not very interesting to him now he makes money on

it but I'm sure you'd rather go crack some hard problems say ah you guys think you got this done let me figure this out all right that is it I'm happy to take questions yes sir so I don't think this looks any different for any kind of software whether it's shrink-wrapped or not right you still want like I said this is a terminology that we use in the industry is called shifting left I'm sure you guys have heard it right it's shifting it's only in the last year or two that these tools have become high enough speed and high enough quality in other words false positive false negative rates for us to be able to

stick it into the development team give it to them raw and then they can actually do something with it so this is about finding those things early so there are other things that you would including here depending on your release model and depending on where you were going we run a SAS business this is how we do it any other questions yes sir

all right so offshore development teams it's critical that you get this as part of the contract so if I'm gonna hire this side of the room to go off and build me something I need to provide clarity on what exactly they're going to give me it's not some functional requirements it has to combine the non-functional and by the way if you already have a contract you might be able to squeeze it in as a quality issue because security equals quality a bug is a bug so you need to find a way to enable them now depending on where they are or how they work you're going to have a hard time with training because in some states you are not allowed to

train them by law you cannot train them so you need their employers to do the training you need to build that another contract so I expect these outcomes they'll start to figure out that if they start to train them maybe they're gonna do it faster so if this is all about understanding what your requirements are providing that clarity in the contract the contract basis and then holding them accountable for it so it's no different but there are usually some restrictions yes yes yes so we're talking about in this case we're working in an IDE in this case this plugs right into Jenkins backlog there are first-class plugins for lots of different systems so I though once I specify the ones that that

we use but build systems ideas bug trackers there are integrations usually for most of the tools that you can go out there and look for I would make sure that they have those as part of the cost of doing business with you part of the critical criteria that you need but yes you and a lot of them are scriptable too so they have API so if you have some unique system that you built you can get an integration as well no yes

very good question it's usually part of an item that you talk about so you have that security champion in the room when we bring up hey I want to add this checkbox or this workflow even though the application we're gonna say alright well let's talk about threats here what are we worried about how we're gonna secure it what are the security things that I need to make sure a gun is part of this task now again everything that the team does has to be work so let's say we're going to do its threat modeling exercise that actually becomes a story in the backlog that the team takes in because it takes real time and it becomes part of their sprint

deliverable this month this week we are going to create a threat model or we're gonna review the threat model it's actual work it gets costed out like everything else and gets done like everything else nothing is free yes and even better to use actually writing the software so let's say I need cleansers who would need those so what if the security team actually put on the shoes of a developer and wrote some libraries that will go through and say this is the standardization function here's the output encoding function if you guys use this we will support it and we are responsible for the security of it so we can cut standardized in our companies how that's done and guess what you get

to start learning that development isn't really as easy as you think it might be how we doing for time sorry okay anybody else I'll be here for a little while not much longer so if you want to talk to me I'll be outside thank you very much for your time [Applause]