DevOps: Security's Big Opportunity

all right everybody you ever seen it down picture Muslim because bizarre I'm here to introduce modern exotoxins we have chef Donna can we talk about dead off material they've opportunity as sugar plum honey he is the director of poker engagement at a secure code analysis and probably more like that company with over ten years o trip epic like to share experience as those a developerworks and a moment leader get a lot of really great backgrounds committee guy a third time any kind of not always something interesting to get rid of any hope that you pay attention so I'm going to hand it over to Pete now we did not on there thanks you're on all right so it is a brand new top

solution all have the time set up yet on this my time all right I love it about me while the been doing this 25 years to software developer so I come to information security advocating from a developer practitioner point of view but obviously working at nothing like Darko a ton of a spec experience so if n is very code almost 11 years now working the agile working waterfall working DevOps I've moved my company America from waterfall through anvil to secure DevOps from a monolithic applications one micro service based application excuse me and I've been sold with so the biggest company the world about pulling on a good app set program sometimes about they are DevOps and agile practices

themselves and if you everyone get my attention vitamins and whiskey that's their job all right so it could set the stage in addition we use that one DHS a 90% all the attacks that we've seen the breaches have something to do with the application layer as not necessarily applications that we build and sometimes data issues we Bible we should think about the security of all of the applications that were running an article Erica Scott also the biggest cognise people customers it were prospects actually who's a vertical customer here thank you everyone else appalling ah the big thing I did got to see is they say hey we're just going to do our Tier one applications our most important ones

okay great and now that's kind of like the national line of security at one layer with that term so yeah World War two French Dumba's well the Germans out but they didn't go all the way so they just program along really this is the way aspect is typically about well just to be important watch so you see things like Target which I'm not going to do at target.com or anything like that was some HVAC contractors building site even target and that's how they got into their courts or some charity event JP Morgan Chase middle for a running event in our open source community - so I'm not saving open sources worst of the first part it's about the same it's just we

don't think about it very much alright so who has enough piece of speech program today alright one so just you're outside program right now right or this right in there's a couple of different outcomes so that each you saw previously a lot of times I see this they come to you at the end or their security attestation they say here's my release candidate and you go yeah on a bunch of stuff you're going to these go fix and then they start to phone calls up the chain all the way through some of the powerful not to call you and say hey we're releasing this anyway I'm sorry I didn't see that before or if you've got a real Gator this is

really cool if you can get this most companies can't this is what happens ramming speed or ludicrous speed they crash right into the gate they don't think about they hope that they can get through it by the time they're changing around the way applications are developed and this is why I'm here giving this talk at to a bunch of security practitioners meaning to think ready about how we approach our application teams and our development teams how they build software so that it can be secure if you look at the release timelines that we're seeing and these are ballpark averages typical what we will see waterfall you're looking at one to four releases a year and he's doing

waterfall here in their shop or knows as a different one one couple rocking back and forth uh big scenes I was on teams of 150 people sometimes more than on earth before building projects right there's huge efforts move to add and now we're shrinking that fine line and we're tripping the team size and this is really changing the way they adjust for these teams and awaiting pregnant a couple releases a month now we're talking about in in sizes 6 to 12 by the way you're going to see a Dakota lodging down so these aren't these are crude related things usually using agile methodology to run your DevOps operations alright to deadlock now we're talking about hundreds of releases a

year or more sometimes ten fifteen hundred times a day any not workshop you're working in this ABC in size again decrees are typically running that agile methodology on top of that all right so how do we get here so that in a day when we have waterfall right we have these walls between everybody and I'm going to skip security over here quality because I think of security as equal to quality we would hand off over the wall right development done passed on quality all genomic that's one operations etc and the problem with that is once we've written our specifications and have a plan for our product owners we lose the fidelity of bother down a lot of plain telephone

similarly our developers are building these applications and why they help them that way and the decisions they made and how they expect it to run is also losses you can follow down towards our production environments similarly upstream feedback from our operations people they know how here's an Operations sorry they know Howard run they have scripts to go it's the thing is that we didn't do right oh I gotta knock this process over every couple of days because it runs out of memory otherwise all those stupid things but those don't make it far enough upstream for us to do eating goodbyes so hence we move to agile let's not done some of these well typically the first

wall to break down you'll stick your product owner your developers and your quality people on the same team and that's awesome so now we've got some really good fidelity around our business intent and application knowledge but a little while still there you still think it's somebody else's problem to operate our software so that app screen information is still getting lost who the deadlines are in deadlocks we have really really good continuity across the spectrum the operations people are in the room talking to the developers about how the software actually works listening to the business problem is being solved so everyone has good information about how it's supposed to work at what are you supposed to do

or would technology point of view and isn't London driven by the timelines if you're talking about waterfall if I have to do something once a year or twice a year it doesn't matter how much it hurts I'll just do it that way every time there's no hanging on a benefit to me to go automate that process but when we get into agile we start talking about something that was months to a year and we're now moving that down to you know a month or two weeks I need repeatable processes that your run when I need to matter whether somebody sitting in a particular sleep with the engine keyboard or not once we get to DevOps and we're trying to

release investments on you know I make everything absolutely everything all right loose amilia with added a lot much you require a little action quick primer over here we have a product on the guy with a light bulb or gala the lightbulb is under a 3-wood stall the backlog this that body of a prioritized order the list of all the stuff that they want us to do okey okay and what we're going to do with that is we're going to take that into a plan session easily a routing section first we're going to look at this time say you understand what being asked for really important part of this process so let's say my scrum team is

down here in this first row where the product owners going to describe the feature at the 8th is what I need and we're into something called planing focus this is a docking mission sequence of numbers from 0 it's already gotten to Wanda at a 1 to 2 everybody 13 etc it's relative sizing so 2 places 31 excetera so we're going to we're going to sit on three show me your car actually this will surprise you can do it on your phone once more fumble cars so we're going to show the numbers in I get a 1 for this guy over here I had a 13 year at 0 there 20 over there my Bo we really

don't understand this Oh point of that exercise to get it down to the point where everybody else can because you might not be just one person's job to go do this we're not signing at work right now we're trying to understand to work once we do that in our plan says I'm going to come in as a team so what we're going to get done or whatever this is printed so spring is a time box duration into the four weeks to the six weeks and we're going to talk every day and five within 15 minutes what I do yesterday we're going to do today what am I being blocked by so we can identify our teams

to help each other we're going to track that sometimes on the board usually electronically I'm going to at the end say an agent again everything done rescue assault retrospective and I'm going to say what well what did you go well I want you to change to be better at continuous improvement process that's built into that not typically is my security I hear something open to it already and what ends up happening especially in those previous cases they're fighting for budget right they come to us or at least get say here's other things you need to go do well and just do the most important ones which ones are those yeah that's like a mule is the barometer you

tomorrow well we just did the pop pop so you're fighting for that budget at the end one is saluting our worst possible scenario they're ready to go out the door all right so let's the personnel you know what DevOps actually is for my perspective is both maybe Harvey a chef I love this quote it is an organizational life change it's how you run a high velocity organization I need to change everything if I'm releasing features daily if you might support people need to get trained differently if you might documentation people need to work at a different pace or work differently my product owner certainly needs to be filling a few faster than these needs to because about blocking up

this functionality you have to change the way you think about this end by the way we can't lighten up the releases from the security plan view everything can't be contested that little doesn't exist anymore I'm sorry sorry to say that it's still valuable I'll show you where I think it should go but we can't do that for everything anymore so is it a little bit off cycle and everything from when I found the idea of planning all the way up and monitoring and production now I fear once people talk about the DevOps team or the DevOps guy or gal this is not what I think of as DevOps right so here's what they operations people do

people have moved the software through production that's not really getting after the whole problem it has to be this you have to have a team that is responsible from cradle to grave from the stuff that they built because I'll tell you what a developer it's differently about what they wrote Monday have to monitor and operate when they have to go write those scripts that say well geez gonna knock over the servers again because I'm running out of memory well maybe I should calm down a little bit different maybe I should go fix that now right if I'm wearing the pager I'm the one getting woken up I'm going to think differently about maybe that I do similarly security out

from our waterfall days and from the way these puppies typically bolt security on a hand is where we end up sitting horrible place all the damages have been done we're only just pointing it out to them okay we're turning ignorant into negligence this is what we need you have to be there when they're planning you have to do things with them of them understanding the threat actors help them understand the design decisions that they're making so here's my strategy I there were six items I kind of paired them together because I think they logically go together first is relationships of accountability seven training and remediation coaching and thirdly security champions and right-sized tested so startled

relationships whoo all right so who's AHA security so who here Isabella side of the fence one quality nobody okay noticeably absent we are dinner operation team all right how about people that are responsible for application security so with those teams awesome so we end up appearing security or in development Missa pogie edges same time all right now okay again now nothing up put your hand down if you don't know someone you're here in the other part of the organization don't really visit the security video organization watch end of them we have to work together now in my model right those people are all on the same team that's not always possible that's not always practical sometime

security has to be on the outside that destroyed were organized and we're just have to deal with it okay fine but that doesn't mean I can't have a relationship with that person I have to know who they are they have to be a real person with real problems do you understand how the gold this cut is going to come up again what they both to do because you look at your typical developer it's move software fast that's it operations people are keep production smooth right experienced people are secure the stuff whether that be your your boundaries for your applications those are all in conflict we have to share the same goals they don't have the

same goals of you that's something interesting to know because when you go to them to say easy to fix these things like know what though and it should fit software that's all that they're about so understanding your struggle so we're going to go in a little bit more depth on this as well and do you meet with them if you don't you should lunch once is okay drinks whiskey whiskey buddy down there yes another one excellent they gone for a whiskey it to know the things that they're struggling with have any ace to the role it's critical to us working together they usually get so one of my first companies are becoming get into this conflict in

the meeting the be teams they are the one pull up your business cards seem is they're all the same we're all on the same team so it's critical that we think that I remember them you need to be a little bit more sympathetic to stop the base for Google yes the sequel introduction you find cortically important you want to get that fixed by game finally I'm going to assume so excellent point by the comment water table if I put a ticketed and I'm not seeing working on it is frustrating but the question you need to ask is why it's not because they don't care about you or what your job is is because they're not hold for that Oh

and now might be too right but we button so I will counter that but I can't sit on the other side and just be for budget about it and you don't have conversation so X avenues relationship is critical in people it becomes alright accountability I have never seen an absent program good after remediation if there is a no accountability for the people at both the datum software in the first place yes if I'm buying software from vendors and I don't have it written in the contracted by the way you gotta fix it security stuff if you put in air or I'll paint ownership of it an at4 it what is their incentive they're just going to

kill you for that is extra work usually you can wrap that up into the quality love the AVP bus securities quality and you might be able to get away with them but think about that your contracts security and development need to share the goal of securing those software that we make the company to get a lot about the security team if by the way the screen it makes the problem the developers did you can also you can do as a developer in application spirit we're the ones responsible for it we should take accountability for it and you see that more and more in this DevOps model as we as we look at becoming more multi-discipline engineers

so having to worry about quality and operations and by the way I've containerized so I need to think about IPM systems engineering that I never had to think about before all of those assets need to feed into that team as requirements so best way to do this is to change the way they paid they have goals everybody typically does in an organization the developers having security goals they don't I might want to start there because you're going to maybe a real bad if those measurements are going to hurt there a end of the year you got a measure and report it back look at where we are and look when you onboard a baseline your application

it's already you're going to find a whole bunch of checkers of that up there it surprised but that is you're not going to fix you overnight we get there overnight and you're not going to fix it over sometimes the software is 5 10 15 20 years old it has security vulnerabilities in them okay fine yes they're also in production traveler can we mitigate against some of that and what are we going to get after we can't shut down the whole company shift the security and human spirit rumors so be practical about it but understand we are going to be honest try to be better on all right huge one here so this is from Erica a

software security report we measure because we've scanned over 2 trillion lines of code across thousands of customers with tens of thousands of applications the kind of things that help them work better and faster training leads the way it's times difference if you train that so the developers are putting in a fewer developer and you can hand up if you were trying to security it is part of your career all right apology one cute couple if this is 25 years ago didn't have application security back there that most people are coming out of college with little to no knowledge of application security alert thank the people who journey of your workforce has never been trained we have to take that

on as a cost of doing business we have to teach them because I would rather them write secure code then I'd have to chase them with my tickets were my ticket was not arias and chase them it's around the stuff that they screwed up the much better herds come out of the finger secure we can help them do this we have to be present for them and we have the trillions else this is a measure of people have used a learning versus not so that on-demand training is great think about that think about doing it in instructor-led way so what I love here is take a look your security policy take a look at what your application

are actually doing the production and say what sequel checks you pick supplies our biggest problem so I went to build this room the stage for pizzas now you going to do this in the right order if you try to filled with developers and then pizzas it doesn't work if you put the piece into the developer whole top so if your comments are moving in there before they get there and we'll all gratification to the room and then guess what you can do you stand up here for now while they're either eating themselves it's factual about age specification as earning your company today it isn't be secure Java coding course which has its merits but it's not

the best use of their time especially if you're trying to get them to a point where they stop doing the things that are heard of you the most so measure those things bills and training around it get into that it's the low cost way of getting a high impact on the food's what's going to be books actually about whose record good excellent get these books this one first and then a lot handbook I incredibly valuable of helping you understand the problems that these teams are facing and there's a national security reason even in a julian reliability is security in technology organizations you'll see some competition and this is all about anyone under certain ship level this is the

whole idea in the industry right now the same way we did shift less of quality so that we wouldn't when we moved as when we said okay let's make quality part of the thing that developers have to worry about so what we can think about it earlier by doing things like test urban development quality is front and center it's like a better at doing quality one do the same thing with security when you've done the tools to be able to measure that early and get better at it over time next again they're not experts to your point exactly about they don't understand what we're the hell are we we should be available or we should

provide services to them and help them extra things that they're finding they don't know what a cross-site by error it is who's there to help them we really wanted to help searching on Google or photos information we rather to buy them something me we can write in security code that wraps some of this stuff does the right input validation says eight and this one of eight wrote approach I am going to provide you some libraries it seems none of this is up for free if it's broken is my problem if you use it I will support it if you don't then you have to 50/50 shot okay so this is a place where we can pitch in

and help them get more secure ask their in if you don't understand it need a place to ask and it can't be a gatina couple leaves so some companies provide services around us the provider and remediation guidance on how to think about these okay next one security champions this is become very problem I talked about at Davos conferences and talk a lot people set conferences people are using this term very broadly in the industry this is a way so who has security organization more than twenty feet okay allow more than 50 people look who's got Red Team Blue team but a lot of these time to go to these customers relation prospects and by the way one of

them was a privately held company that if they were on the S&P 500 would be like one of the top ten biggest companies almost like one guy one I might not defecation spirited multi-billion dollar company whoa that we need a witty and by the way you need to have help appear more security people how many developers we trying to support the best hundred thousand typically you can't possibly be in all the meetings all of the places so what is done and this different talk on this button to to briefly give you this they becoming eyes you experience on every single development there should be at least one of these people with this information super girls on England's exams they need

to be there in the meetings part of the development team I pick up a security expert but they're more expert than the regular game you're going to give them additional training we're going to talk to them about the basics parity where I talk about CIA we're going to talk about threat modeling we're going to talk about and give them some instead of grooming guidelines it looks like that smells like this then Security right if it's a form field and you need to worry about these things if it's a HTTP header if you're brought up worried about how to be able to take those turtles in the acceptance criteria inside of their stories so that way they can make sure

that security take a character talk about the verge didn't teach them how to do secure code reviews now I'm not going to be able to catch everything but there's some small set and you can build this over time about them understand what to look for and what to do about okay this is treating inside eating armor also the red light goes right if it touches my crypto touching my offend and the toxic touches on see any of those things security and probably pen test okay so some of these things absolutely positively pen test period and they need to know that and you can take that in that PT does they need to think about how they're going to bundle

that software get into the puzzle now we also do something at their code CCI exercises with the developers will show up in the bag everyone's going to laptops after dark sighs bail will be hackers it's super fun for developers to go and do this and see the other side and it helps a bit more offensively about how they code understand that they have limits at $80 animals when you get beyond me me I go to one of my security guys a moment of this but it doesn't follow that I'm talking because we're going to do something different with this so this is capture eighty to ninety percent of the stuff that happens every day inside of your software

organization if you train them right so here's a key strategy around doing this Whiteside secured right like I said value dependency training all the time we are constantly training our developers and you should too I want to start doing some of the tools so these tools can find security violations easily quickly and cheaply a lot more cheaply than the calendar and money China would take me for - go get a pen doesn't go do something standard analysis third party open-source risk I want to understand that as soon as possible the data start writing code I want to start sensitive with developers country these tests come back and if I'm that dirty a lot of private network so

we're going to go start annex means ask me questions about hot Rebecca dynamic analysis the second area couplets up in line I just want to look at it from multiple angles like I said not to say the pen testing isn't useful I might do some anyone heard of Rath more time application self protection so this is a up-and-coming technology but here's the security huge role big important job helping in the very beginning those worth here we're not all the way down there and testing the staging right doing threat modeling is pure design or Jews we're here in the middle biggest section hoping that this will define what's the point of a security program for not fixing stuff right if we're not

reducing the risk to our open and at the end mainly fomentation testing outright routine stuff that's all valuables it takes that little about helping monitoring and any products alone alright so here is an example I want anyone not heard of cix movie curvature we're not any like rid of warnings so here's our backlog and rest from pain we think I've been developed now the way I think about this I break this about and attach two pieces of static analysis and you go to San Antonio chicken is a really important point people the stuff that they're doing in that last diagram the day they start coding what I would call educational skin educational analysis it's a constant

quick feedback and then data say from a Dennis they can be devastating to the next night maybe I start thinking about it and I don't make that mistake anymore where's does something that's father done in line before attacking my logical application another look good not just encode that I just wrote again for checking this is part of in what Adam would call a definition of done if you do your static analysis Macon documented right because if you find something later on in your assurance can't and it breaks the bill well who is the one that didn't follow the process now again that's my point about annual stuff if you have to do video accessing you have

to do Co groupings we have to do you eighty anything you have to do that is we annual in nature requires a person a four-decade so maybe I took it into a branch I deploy this up up tell my fantastical add I don't care how many things come back to you when you're done once we agree that it's fixing ready then I'll check it and now we're going to the pipeline so this is all the network so whenever you check in in a highly functioning operation where all the tests you trust etc will fit into this CITV pipeline and CI server my continuous integration server is going to come up it's going to build it it's

going to run unit tests static analysis I'm going to get a result of those this is the continuous integration portion of this pipeline I don't think you go to a spin test this path all but easy typing and I think you against me if it doesn't a bill the bill synchronize the backlog hey guys it's all ready we're going to stuff to giggle fix not again uncle and non-functional side by side equal partners it bums if it doesn't ask them to start moving into operating environments so I'll go to QA is aging and production I'm going to run more security against it in more testing so I brought you some fresh intestines dynamic analysis against it again I get

results Kassadin loses the next day it doesn't break the build this is the continuous deployment or continue with delivery portion of this pipeline this doesn't have to all be automated but if you keep one on at least hundreds of times a year tens of times a day and activity you have to trust all of that ethic go right and that when those results come back it good you know say well let's pull it over so that doesn't right understaffing you need identify those things early it makes if they were given for socialism so what all right so some conclusions down you have to learn this stuff we'll get those books read them talking ability to attend some of their

ceremonies so it you know a July this one called chickens and eggs but things you know wanted to do the work which it ought to spin and quietly and observe it should allow chickens to come into the room and watch these pheromones understand how to build software you should understand how the sausage is made you gotta build relationships and you have to demand accountability and it can't be the lowest level security out of the asset that is doing them it has to happen at the high levels of C so and a VP of development have to agree that this is important to the company if it gets more visibility all the better that will drive all the right behaviors down

a lot won't end up a ticket to sit there silently with no comment on them on my work on care yeah you have to build it into your culture that are what you do is if your work Traynham they don't know they're cut from Nathan when it comes to security typically now that doesn't mean that they don't air just that they don't know if they're all smart and in all the worthies thanks open stuff you have to adjust to the speed that that also happens right yeah think about did that really require pendants like you have to pull it over for a two week or longer review or going to use the tools that I

bought it built so when you're looking attenders and you're looking at companies with bringing in too late really should look at most positive false negative rate is are the results all you're not community say when this ass is hot good or are you always going to be skeptical are you always going to happen touch with anytime you put people in the mix you slow everything that when you breed that person that so cause developers push with the ax different ways guys we're all set right so you have to fit into their working style you have to work over there in mind I will take some questions if it how'd you rate this music sovereign outside

Michael finger

so it is for specific for software that you sell so it's operated as a service or a software that you should preppin and the customers okay but so you need to build it telemetry not my hockey so I'm recruited you eat your hands on immediately right so if you think about looking as a small person comedians right I have all this stuff you come tuned up all of my hoppers or my blogs and other things that they come in here and allow me know whether it's offertory well or not you need to fill that same pen environment I would suggest dogfooding it or one you cannot use II don't you how it actually works so you can use it

as you can start thinking well what's our material want to build in it's not only when a customer calls you have a playbook you can give them or you can say a rocking these lines that would tell me exactly what I need to know so it's a little harder but it requires you to understand how your software works and if you're doing it just in the quality environment I would already that's not enough you really need to experience the pain of the users

hi

is a different organization so I think I understand much the way you a different organization does it but because of running a customer environment yeah I do it from we should fight to pick that one offline there'll be a longer discussion of panel man yes sir what resources would you recommend for helping to get a team they wanted to personal all the theme of the speed with this deadline so ask the question again getting for a small company yeah a lot of times what you're talking about my pains met with a big company in the end i working with company head of life down and 104 so how could I get them up to speed with its blocks so

they're able you got offs so I'm not a rookie keep it at like I said most if I'm going to see devil on top of an agile methodology so I need to understand those four cycle with feedback loops certainly being a boss is enough to get that started on that journey attending conferences like the dephasing around the world there's always one going on somewhere in the world there regionals if are all over the place there are lots of for me to go talk about this stuff you learn by doing so back I've even arguable it's built by the practitioners themselves and understand what doesn't work in the move on the target sentence doesn't exist what they don't have in change earlier

learn accountable not late operations security humane all things equal same time with your pushing left on the process business pushing right here in a world where you've got a different change profit margins 8:1 something that makes money they wasn't Alan this one accountability not designing avoid yes sir really good talk when things mention willfully really wanted to entrust with your developers want them to come be sometimes that means getting in the fight for them now they're they're 40 hours a week doing coming home I don't even think about the other stuff sometimes there's enterprise level you have to building programs give them tools given trains or enterprise level tools creating trust yeah buildings going for them like I said before that a

forward approach works really well because they don't want to have to do hope you could help them get up what kind of do that go out Kate training and YouTube code things like code manners yeah development I think take your time learn how to write code or write code capture I can't read efficient collection rate over time tortilla systems later is one more listen have you approached customer pregnant intimidation when they're not going to take up things cuz we're producing to take up well so there is nothing less right if you are releasing something that has security changes in it look at you all you can do and what does the place like the thought of it

Security's head was development so you can think about that as companies you can stay a lady think is because there's lots of reasons an organization where they might not be able to being these changes they might have change walk or they can't black house where they can't be choosers and project all you can do is educate them and said here's what's in here applies importance excuses and a vapid take or not take days of endless just so long educate the properly thank you [Applause]