Security at Speed: Securing Code in your DevOps Pipeline

all right go ahead and jump right into it it's three o'clock and I've got a lot of material to get through to make sure that nobody asked me any questions so hi everyone my name is Daniel Burnside I'm we've given a talk today about securing code in your DevOps pipeline it's gonna be a bit of a journey we're gonna have to cover a little bit of DevOps to begin with to make sure everyone's on the same page and hopefully by by the end of this everyone has a profound understanding of securing code near DevOps pipeline as well as maybe learned a little bit about yourself a little bit this my obligatory Who am I slide I've been working ten

years in the ciders with background and sec cyber analytics instant response bug bounties and tons of tons of paperwork a couple of companies I've worked for and to include government entities to include South Carolina National Guard operations officer for the 125th cyber protection battalion and I'm also been working for a couple other various government agencies in that capacity when call to duty shameless plug and I told my boss I would do this I really need network security engineers so if you're in that field please let me know we need to we need to find some folks thank you um so let's start off what what is DevOps agile and all these different sort of software development frameworks but

DevOps is kind of a momentum shift in the way that we produce and push code out to production and the highest possible level DevOps is the means by which we are able to write develop a push code out to production with the most minimal human intervention while writing secure high-quality code and there's a lot of pieces to this it's not there's no single tool that gets you to a beautiful DevOps environment it's not a single person it's a organizational change and mental shift to bring collaboration between several dispair teams specifically development and operations and I'll go ahead and let you figure out that that gets you to DevOps we'll also talk later on a little bit

about dev sec ops which is really the crux of the discussion integrating security into this and why that is such an important feature and how it allows us to push code secure code with minimal human intervention which you know for those of us who are in the pen testing field or those of us in the security field really without manual touch or intervention automation can be a little unnerving because what are we missing what are we not finding so we will get to that discussion later so let's talk about software development in general release timelines so in software development when you release software you are taking a what should be a customer demand of the capability for your product and

putting it out into production for the world to see and use companies that get their product out faster better more secure and more reliable tend to succeed companies that don't tend to fail you know in the 1970s you know we're still on mainframes obviously things are gonna take a little bit of time when it gets in 80s and 90s we're looking at quarterly yearly releases when's the next you know Duke Nukem game coming out right the 2000s we found them moving to quarterly and for kind of more advanced companies monthly releases the 2010s you know we're looking at and I made those words that they're not until I promise we're we're actually pushing out code by

the minutes and seconds and that one down there at the bottom that is accurate that is an F as a statement directly from Amazon that they push out 50 million for releases a year that is one code release every point six three seven so in the time that we've been talking Amazon has been pushing out probably about a hundred or so releases and those are all the way through the pipeline from development to build to test to the battery of quality assurance testing such as user acceptance testing and all that out to production and and likely on a red or a red black or blue green cycle so the pace of releases is reaching a breakneck speed Amazon is at

the top of the game right now truly Netflix is is really big in this field as well and shockingly one of the kind of founders of the deficit office philosophy when the companies that really kind of kicked it off a good bit as well as Etsy out of nowhere but they apparently had a real crack team that I pulled together and started pushing out releases a very quick pace so here's a history and origins DevOps I will do a shameless plug for this is not my book but the DevOps handbook Jean Kim and the rest of individuals here back in 2010 were attending a development conference in San Francisco they saw a talk at development basically

essentially development speed automating development pipelines and they had an epiphany and as many of these things often do they gathered in a bar afterwards had several drinks probably and and came up with a concept of DevOps and it all got together and basically wrote the DevOps handbook another great book to read as well if you're looking to get in this field is the the Phoenix project which is a story of a failing project and how they got it back on its feet through the continuous integration continuous deployment but as you can see you know the impact of insecure poor software releases are pretty profound you know when we look at companies back in mainframe days if we wrote that code

or we had a bad release the impact would be a substantial was literally the whole company going under if you're in a tight market with fierce competition you push out a bad release with bad features that's an unstable you run the risk of company going over over time when we get to client server you know we were on the risk of product lines or divisions failing but you know really the company's probably be secure although the CIO will probably not be around very long and now that we're in the 2000s we're looking at you know the much lower cost for failure and really we just were at risk as a product feature and the cost

is more negligible at this time so as we were able to push releases out faster and smaller increments the impacts are smaller to the organization and arguably less people are chewed out and/or fired at the end of the day so let's talk about DevOps so there's kind of three core tenets or philosophies of DevOps and they all build upon each other one being flow so when we know software when we develop software and we push it out to environments that is the the pipeline that we're building flow is how can we build a pipeline that's automated where as a developer writes code commits it it never touches another human hand or has any manual intervention until it

rolls out to production that's the goal of the first step is flow the next one is going to be feedback how do we get feedback from from things that flow out so nobody builds a CI CD pipeline or a software development pipeline and gets away with perfection on the first time I don't think that happens in any field in the IT industry or cybersecurity so how do we get that continual feedback coming back and so that builds once again now into the third one that continuous experimentation and learning so you see much more smaller feedback cycles it's a little on the screen here but a lot more smaller feedback cycles within that pipeline so if we see an error for

example on test we get that immediate feedback it's not something that we see back in the production work its way back in so we're able to resolve issues faster and get that code out more securely and with a better quality because it's not all about cybersecurity with debt off certainly not it's it's about getting code out in high quality manner well I had will bake security in in a second so agile equals DevOps right it's not quite so DevOps isn't that that logical extension so agile at its core is focused primarily on the development of the software whereas DevOps incorporates the not just the development of the software the building of the software but also the packaging

on environments the creation of those environments to host it further both testing and QA environments and production and automation of the rollout to those environments for the testing so DevOps encompasses quite a bit more than that not only that also includes a continuous feedback so if I'm just talking about rolling code from development all the way through production that's technically continuous integration / continuous deployment which I'll get to the next slide on but DevOps adds that continuous feedback it can adds that automated deployments in operations to agile and very importantly sweet sweet sweet metrics everybody loves metrics most importantly your C suite loves of metrics and if you can demonstrate increased success with every iteration of code rollout you're

gonna show to your your C suite your directors any executive in your organization hey this this was worth the investment not only because we're getting code out more quickly but from a business perspective now against our competition we're rolling out features before they can and that's very important particularly in this hyper aggressive you know IT industry that when you find ourselves in today whoever gets it out to the market first generally wins that statistically proven pretty true over time so getting features out faster than your competition is absolutely critical so talking about CI CD well when we look there's two CDs which is greatness of a different letter so continuous integration is essentially is the build

process we're getting we're getting code built we're doing the unit testing and we're to point to the various testing stages continuous delivery is getting it all the way out and ready for production however there's a manual step of approval before it it's production for some organizations this works better for them given the sensitivity of the material depends on what works best for your organization whereas continuous deployment is the second that the developer hits commit run it back into trunk it runs to the battery of tests and rolls all the way out to production assuming it passes every gate if it fails a test obviously it doesn't make it to production so that that's kind of

obviously the premise there so what does DevOps look like and so and I'll be upfront the tools that are on here lately stolen from obviously so there's a variety of different tools that you can use to build out a DevOps pipeline these are just examples but for example in your development phase we have github we have a repository where we're storing information over committing stuff back to trunk once a commit is made back to trunk Jenkins which is going to be our automation software it's going to kick off that build and coordinate some of these other pieces once that builds complete it goes over to docker hub and it gets thrown into an actual OCI

container at that point Rancher says hey I've got a new container that got built it means we have new software it's time to test alright let's take that container let's stand it up on AWS or if we're using vmware locally we can stand it up there too you do not have to use the cloud for DevOps that sure helps but you do not have to use the cloud for tech for DevOps so it deploys it out upgrades it as necessary and then we get to operate and so there's a variety of different tools that help us along the way to get to that but what you don't see in there right now inherently is security so we need add it right so

let's go ahead and pause security in there and I think a lot of folks have if you're especially if you're in the software development community understand that that bug's cost money you introduce a bug as a developer you're costing your company money it's not meant to be a dig we all make mistakes we all introduce bugs even senior developers do it all the time but what we need to do is try and remediate those issues as much as possible and I'll be frank security vulnerabilities in code or bugs its unintended functionality of the code that unfortunately in the case of a security vulnerability results in potential compromised down the road and we all know how that goes and so if we fix it

in coding it costs just a smidge more but as we go down the the pipeline all the way to production those costs begin to increase and not just a little bit I'm talking exponentially to the point where a vulnerability that could have been fixed in requirements architecture coding is going to be on the scale 25 to 30 times more expensive to resolve in production why because we have to take that we have to either stand back up the old software or just anticipate continuing to use it and then it becomes an all-hands-on-deck emergency people burning the midnight oil we're coming in on the weekends after we released to our mediate issues everyone's stressing out the CIO is

pulling out his hair that the c-suite is is you know got it because of grinding their axe to lob off heads and so in addition to that just the sheer cost of bringing it back to word to development to get it done we find ourselves in a situation where we've built a toxic high-stress environment because we obviously don't want to deploy bad code to customers that's how we make our money and we don't want to do that so we have we have a couple different words for it is it dev sec ops as a sec DevOps as a DevOps sec and you will actually find these different variants all over the web but at the end of the day i say

the bulk about 80% of the community uses dev sec ops i think it just rolls off the tongue better but there's also kind of a mentality of what order things are going in right development happens first and you add security and then it rolls out to operations right different organizations may do things a little bit differently maybe with the big security and before development but but where do we integrate this where do we integrate security into our devops pipeline essentially everywhere right we can we can integrate security into every single step of the process that we our industry has become that robust enough where we can actually accomplish that so first one development development is the best

place well right after architecture of course you want to make sure you architect a strong security but immediately after that where rubber meets the road and we're actually putting code down where we introduce a sequel injection vulnerability it says cross-site scripting cross-site request forgery all of these vulnerabilities this is the best place to do it because if we find a vulnerability here it takes almost a zero time to fix it if we find it in production what we found is something that's running in a rich graphical user interface where find that in the code that's what takes that time that's what adds that cost as Oh God where is this so when we fight we

fix it in the development phase what we're doing is we're taking that whole piece of investigation piece out of the equation we should know immediately where we introduced it one of the best ways to do this is to integrate plug-ins into your integrated development environment so if you're using IntelliJ or Eclipse you know or any litany of other integrated development environments there's ample plugins that exist out there where essentially if you're typing the code if you for example absolutely input a you know a sequel injection vulnerability it's going to tell you right there there's a little box that hey you did something bad before you commit this fix it and you can also integrate these plugins

with Jenkins so if you attempt to commit code in that has existing vulnerabilities and get that set thresholds high medium low whatever you want it to be then we'll say stop don't pass go we're going to break this and you're gonna fix it so you will not be able to commit your code until you fix the vulnerability what was really great about it as well as when we trying to implement security programs in companies what often happens resistance right cybersecurity is is not a is a cost center right we're not generally making the company money and less for a cyber security company right so when we introduce cyber security into organizations the first thing that

generally happens is look what is what extra work do I now have to do the beauty of integrating plugins into our IDE s essentially we're telling developers like there's nothing more to add you're just looking at bottom left of your screen when you're done coding did you introduce something just fix it real quick five minutes later we're done in the better part of it too is that you're educating your developers along the way as well we're not waiting till oh crap something happened out there to do the education it's happening at this time in addition to all the other education programs that you're running your company for cybersecurity which we all know beneficial so moving on we'll get

to our bill phase sass tools so you can also integrate these with your automation suite so and the beauty of that is putting in gates so for those who are not familiar with sass tools these are tools that essentially scan your raw source code for vulnerabilities which could be hundreds of thousands millions of lines of code and it tears through it looking for going through its heuristics and all the signatures and looking for potential vulnerabilities in your code great that's fantastic unfortunately ll code that is it is written by human being is essentially custom code unless you're using a library or framework it's custom code so to write signatures for every single piece of custom code on planet

Earth is very difficult resulting and of course false positives millions and millions of false positives SAS tools are important I'm not trying to discourage people from using they are quintessential to to the security process but we have to have a fundamental understanding that there will be a lot of false positives developing a program that works hand-in-hand with the developers who wrote it to kind of mitigate those as kind of critical right you want to make sure that you are you know you didn't write the code you're the app set guy you're probably one app set guy of a hundred developers which currently right now is the industry standard 100 developers one app set guy gratulations

you are literally got your finger and the dam holding the water back so if you have to work with all these different teams that's going to be a problem a lot of SAS tools will allow you to kind of fine-tune them over time or if they integrate with dynamic application security testing tools then you can often mitigate some of those false positives as well but the beauty of this is if I write code somehow it gets past my IDE plug-in the SAS scanner finds it hey you introduced a high sequel injection finding stop don't pass go fix it so the problem is now so starting to cost a little bit more money now right we've put it through the scan it's taken

several hours potentially depend on the size to build and and now I've wasted developer time waiting for that scan to finish and now they've got to fix something and do it all again so as you see here we're starting to add a little bit more of a headache to remediation a test face no dynamic application security testing tools basically there they're testing your running software so essentially they'll you'll run your application need a web application this - tools gonna go in there say hey I see this form field let's try some sequel injection - taxes didn't work cool no fine and they did problem so - tools are great in software development organizations because of one

very important thing they provide the proof of the vulnerability if you go to your developer with a handful of say a thousand static analysis findings they're gonna be like what what is this how many of these are false positives which one do I really need to concentrate on a couple of things happened one they're apathetic to your cause and they lose confidence in you as an application security person right whereas dynamic application security testing is great you go to them it's like hey we ran the - tool against the code and no joke there's some cross item across forgery issues here and let's take a look now once again so low false positive rate increased

confidence but it's now a higher cost - remediated right we've moved further along the pipeline we now have an issue in a running application that we don't necessarily know where it is in the raw source code the developer probably has a strong idea and they'll be able to find it quickly but in order to remediate it now we were mediated back it back at step zero when we're developing the code it goes all the way back through and only until it gets to this step again we'll be able to find that vulnerability so you can see how that can be an issue I mentioned before a correlation with SAS finds there are some applications scanning tools that will do both your

SAST and DAST scanning and then actually do their best effort to correlate the results the beauty of that is oh I've got this issue here I go back to this line and this file and fix the code there try it again done packaging all right so this is this is some of the the more newer stuff that's been coming out lately obviously we always secure our platforms that's not new right let's lock down Windows you know let's remove all the non-essential services for Linux these sorts of efforts but now we've got contain oh boy so containers are fundamentally changing the landscape of how quickly we roll out go to production environments and how effectively we're able to

maintain those environments and keep them up and running so there are a couple of tools here some of the bigger names twistlock is a paid service that mostly focuses on docker containers and open SCAP which is a scanning tool for the OCI format which could be turned in basically essentially any container right so oh say a format can be turned into a pot man that docker redshift any of those sort of containers that are out there obviously a lot of people focus on docker because they are the biggest name of the game about 88% of the market at last I looked was docker containers so twist locks about to have a pretty awesome year get on I think it's Palo

Alto who bought them and they finally get to production and so production here you're gonna see a little bit more of common more standard tools that we've used in the past but the biggest focus here is going to be on monitoring so and this is really the ops piece of dev sack ops where we're trying to find here is that health the security of a running app in obviously performance and performance is a security issue because if a performance is so low that users are unable to use their functionality we've ddosed our own application and it is positively absolutely the worst place to find a vulnerability and I think we all know why it's in production so you

see that vulnerability so does the adversary so is somebody looking to do something delicious to your network there are a couple of tools runtime applications self protection tools that you can use that essentially live in your code and let you know if something is attempting an attack on your system it'll give you a feedback on that and there's also integrate with you you know your comments team tools else tax blanc and various analytics engines like l elasticsearch so it's and really that's really important too because in deficit hops in the ops piece of desktops you're fundamentally looking at the performance of the system you're all about the production you know it works but we're looking for is you know average mean

time for a user to get results back or you know how many vulnerabilities did we find along the way that sort of thing so we can can do continuous improvement once again it goes to but it talks back to that third second and third phase of feedback and continuous education right in the DevOps way so when we do all that we come back to our CICE pipeline same one i said before but now all these great tools at the top so rips tech integrates in your IDE is very code you know and once again example companies that could be any of them these are just ones that I was able to Google first but very code I'll actually have used all

these but so very code you'll find can do both the SAST and DAST many other tools do that as well now it's not just singular to amateur on single a single company I'll say we're using docker hub twistlock is probably what we're going to want to use for this environment and then spunk down the line to give us our logging and feedback from those running systems right but you know obviously all of this cost money so I found the free version as with any open source freeware tools these are not as generally capable as it open source tools that require a little bit more work on your end to set up and configure and maintain but if you're a small

company and you want to set up a devops pipeline there's certainly ways to do it so you know you've got fine security bugs in the left hand corner that's an integration predominately focused on Java I might add for your development phase sonar cube is a QA and security tool that you can integrate in the build phase we've got clear for packaging Oh everyone knows that oh is that simple right so we've got that for our testing phase and we got running environments we got Falco for the deployment of great phase of course alas a search on the operation and for our metrics reporting so once we all integrate all these pieces in we finally get there's this

glorious dev set ops environment where we are able to produce code roll it out into our build tests QA environments and out to production and monitor the health and security of it we've reached this panacea of dev set ops everything is is going rate you know people are working together we've got teams that are happier we've got operations that support security we've got security that supports our developers and likewise back up the food chain and we got continuous feedback so what are everybody in the entire pipeline and basically this factory of development has become now is there's one more unified team core focused on this technology but what about manual testing right it's still a thing so manual

testing still exists right we can't ignore the fact that automation can can get us pretty far but not all the way so we try and minimize it though so if you have a man essentially in dev SEC ops the objective is if you have a manual repeatable process we're at the point in history where you can code it and just have it repeat itself so one of the one of the kind of founding tenets of DevOps in the past was was manufacturing if you know you have a repeatable process if I am putting a tire on a car and screwing lug nuts on does a person have to screw every single one owner can have a

machine do all five at once which is what happens now and so a lot of deba DevOps functionality or DevOps philosophy is built around some of these core tenants the biggest one being the Toyota manufacturing process believe it or not so for those who don't know the history Toyota is kind of surge in power and eminence and really reliability kind of came around from this process where as a cars moving down the line they did immediate feedback in the process and there's little tiny loops in the process that's what Toyota was doing as a Toyota you know we'll put her on ER if I felt that something could be improved there is no joke a red cord above my station I

could pull it and it will notify somebody to come over or if something broke if something broke pull the red cord manager comes over if within 30 seconds they can't resolve the issue the entire Simula engine shuts down and it gets fixed sound familiar it sounds like the software development pipeline and so what we've done now is we've taken all those repeatable processes of manufacturing we're building a widget here all the way to the end we have a raw source material which is our code and we're building out this beautiful car down that at the end of the line so that that really should be the focus if there's something that can be automated automate it it makes

your job easier it makes the pipeline go faster and allows you to focus more on on some other core things like for example it's very hard to automate all the pen testing you know this because you know we have to write custom code right so there will still certainly be manual penetration testing but you're minimizing the amount of manual effort that you have to do in your organization to do it and then we also have secure architecture design no one's gonna be able to push a button and say building a secure architecture done no way that's why we pay you know networking companies millions of dollars do that stuff for us right but so there's always going to be

a human piece to this pie but minimizing it as much as possible is certainly the objective um so just it just kind of a quick overview benefits of DevOps is that security testing automation we're not doing it manually I'm not running a SAS scan and eight hours later you know shoving the results over all my dev team and having to suffer through it's running continuously ideally you've got a SAS scanner that does incremental scanning just checks for the small changes and that way they just get a small little plate of vulnerabilities they need to find shifts focus from that manual value evaluation to refine automation which of course over time saves money so the less people I have

with their hands in the pot humans are prone to error I think we all know this so any time that I had manual intervention in my DevOps pipeline I'm also potentially introducing bugs vulnerabilities flaws in the code obviously we discuss how it saves money over the long term standing up a DevOps pipeline in the beginning can can be a little bit of a shift because you're probably going to keep some of those more legacy pipelines built in place until that DevOps pipeline step which means you're buying the tools and products to get it there and also the people to kind of coordinate this ever you don't need a lot of people for devstack gob truly just a few because what the DevOps

team is really going to be focused on is bringing other teams together you're not building a brand new dev sack ops team what you're essentially building is a organization that's just working more closely together obviously the higher quality more superior faster code deployment and then getting it out to the market first I cannot stress that enough one of the biggest reasons while like Facebook and Amazon and Netflix have become very successful companies is because as customer demand for features is fed back to them they're the first of the market to get it back out they don't wait they don't wait a a month or two or three months they literally wait they're not waiting that time at all they're

waiting hours and that new code is on production fully functional secure and been tested against the whole battery of tests all the way down to unit testing so really that's that the crux the issue that that's what we're looking at there so with that said that concludes my presentation I have a unfortunate way too much time for questions I'm more than happy to take them so if anybody has any questions feel free to nope no question I ran back

okay so so the question was how do we how do we convince our developers that you know to start looking at these false positives and comb through them and find out which ones are true right and and that is that's actually a really great question because a lot of us deal with that that are in the abstract field how do you get how do you convince your developers to it a big piece of it is actually for your app set team it does benefit them to have development experience in their background you know so first I'll take me for example I'm a computer science major do not ask me to code anything beyond just some basic scripting stuff I

can read code all day I can understand code and I understand what it's doing and I can find the flaws in it but what I can't do is say hey build me a rich web application that does this this room I'm not a full stack developer at the end of the day what I found is the best method to resolve this because before I'll give an example I started a company they had a somewhat older tool that was doing their sass scanning right they got it on a license very early in the SAS scanners inception as a company and they got a great deal in it and they stuck with that deal even though the tooling

capability continued to improve to reduce the false positive rate add more functionality and just generally scale with the organization but they refused to pay more because I think somewhere out of ballpark ten thousand dollars a year it was she we were scanning millions of lines of code millions so scans would generally take 24 to 40 hours so first issue how do I build a pipeline if my scans take 24 to 48 hours to complete scanning against a monolithic application you know I told developer hey you need to wait a day for me to get these results back oh by the way you know I think we calculated 93 percent of them were going to be false positives 93

percent and and by and large a lot of SAS scanners are not a whole lot even though the high performers and a whole lot below that I mean I think we're looking at 50 60 for the best creme de la creme painting out the was warm so when I take this pile of false positives to to developers that was the feedback I got and that was a good learning lesson for me too it's like wow you know they're right I'm asking them to look at this pile of junk and find the gems in the desert essentially right so so one of the things that that we asked them to do is sit down and explain hey we can refine

the tool and as long as we have a fundamental understanding of why these are false positives then we can move forward and refine the tool and do that also as I get results even at the tour time it doesn't work I can do an initial parsing through these findings and minimize as much as possible there are some false positives that even junior developers can't find or junior mid-level Avella personally seen developers made wrote that code will be able to say yeah but it's really not a finding because we have this mitigating effect several you know steps up a good SAS tool will look at the connective tissue in the code essentially so I've got this I got this function here and

does a function call higher up and another function another function other function at the top of that stack it has that protection there you know it does the input validation at the very top of that assessment process and so that in the eyes of the developer depending on the organization it mitigates that false positive but we don't know that until we look all the way up there so a lot of it comes down to the education of the half-second Geneva's work in the issues and the developer who actually wrote the code so it is it's a difficult thing to figure out there one of the ways that I try to mitigate that was was through

education so one of the nice things about the assassinator is like hey these are these are injection findings this is cross-site scripting findings you got reflecting across site scripting here but it categorized them so one of the steps I took in that organization was to say okay I'm we've got a mountain of well we had amount of tech debt start with that but we also have a mountain of technical security debt unless let's take them you know you don't you don't even elephant in one bite right need of one bite at a time and so what it is like all right so next month let's say January first I'm not versus royally hungover January 2nd January 2nd we're going to have a

class we're gonna have a brown bag and even better if you can get the company to buy pizza because God will developer show up for pizza you could you could put pizza down for like the worst thing everybody's got to get their flu shots but hey there's pizza and you get the whole company there and we're gonna sit down we're gonna have a brown-bag about injection findings and so what I did is I gathered everyone in the room we know every injection findings but most importantly I stowed it I stood up an awesome Attila day server in the room that no one could login to was Mattila day - sorry and allowed them to actually

test sequel injection attacks on that vulnerable server just to show them what the impact of this vulnerability was so they got to see sequel injection vulnerabilities command injection vulnerabilities basically the plethora of injection role of those that do exist but have one very simple solution is input validation and they had a more fundamental understanding at that point I said hey over the next 6 months we're gonna clean out all of the injection findings in our code and once it sounded zero on the last scan at six months we're closing the door so anytime that there is another scan and we find an injection finding we're gonna break the build it's going to go back to you

you're going to fix it and your code does not pass go and that is by order of the this is Oh CEO and that was one of the ways that we kind of made it a little bit of a forcing function but if you ignore your code for a long time you're gonna have a much bigger mountain if you get that mountain down to flatter the zero vulnerabilities and over a couple of weeks maybe to get five or six more that's a lot easier for developers to chew on had a brand new skin so yeah another question okay I tend to buy their own yeah anybody else have any questions yes

yes I haven't seen it as much of a comment practice I have heard of it so it the difficulty is is that you're generally when you run into organizations that are transitioning to dev SEC ops they are finding themselves on the back of their heels already and trying to catch up to the industry and at the same time don't have the investment potential that Google does to bring people in so most the time when I have joined organizations that are trying to stand up dev ops or dev SEC ops in particular they're grabbing existing bodies and say go educate yourself and bring it back let's stand it up so the the reliability you know the position that could best be

described as reliability engineers organizations I've been working with is that senior Deb who who also has spent time on the platform team on the architecture team kinda as a fundamental understanding couple ons about their level of what could break and what needs to be fixed that person tends to be paid very well and get burned out real fast so having that that site reliability engineers I think I have to agree is actually a really critical position so and to make sure I'm capturing this the question was do you see value in reliability engineers being part of the DevOps process any other questions

okay so the question was in in my experience what are some of the best practices to transition from a DevOps pipeline and and and culture to a deaths a cops culture and there's a great question because it's like I mean frankly it's probably similar to many of our other experiences where we try and integrate security to anything in our organization immediate resistance why do we really need to do this we're adding more overhead what why are we doing this and at the end of the day the beauty of the beauty of going to DevOps is that you've automated all the processes you've taken a lot of the work out of not just the developers hands but also your your

systems or operations team whichever flavor you call it and you've basically taken what would have taken months that weeks days to potentially hours and minutes to actually accomplish so when you introduce security into a devops pipeline to make it dev set cops you're you're not significantly introducing a lot more effort or at least you shouldn't be because you're going to be automating a lot of these pieces now obviously that the pen testing the pieces the end base selector curve either by regulatory requirements or whatever that that looks like and that can't necessarily be changed but if I'm adding it like say for example find security bugs if I went to my leadership it said hey I want to scan our Cove or

App Center for application vulnerabilities every day give the feedback back the developers you know and have meetings with of god more meetings you have the less development happens but if I need to have a meeting with them to discuss these findings that's a problem leadership is gonna push back on that's like well no you're wasting our developers time their time is very precious as it is this and if you're a software development company it's your core resource so any little thing in that I introduce needs to have the most minimal impact developers and that's why I was kind of foot stomping a bit on the IDE integration anything I do for my developers to see the issue at

the beginning is super beneficial education I know we said just about every cybersecurity talk from you know phishing attacks all the way down the line but education is important too and there's a different education for developers right if I'm educating an HR person on how to stop a phishing attack yeah a developer needs to know that too but developer needs to know a good bit more my HR person doesn't need to know how to you know resolve a cross-site scripting vulnerability but by golly my developer really needs to and so there there's a secondary education that needs to take place the more educated your developers are about unsecure coding practices or using insecure frameworks or using outdated

struts to like a particular company did and lost all of our personal data related to our credit then the that that's really kind of the crux of what we're trying to do here is get as your developers better educates their writing more secure code there is some initial investments in that upfront of time of course but as it pertains to the pipeline if I've automated the SAS scanning if I've automated the IDE plugins of an automated that - scanning those are all at this point in time without DevOps manual processes I have to schedule it right I have to wait for a new environment to be stood up as QA done testing so we don't Dass our own QA

environments when I hit it with a desk scanner and then what's my timeline to get results back do I put it in the sprint iterations do I had it - they're their backlog but so if you have an agile to build an environment and you add dev off DevOps you've already got a very flowing organization already and then you just are squeaking a little bit of security incrementally at a time leadership I think it fundamentally understands that security is at this day and age and necessity but it is a cost center it cost the company money so the most minimal impact we can have but better and if that is just something as simple as that free IDE plugin but well that's

a win I'll take that and then we'll talk about the SAS and - scanners down the line because they'll likely cost money so it's like anything with security as long as you minimize the impact to the actual money bringer and nurse which is your developers they they produce for the company they produce product it rolls out production that brings company money minimize impact to them and I think you'll see 6x that's what I've seen thank you

perfect so but there's a lot of metrics you can grab right and it's going to depend on what your leadership wants to see but so the question was what metrics do your senior leadership want to see in the organization gotta make sure I capture this the biggest things are gonna wash you believe it or not are not explicitly tied to generally security and security this is oh is gonna care about that right the leadership will care about it or they'll see the value or the minimization of impact their organization but what your what your CCO is going to want to look for is essentially hey meantime for idea of the tool or feature the feature you know

spark of interest from the start of organization to production average mean time of downtime in production they're gonna so you know those metrics of you know Amazon has it ninety-nine point nine nine nine nine nine nine nine percent uptime I think it's somewhere around there and by golly if it goes to ninety-nine point nine nine nine seven it's somebody's getting fired those are the kind of the things they want to look for they're looking for that that mean uptime reducing any available downtime as much as humanly possible and then minimizing the amount of environments they have to stand up so the least amount of environments you have to stand up that saves you money if I'm paying

for you know ec2 instances and Amazon and have to stand up like three hundred of them to get my application running but I found a way in DevOps to minimize that and then that pipeline down to 250 that's cost savings to the company and if I keep it as minimal as possible let's say I mean I'm a toy maker and have a website that needs to scale dramatically right around Black Friday then you know I've got an opportunity there to keep my cost at a minimum and tell of course day but then also only scale of an accessory with beauties of DevOps really robust DevOps programs so we're talking you know Netflix is a great example right let's say they come

out with things for it's um for now I think certain thing for comes out what's immediately gonna happen that night to bandwidth requirements and never is this going to just screen through the roof I mean so what Netflix has done is a built and they they have their own built home grown tool that is free fifty free I highly recommend it called spinnaker and it is your SPI and a keer and it essentially takes your code from the second is built and done in the test process and runs all the way through to production oh no by the way it'll automate scaling for you as well so it'll work hand-in-hand with your your load balancers and everything so the

case to scale your servers any sort of other instances that you have in the cloud appropriately based on that feedback so hey this the servers at 75% capacity boom stamp another one it said this one just hit eighty percent capacity in three minutes stayin to five more real facets or scaling pretty quickly so it allows the organization to minimize like that so wasn't a question here

boom that bad boy right there is a DevOps handbook as well as the Phoenix project so those are kind of quintessential Bibles in the process there's a lot of the companies are freely putting out information of their successes so Netflix is a great one Netflix is they built a series of tools to help them in the DevOps and they've made those available for anyone to use or just really amazing for free 50 treated which is the best way to do it but if you're looking at you know how do I break into this this environment of this field you know the IT background from your education is a big piece of it right understanding full stack

development is really important what pieces go into this web application what parts what are the phases of development and how does that all work the other big piece to you is understanding systems so when we stand in in oh man at your point in time we are in your career you've got to dive into containers because they are they're coming fast and furious you know you can stand up a VM and consume this many resources on a server and Sam three or four you know VMs on that server they're robust of course but inside each of those VMs I could stand up five or six docker containers because they're only using the core resources that they need

inside of the operating system so containers are extremely lightweight I can stand up a you know a blue tube container in in seconds it's extremely quick because it's only pulling the resources absolutely needs to do its core functionality so focus on containers container security is a big piece so if you're jumping into this field that is rapidly becoming a very big concern for a lot of companies because as any new technology someone's going to find a way to exploit it and they already I started started to so if you have to go out that you'll find there's quite a few container security positions they're being looked at in addition to cloud security so understanding that various cloud

environments AWS Azur I guess Oracle's still a thing they still have a cloud so understanding how those various cloud components work each other and the security of them too you know whatever the security group says how does Amazon handle security differently than Microsoft I think it may actually be instantly surprised that despite this smaller market share of Microsoft has a very robust product so this answer your question perfect any other questions no no excellent all right I thank everyone for coming out today I appreciate it and take some away [Applause]