GF - When DevSecOps Fails

our talk today our talk for this session is when devsecops fails with tonya janka before we get started i'd like to thank sponsors especially the inner circle sponsors critical stack and valley mail our stellar sponsors such as amazon the national security agency silence and others a reminder everybody cell phones this is a live stream event so please double check triple check your cell phones feel free to check them right now make sure they're on silent and at the end if there is uh if you have any questions at the end there is a microphone in the center of the room so please use microphone to ask the questions so with that um welcome tanya jenka hi everyone hi thank you thank you um so i uh i have a weird career and i do a lot of consulting with different really big companies like companies that you know the names of and then i also my company was bought earlier this year by the nice folks at bright security so i work for them but as a result of all my weird consulting i've seen a lot of companies do stuff and not all of them went well and so over the past couple years since i've been doing devops and dev tech up sorts of things there's like a trend of crap that goes wrong and so i have noticed quite a bit of this um so we're like we're gonna do devsecops and then it feels like this for the dev team um stuff like the security team's like cool we bought a sas and a desk and an sca and we got a secret scanner and then we bought this like weird startup thing we did that and then we put it all in the pipeline and then the pipeline ran 10 hours instead of 10 minutes and no one wants to hang out with us anymore and so i at first was rang this talk and then i i was like when devsecops fails and so i want to talk about how we can avoid those common mistakes and the thing i wanted to tell you at the beginning of the talk to try to help you pay slightly more attention is that i brought what i like to call the maple secrets from canada so in canada we have these things called maple trees and they leak candy i kid you not their syrup is sweet and delicious and delightful and we make candy out of it and i brought candies so if you ask a question after i'm gonna give you a candy all the way from canada and everyone gets stickers because that's how i am okay so i uh so uh who so what is devsecops so this is what this talk is about so if you're like devops software development that's the worst you might not want to be here so what the heck is this it's like what is tony going to talk about so we used to make software a zillion years ago doing waterfowl and i mispronounce it because sometimes i say water fail but basically waterfall at first was good because it was way better than no methodology for building software but we realized quickly about 70 of projects failed i don't know if you know that um so i went i went to college in the 90s and like yeah it didn't go that well um so then agile like really smart people came up with this agile saying and it's like instead of waiting two years to ask if people liked what we made what if we asked regularly what if we met every day to like see how things are going on and so that improved a lot but then devops happened and the idea of devops i'll get into it more in a bit but devops is not only a methodology for building software it's a complete culture change and so if you're like we got a ci cd and we run it sometimes you're not doing all of devops you're doing part of devops and that's still cool and that's way better than doing none but there's so much more you could have and so lots and lots of developers have been very excited about this for a while the moment i heard of it i was like this is the best um and then eventually security folks are like why are the devops people breaking stuff why don't they stop for my nice gates that i made for them they want to release eight times a day and i only have time to review code once a year and so we had to as security folks kind of like get get the net we had to like kind of catch up and see what they're doing and so then i start getting a lot of consulting calls about like how can we do this better how can we not piss everyone off constantly but still get things to be more secure and so i i just started having clients say just tell us what goes wrong so that we can like work on that stuff from the beginning like we don't because a lot of teams that they misstep at the beginning and they lose a lot of trust in the process so i'm hoping we could go over these and then after we could have questions and then maybe at 9 00 p.m tonight some of you want to sing karaoke with me but you don't have to you can just watch okay so who am i i am a giant nerd this is the mandatory about me slide um so i work at bright security we make a dynamic scanner and i am in charge of developer relations which means i get to talk at conferences all the time and they pay it's the best i founded my own non-profit or my own it is for it was for profit just to be clear if you don't make profits you're non-nonprofit i don't know if you know that but anyway i founded this online um community and thanks to bray everything's free now so if you're like oh this was awesome i learned a lot i have like a zillion courses i made and videos and tutorials and all of it's free so go check that out i'm like i have purple hair and i wrote a book and i do stuff a lot but basically i'm a nerd at large on the internet that's like the main focus you should get away from this and when i brush my hair i look pretty good okay so yay devops so i love devops i'm really excited and so security folks we've been like trying for quite a while to like get people on board the security train especially software developers and so we did we did not have good luck at this at first so like i don't know about you but when i was a software developer when the security team started talking to me i'm like why are they bothering me and getting in the way of my deadlines that was my opinion as a dev um and so devsecops became a cool new look so who's seen this slide this is a very famous slide by a dude named pete schlezlock who's very smart and he was like this is how some security people see devops uh they're like you know they're magical and amazing and they move fast and they break stuff and us security folks are left cleaning up their as my friend nancy calls it rainbow poo um but yay marketing let's call it devsecops instead and then it'll be cooler and then people will do it and so this is this is the way that i view so they're still breaking things sometimes they're still moving quite fast and sometimes it doesn't work out but we're trying to train them we're trying to give them tools we're trying to help them we're trying to like keep them on the track so they get there successfully and we've learned that if devs do their works in sprints we should try to do our work in sprints if they're using a ci cd some of my stuff should be in that ci cd i should not just close my eyes and pretend it's not happening because i'm not going to get a good result so marketing basically got involved and we're like yay um but so then vendors got involved and like we're at a community event that we would not be able to afford to be at if it weren't for nice vendors that sponsored it and so i work for a vendor now i'm vendor scum i think i'm making a t-shirt that said that but anyway i digress um but so devsecops made a lot of promises so like promises that life would be like this we would all feel like this all the time but really it felt like like what i saw with some products because i've played with a lot of products i saw this so this is the system development lifecycle so requirements would happen no security folks complete silence design security folks know where to be found code security people totally ignoring them doing their own thing and then they're like oh it's test let's run five tools on full blast in the pipeline and see what happens um let's have tons of false positives and break the build all the time let's break every build no uh and then security being like you guys suck uh and then the devs being like i don't like the security team anymore that i've seen this cycle a lot unfortunately and i know it seems pessimistic but i wants to avoid this so vendors promise life would be perfect but they left out a few details and like this kitty's like i thought this was business class um so i want to talk about how we can avoid failure how we can like look like we're really smart and we've done this for a zillion years even if we're new and if you've been doing this for a zillion years maybe you're gonna learn something that you could do even better so that is the point of this talk and i'm just checking my time because i talk a lot i'm so good okay so the first thing i see a lot is failing builds for the wrong reasons i talked to so many security folks they're like i can't wait to start breaking builds that's a little bit like saying i can't wait to break their dreams like when you're a dev so who here has written software before in their life okay how many of you have pushed code through a ci cd and you're full of optimism and happiness and you're like i'm gonna make it to prod it's gonna be that day yeah finding out the security folks can't wait to fail your builds feels bad like when you push to the ci cd you're like i'm gonna run a bunch of tests but you're not like this is crappy and i'm gonna send it you've like done a little testing yourself you're like i think this is pretty good let's run some checks okay today like i think i'm gonna get there so if we're planning to fail the builds just from the start like this is not good so i want to go over good reasons to build to fail a build so yes false positives really suck um okay so the first one if you need gates in order to get security done because a lot of people are like i need dev gates it needs to be part of a formal approval process it's not security professionals who are using breaking builds to get your way like it's not like i'm gonna break their bill to teach them a lesson i've heard that like i've literally heard that from adults i i walked in on these two guys once in the hallway where i had joined a team to do absec and then i quickly left and worked somewhere else and they were making faces at each other like and then they would laugh and the other one make a face and they laugh and i was like what's going on and they're like we're making faces we're practicing our your stupid face and i'm like what is this for and they're like it's so when devs ask questions in meetings we can make them know how stupid they are and i was like i can't work on this team i do not belong here this is wrong and so we don't want to break fields we do it if we have to okay so the next one is the pipeline should not be the first time we look at security like that shouldn't be our first thing that we ever do like if they're if they're spending time gathering requirements about what they're going to build i want to be a part of those requirements i i want to have a requirement that's like we're going to do this type of scam ps here's your license if you want to run it first so you always pass like here's how i want you to build it securely etc in the design phase threat modeling like this shouldn't be the first time and we're going to talk a bit more about that in like a few minutes in this talk so the next one is test the accuracy of your tools for a long time before you break builds so i put a tool in the pipeline if i just have it set to give me data and alert i don't have it so it breaks the build immediately because who here has ever gotten a false positive ever on a product okay all of you are lying or your arms are very heavy because like the first couple of tools i put in i was just like no but it told me there's this wrong and then it's it's not wrong i don't know what to do and so it i think it's important to put it in make sure you've got it going as fast as it can go you're like doing all of the things and it seems to be going well they've fixed the backlog of security problems that were already there then i turn on breaking builds and it's for high things and we see if they can start getting to prod with that in place cool then maybe we move on to mediums but most places i go to work they have like thousands of critical bugs that no one's fixing and running around breaking builds isn't going to fix that right and so if i can alert them as to the problem and then eventually they start fixing things they start getting better then eventually i can start breaking builds so another one is you can't if you can't get accurate results with your tool that's okay do the testing outside the pipeline so i don't know maybe some of you have bought a sass and not all sas are like this and people that make sass will be upset i'm going to say this but some of them give a ton of false positives like 90 like you said it to be very sensitive and it tells you everything that could possibly be a vulnerability and you're like this does not belong on the pipeline it just doesn't we can't break a build all the time with those positives because we're breaking everyone's trust in us so that doesn't mean you don't use your tool so let's say you bought a tool and it and it's from a super awesome company and you bought it for three years so you have to keep using this tool no one's going to give you another couple hundred k per year to buy another tool so you're like i'm stuck with this tool that's fine but do the test outside the pipeline or only do a few tests on the pipeline and then run like your big huge long 18 hour saskian outside the pipeline you could still have friends this is important okay so the next one is use out of band tests um so this means like sorry i i like saw the next slide and was like what what am i talking about um so use out of band tests so what you can do is you can run a test overnight take those results um so my friend that works at a big company that makes really awesome athletic shoes so you've heard of them they took all the stuff out of their pipeline and what they did was is they ran it all outside the pipeline every day and then they send pass or fail based on their policy to an api and so every time the build is run it just checks the security api and it's like pass or fail yo and if you pass you go to prawn if you fail you already received an email telling you how bad you are you already knew um and so that worked for them and they went from an hour and a half long pipeline to a seven minute pipeline and everyone was like yes okay so another thing is consider if you need your security to be perfect or just good most of us just need good a lot of places i consult they have like i said thousands of criticals thousands of highs an absolutely insane number of mediums and then they'll get a tool and they're like well it's got to be perfect and find every single possible low i'm like really wise you could just ignore them and never you're never going to fix the lows let's like you're you're begging to have one critical fixed a week so it doesn't make sense for us to spend 10 times longer scanning to try to find those low vulnerabilities when instead we could have a scan that runs in like one tenth of the time and yeah it's gonna miss some stuff but it's gonna be good and so a lot of companies i'm like do you need to be perfect like did you make the covid vaccine are you doing anti-terrorism activities that are top secret are you amazon marketplace well then you probably could just be good most of us just need to be good we don't need to be perfect and so if you can be realistic with that you could choose a completely different tool set and save tons of time oh sorry i'll put that back for you okay um so then the last one i want to say is celebrate success when people get through the pipeline for the first time and they had to fix 400 things to do it tell them how awesome they are this is a thing a security team forgets we forget to say good job where usually they're like you made a mistake you're terrible let me tell you how terrible what if we were like hey this team made it through the pipeline the first time let's give them a shout out it sounds crazy but i've done it okay so this is the picture slide so if you wanted to take a picture of all the points i just made you would take a picture of this slide with your phone you don't have to but you can if you want and there's going to be a few more of these and i do these because i hate taking notes but i really want to have notes so it's like it's a heart it's a conflict right okay so i think most of the people's cameras are down i'm just going to give you one more second but you got to be like faster for the next slide just kidding i don't really have anywhere else to be um okay next so then also i have the unicorn i spoke at grim con this year and that's their logo and i just love it okay so slow tools so this is a thing i see a lot of so so i work at a startup and then we made a really fast tool but some of the original tools are like not fast i would call them not fast and so i want to talk about how we can go faster so the first time i had someone run a static application security testing tool on the app i was leading the team to build it took 18 hours to run and that's not fast at all that actually really sucks um so so i'm just gonna show you fancy cars i don't know if you like cars but they're very fancy um so you can perform as much testing as possible before you go to the ci cd so like as soon as someone checks in code you can run automated tests while the dev is deving i know it's not word um you can actually give them an ide plugin so they can test things themselves they can push things to their dev server and if you are an open-minded security team you could give them a tool they could be like pew pew pew pew and they could find things wrong we don't have to do stuff only in the ci cd so if you empower your software developers they could do really cool stuff so in the ide rate when you check code in then you can test as like a final last just is everything fine okay and then go to prod and this can save a lot of time another thing is you can perform just a subset of tests in the ci cd so there's usually things where i'm like that will give me nightmares if we have that in prod and so i will select those tests and then i'll run other tests against the dev server i'll run other tests when they check in their code but then in the ci cd i'm like i'm gonna do these like six tests like if there's injection i really need to know but i don't care if the security headers are messed up and i love security headers but i'm not gonna break a build over it so this is another way we could go faster um okay so use a har file to ensure your das acts like a laser so let me explain this so a har file h-a-r also known as html archive file is a thing that you can record of you doing stuff in a browser so qa people use these constantly so who here has done qa before in their life okay so doing the same manual test for the 10th time it's not awesome um i do not like it at all and so they're like oh let's just automate it let's record myself doing it and then as the devs continue to push stuff out i'll just play the recording and it's like it passed it failed so you can take a copy of those and shove it into a dynamic scanner and when you do that it only tests that thing so let's say you're working somewhere in your agile and you're using a ci cd and you're like oh in the sprint we made these two features the qa person's like awesome i'm going to record a thing to to do that and then they flip the har file to me i shove it into the desk instead of the dust having to crawl and look through things and then brute force everything and figure out what's going on it's just laser focused you can make it take 15 to 20