GT - Sympathy for the Developer - Sarah Gibson

hey thanks for the big noise in the small crowd um yeah so a sympathy for the developer my name is Sara Gibson and I work at Perico I know that ro has so we we scan a lot of applications we've used up for a security testing do static and dynamic analysis and we do a lot of it but really large amount of it and I've been working with scanned results at very code for a little over five years and I'm working with the results one kind of another servo static and dynamic in a bunch of different ways and right now what I would I'm back talking with customers so I'm dealing with results that are down in the weeds dealing with

developers talking about what we find helping them fix it instead of working through their skin results and what it is it's actually been found and I got into this role about a year ago also about a year ago certain audience member so I need a research topic for Minkow security course at Tufts University I had a really great idea I thought and I was gonna fix webgoat everythin like get down I was gonna become like that developer has been doing auto etnographic study of what it's like to fix web code and I submitted this proposal and and Bing said it was a bad idea he was like so you always research on what's out there as far as these

flaws why don't you look at that is there anything that's been been published in the last a couple years such as in the state of software security report that's interesting over time is there some trends is there some data that's out there that's more interesting than just fixing webco it turns out yes yes there is so as I was saying before very code has a lot of data so we do a lot of scans and for a lot of different organizations and we have data and it gets published in this form about once a year and mostly annually and it's talking about trends across applications across across different industries a flaw that we find for how they're fixed what's

happening some of those findings I was like really surprised cuz they're really similar over time and what I was focusing on my sequel objection it's the flaw type I wanted to fix and webgoat it's fairly prevalent we find it fairly often barracuda finds in static scans fairly well pretty good at that and we can detect fixes which was really important because I really wanted to know if people were fixing things that's what I was interested in because I was helping people fix things and I wanted to know if I was effective so it I had to be able to figure out or programmatically that things were fixed and I had made some assumptions I looked

at flaw prevalence data over previous sauce reports and the numbers were really flat and I didn't really understand why I didn't really understand what the metric was I had assumed that was a metric that was across the entire application base so the prevalence was similar because people were either not fixing very fast but reducing what they were introducing or as they were introducing flaws or fixing them so this is an equilibrium I had assumed this was flat because of some poor of equilibrium and I want to know why but it turns out that's not what measurement is umm the measurement is on first scam so of an application on first scan a first static scam does it

have a sequel injection flaw if so then it sort of shows up as in this prevalence metric if not it doesn't and it was a standard measurement across a report one of the reasons why is because it gets rid of some the variance a lot of these applications are really different really smalls and they're really big so they might have lots of different sequel injection flaws and might have only one and this sort of just take some of that away so you can get down to just okay across this application base of all applications scanned with us how prevalent is it to have an application that has sequel injection and everyone poops so the flaws happen I'm looking through

there seeing it really flat across all those years I think that this just comes down to flaws happen we know that regular code quality issues happen we know that you know we make the same spelling mistakes over and over again we know that we have as humans you know I I know how not to have bruises what I move in my house so I know how to do that I'm still covered in bruises um I know how to do these things but we make mistakes we're human so why will you pretend that security flaws are different why are we thinking that if we just teach the developers enough it so he just gives them enough

information that their flaws are going to go down because they're not so this is the data this is so this is from 2013 through to mid 2017 and this is the total sample size for this presentation is about 150,000 scans this isn't but the total sample size is about that big of this so the mean is 31.9 that doesn't mean that's how much exploitable sequel injection that's how much we finding of static analysis it's a pattern through the application of tainted data from an entry point through to a sequel statement we do see a fairly steady rate of exploitable sequel injection as well through through dynamic but I'm not really going into that here and then the

standard deviation this is ridiculous it's it's it's tiny I just like I'm really not gonna do any more of that it's just flat but I did take a look as I was curious okay fine it's really flat against all of our data just they're just like one organization that's just dominating this is there if we slice to the data by organization are we going to see the same sort of pattern over time yep we do I mean there's variance right there not every organization is exactly the same so this for me my husband thinks they stopped using sequel hi I didn't check this was randomly selected in blind I don't know what accounts they're they are fairly large

accounts or larger there are over a hundred applications with us since 2013 and and 30% is is right around here so you can see that it's pretty cluster it's it's also flat and within these organizations it's also fairly flat over time so across these different organizations of different industries and different sizes because you above 100 but there's variants there and how many scans there are we're still seeing that same pattern which is I thought was um kind of cool actually okay so we find sequel injection we find it about the same rate organizations don't seem to get better over time some get worse at introducing it right on first scan so it's not that they never get better cuz

they do they get better once they have that data in hand so fixed rate by applications so this is similar measurement to before so this is looking at but at but a smaller sample size one of the things that we we do is we'll scan applications at different will say this is a bucket of an application you can scan whatever you want in the hope is you're stating the same application over and over again so I'm sort of controlling for applications size to try and control for that but we're seeing that after three scans of the similar application size all fixed twenty three percent that's pretty great some fixed twenty one no change okay thirty that's guns that's not good

an increase in twenty six percent so they kept going they were like let's go I just assume that everyone's actually as they're developing introducing more flaws I suspect that what's happening is that the that might be on me that's game that might be something with it just not fixing them and this is looking at individual flaws so this is the individual flaw fix phrase about forty eight percent so of a flaw introduced on that first scam what a forty eight percent of them were fixed but on the second scan more flaws are introduced and they also have a fixed rate and and so on so it does go down it's not going up we are seeing a

decrease over time and flaws in in applications but it's not that full forty percent an individual flaw that that much but in full applications because as developers go there's more flaws being introduced it's a slower slower slope so good news but maybe not perfect something else that seems kind of cool and I'm ripping through this so I apologize but we have a feature that came out actually you can see when it came out I'm so this is this is the same data we saw before but a little different so this orange line is all of the scans again and I apologize if that's a little blurry but the blue line is not all the scans the blue line is

policy only so we introduced a feature called sandbox which allows developers to play it allows for consequence-free scanning that doesn't go the development team development team or so to the security team the security team can look if they want to you but doesn't get pushed up to their level of alertness right away so the security team is doesn't you know if you have flaws there's no consequences we've seen an increase so we introduced it in 2014 hit more across from 2015 in 2016 it became generally available and in 2017 we're seeing because it's not it's a minute in one relationship we're seeing 90% of first scans and sandboxes so you've many look you've got several developers

working on a project they can all have their own sandbox all scanning all checking for their own code checking for their own flaws fixing as they go so the time it comes policy that's the only 26% instead of 32 so I thought that was kind of neat and also this idea that you know developers are fixing flaws so it's not just yeah we see the flaws we see that flap we see that interdiction duction rate sometimes if you're seeing that over and over again there's this idea of all developers don't care they obviously just keep introducing clauses or not learning why is this happening well they they are and you have to but they're not learning on

the introduction side right so it's not that flaws are being introduced at a different rate but we can see that if you're giving them the ability to fix them if we're giving them the tools to be able to play and learn little fix+ we'll go through it and do it around the time I was preparing this there's also was a study that came out on general code quality which I thought was kind of cool this was a survey by O'Reilly and and the software improvement group the survey questions act about these are general code quality tools so it wasn't necessarily specific to security it was general code quality sir but I think it's important to see how we relate how

we fit into that landscape and it asked a book if people use them and then of those people who use code quality tools and ask questions about native fix and in that but 32 percent of respondents reported finding before fixing 80% or more of issues found using code quality tools well we saw the 23 percent all fix it turns out of that but sample 34 percent had 80% or more so it lines up pretty nicely with with general code quality with what people are self reporting so that's kind of cool what that was neat the bottom half does not line up at all in the O'Reilly study we had only 11% that that reported fixing less than 20% of the issues we

obviously saw more than that we saw people introducing more issues and we have actually it's like 50 percent of the sample had less than 20 percent fix no change or more flaws so the bottom end doesn't match up that might be one of those things that self reporting you say that you're better than [Music] or or it's possible that security has a higher buried entry and that this is actually where we start to see that you know we do have some room to tip for education and for help I'm from being able to get in there and make a difference and it's gonna be on that low end that doesn't quite match up again I suspect it's the mixture effect

it's probably a little bit of our low self reporting as well as a as well as just the fact that these plots can be tricky like when you get these sequel injection they're not always straightforward sometimes it's just okay make a prepared statement stick that parameter into a prepared statement in the way you go sometimes it's more complicated than that sometimes the data is coming from the sources and you really just care about where it's coming from you want to make sure because it's a white list or what have you and depending on how its set up and what the design is so that I'm not sure at that point where the the difference is

there but it might be a combination and and so everything does anything affect fix rates so this isn't directly from the data I was looking at here this is from the state of software security and we did see that there were two things that we track one is what I do the remediation coaching so that's big having a helpful ham somebody in the security team sterve understands what's going on who's familiar with code and can get in there and help people understand what the flaw is why it's being reported why you should care about it and then what you can do about it it's that full range it's not just here's how to fix it it's also hey why

do you care about this where does that data really come from and then the other part was elearning subscription so an account that head e-learning also had a faster Ryota fix so that was just having some information we were using that as a proxy for some information with security and having some developer knowledge of what security flaws or in how to fix them and it's like we're helping so what does look like we're making a difference so we're seeing that flaws get introduced sort of no matter what but when we're giving those that information to developers they're they're fixing them again maybe not fast as we want but they're fixing them and when they have

that access to those consequence-free scans to that sandbox we're seeing that they're getting to them before security even really steps in so they're able to sort of do a lot of self help and then be able to then bring the really hard parts to security where we can work in conjunction with them to allow everybody to be successful so we can have that collaborative relationship we can not be necessarily just gatekeepers but also people there to help push things forward

and I have rush like this I apologize but um but so but me they learn for questions but so which is why I have some sympathy flaws happen you know these are not just flaws because Avella / doesn't care whether you're not showing up because your developers are special this is normal and really where the the difference is is going to be in the fixing and non the finding you're going to find flaws this isn't something that you've punished over it's something that you've worked collaboratively together to try and get better and cuz devs will work to fix findings and it'll do even better when you help out and then just if you look for flaws I think

that's the the big one the other part from the O'Reilly study that actually came out not in that paper but in another article was that only 25% of organizations for reporting running static analysis at all which means that a lot of people are not looking sufficiently and people were self reporting that they didn't feel that they were doing enough to even look we know the flaws are there we know that if they're found and you have the information and tools to fix them that they get fixed so be really cool people look for them yeah and I Paulie oh that was really fast but I love questions yeah oh and there's gonna be a microphone does that other can we just

be like really bad with the lovely

you're really official here cheating so I feel like full disclosure I'm a developer yeah and I've I spent a lot of time helping developers solve this kind of problem SQL injection specifically I'm honestly a little astounded at how low thirty percent is it's higher I think that's astoundingly well I think it's way higher yeah you're probably right this is across all of our application base not just applications that you use sequel yeah at double mic action so yeah so I probably probably has higher that's sort of one of those I'd like to know that too there was a really cool and different project that looked at PHP fine it just looked at PHP samples from

Stack Overflow over the last six years and it looked and it starts up really slow like there's not very many of them and it goes really high but if you look at it as a percentage of questions asked on sequel it was fifty five percent and it was fifty five percent the entire time I think there was a little wiggle because it went for a really small sample size to a big one yeah I suspect it's higher I mean you get it I'm just gonna do my own evilly I can tell you looking I'll be brutally honest and tack on to that I'm looking watching your presentation I went from wanting to rage quit to being completely

mind blown that you know subscriptions read learning subscriptions really help yeah I mean how is that possible I mean that's that I mean there's been stuff about this stuff we have me above sequel injection out there for years and they've made that big of a difference I'm like it's not it's not on the introduction rating to probably has more information on this because he's [Laughter] conveniently in the room but I'm probably screwing up the TV thing and not a question but our response one of the things that we learned as we started to do the studies that serious citing here is that there's a lot of different things that organizations that are actually serious about educating

developers and fixing vulnerabilities can do to to make you know things better and eLearning you know actually offering developers like formal training Eva to see if it's CBT is one of them but that also may be a proxy for other things that they're doing like putting it in developers goals or you know or you know or something else like that so I one of the cautions that we always have about the data in studies like this is we're measuring what we can measure we can't actually measure that you know Bank XYZ has their developers gold on application security because we don't see that that that data right I think in there yeah I'm so Mike there was an extra man

[Music] did you uh did you ever break this down by language because I'm thinking like for instance dotnet like the MVC they've kind of mitigated some of that some of the problems you can have a sequel injection so again I kind of live in this data right I'm about a large number of the white man tattoo is done MVC and and using sometimes it's using old parts the framework and like so using legacy systems and sometimes it's just getting around the things that are there to try and protect you so maybe um the language data we have and was that I had access to I didn't completely trust because this is you know coming from our

database so it's being pulled out of our customers skin information and sometimes something some of those flags aren't the way that they're recorded is a little because you can scan multiple languages in one in one scan and some in that flag when I float it only has one language so I know it's not perfect and so I didn't want to do that I'm divided just in case it wasn't but yeah I know that there is variance by language I don't exactly know which direction but I don't thought and I him I don't think was um beautiful and perfect okay so it's a little bit of background I've gone back to my old University and had polite conversations

about the curriculums and I I would say that wouldn't think that you should do if you continue with this is break it down by wind the new hires from universities come in and see if you see like spikes that last because quite honestly um people say in this room Oh sequel injection has been around for years yes but the people coming on board this is the first thing they've been heard of the concept of sequel injection often times this job is the first job they even heard of databases so the other my other my other rage well in that rage point it's just one of those things I've come to the point of my life of polite resignation

because it's a whole lot better in the heart is that so often times you see yet a new framework come out that talks about how easy it is to create applications and in the security part of that framework is non-existent or shows up later as an optional package so and and I think they'll be interesting if you could slice your data like on does this application have those libraries that represent security on and enabled and see what happens to those numbers that that would also be awesome but it turns out that's really hard to see does that but so if you wouldn't mind if I ask you're doing development right now so it's a very simple question how did

you learn like things like equal injection and cross-site scripting and all the application security stuff if you wouldn't mind if you wouldn't mind if I am and not at all for for me it was just personal interest I'm kind of overly pedantic and so sequel injection is one of those places where it pays off to be overly pedantic and so it was something that I was always interested in and wanted to talk anyone's ear off the college so I'm not a good case study for that I had a hard time getting anyone interested in actually solving the problem and the ones that did get interested I think I think probably hit that 30 percent mark more often that the

30% mark I would say would was further people that were interested okay thank you very much [Applause]