How SecOps Can Convince DevOps To Believe In The Bogeyman

[Applause]

hello all right hey P sides uh welcome to day two I'm here to talk about how security can convince devops to believe in the boogeyman thank you for the intro um I'm leaf dler formerly an application security engineer at RedPin I would do third-party application security engagements walk you through remediation strategies retest your application once you think you fixed something and occasionally I do developer training so from an application security perspective basically anything that I could do to help you make your application more secure that was my job I recently moved over to bug crowd which is your elastic security team so sales pitch slide what is bug Crow actually do we incorporate up to

16,000 freelance researchers as part of a public or private engagement we can help you run a crowdsource pent test which replaces an existing pent test of one or two people with one of 25 100 200 you name it that will be your pent test for those couple of weeks we can also help you manage an ongoing bug Bounty program so most of you are probably familiar with bug Bounty programs but in case you aren't um here's a brief introduction bug Bounty programs provide a symbiotic relationship between researchers and companies it allows researchers to look for vulnerabilities and get rewards without risking prosecution it also allows companies to be alerted to security issues and fix them before a malicious actor has a

chance to exploit them if somebody comes in tells you about a security problem and you can fix it before somebody exploits that and you pay them a little bit of money that should definitely be considered a security win for your team the first bug Bounty program actually got started about 20 years ago with net ape now hundreds of companies run bug Bounty programs here's kind of a a rough timeline there's definitely a ton of people that have been left out um but bug bug crowd got started in 2012 along along with a couple of our other competitors and kind of started the space of bug Bounty as a service helping people get up and running with bug

Bounty programs at the end of two 2013 Microsoft joined the party which kind of signaled the end of it just being for mostly B Toc tech companies and showed that pretty much anybody could run a bug Bounty program program successfully so here's the marketing slide uh these are some of the brands that trust bug crowd so what are some of the things that we're going to cover today I'm going to give you a brief introduction to devops I'm going to talk about how to incorporate security into the devops life cycle going to talk about accelerating your return on your security investment and then most importantly what's in it for me as a security person so how did we get to the point of

talking about devops SEC we need to adapt previously there was companies had a single corporate Network and from a security perspective you just built a giant fence around it applications were hosted internally and software release Cycles were in terms of months if with a big fence if somebody was trying to climb over crawl under or tunnel through um you knew you could just blast them out of view Network this model was reasonable at the time but it doesn't really work anymore with the proliferation of cloud computing BYOD SAS your business data is everywhere with the introduction of Agile development and distributed architectures um your business is becoming more complex and evolving faster than ever most development

Sprints have dropped to two weeks and the idea of a version is completely changed production code changes are being made within minutes or hours if you find a problem you can deploy a hot fix within minutes assuming that's something that's easy to fix um you used to buy Microsoft Office 2010 um 2013 and then whatever the next version was now you just buy Microsoft Office 365 you never worry about installing another version but as a side effect a lot of your data is out there in the cloud it's not within your corporate Network so it's impossible to build a big fence around it your data is being hosted by Google Microsoft Dropbox whoever else is a s service provider and

at some point it's probably ending up on Amazon web services so the new fence is moving security as close to the code as possible we've seen something similar with with Dev Ops creating Dev Dev and Ops to create Dev Ops bring configuration and code changes as close of development as possible to decrease that feedback loop so that if you see a problem you can fix it as quick as possible dev's all about Innovation building new features creating applications and they're generally pressured to meet these deadlines by sales and marketing as a result security and sometimes performance seem to take a hit and they're generally can be neglected as part of the build process on the other hand you have your Ops Team

they want to make sure that nothing breaks systems don't go down customers stay happy this generally translates to something along the lines of change as little as possible or don't change anything as long as things are still working when you combine these two you get devops which is core to the Agile development methodology it allows you to deploy code quickly or even continuously increase the speed of release cycles and reduce the time to fix bugs it also allows you to reduce the tension between stability and new features because you know that if you make a mistake you can backpedal release a previous version or Implement hot fixes depending on the severity you might be able to make

mistakes faster but you can also fix them just as quickly this can be a double-edged sword this is a sword from RuneScape which is a game I'm sure some of you guys played and I certainly sunk a number of hours too um back in high school the devops emphasis on rapid change can be a breed ground for bugs and security vulnerabilities as code is written faster more vulnerabilities can make it into production because you have less time in between the code being written and the code going live so how do we fix this we introduced security into the process similar to moving Dev and Ops closer together to create devops we're going to move security closer to devops to provide

increased protection to code and data we're going to identify problems in the code and configurations as quickly as possible and relay this information back to the dev and Ops teams to make changes typically Builders and Breakers have been seen as adversaries and and being different and I'm not going to try and convince you otherwise they are different and that's good you're incentivized to be the yin and the Yang that's what you're paid to do the competition and strain is a critical value to the Builder and breaker relationship but how do we get these teams working together when all three are incentivized in different ways which can cause them to act as adversaries this is going to a little

bit difficult to navigate they're not going to fit perfectly but you can get pretty close developers are incentivized um to build new features and and push applications Ops is incentivized to maintain stability that means keeping the Train on the tracks and make sure that it's moving in the right direction security wants to make sure that nobody does anything that lets the bad guys in as a side note security people AKA good guys that think like bad guys tend to overestimate the ability of everyone else to think like a bad guy this doesn't make security people better than everyone else but it does make them useful and potentially annoying uh depending on if you want to listen to

them or not as a security person the next time you feel like calling a developer dumb for making a security mistake try and manage a whole technology stack and build an Enterprise application and if you can do all three of those by yourself without making any security mistakes um then you're probably way smarter and you could probably call people dumb but that's pretty unlikely so one of the problems as a developer is that you think that the threats that the security team is talking about are overblown the operations problem is that Dev always wants to be pushing new features and security wants the latest stable version of everything the day after it comes out or even the day

before it comes out they read the open SSL patch notes they say I want this patch yesterday the security problem is that you feel like you don't have the time the energy the resources or potentially the people skills to convince the other teams that the boogeyman is real and a big thanks to this ghost is some security vendors that have gotten people accustomed to the message of the sky is falling followed by a sales pitch um people tend to not take these kind of things seriously when you convince them of a problem and then say give me money fear and certainty and doubt shouldn't be the only way that you educate people about security problems it's a great way

to start if people have no knowledge of security threats at all but over time um people just become immune to it and stop caring so what are some things that we've tried to do to make sure that code gets pushed to production and doesn't have security flaws um we've got a bunch of different things in place but if you're a developer and your main goal is to get features built and you're not concerned as much about security um these are all kind of just seen as blockers they're things that slow you down at least to some extent so as a developer neglecting security you're kind of like somebody standing next to the edge of a lake Lake and not

realizing that somebody could push you in um so security pushes you in um we all share a laugh and we hope that you'll take security more seriously next time and that might work a couple times but it's really not sustainable in the long term at some point it becomes the boy who cried wolf because people keep hearing that they should care about security but at the same time there haven't been any incidents within your organization so it seems like it's overblown maybe developers are better than security realizes maybe they just haven't been targeted or maybe security is redlining themselves all the time to make sure that things stay safe none of these are sustainable in the long run

developers need to be taken seriously by security people if they have the prowess um you will get targeted at some point and it's not sustainable to have security working overtime all the all the time so how do you integrate security into devops well the secret is start simple take small steps and leverage easy wins if you try and do everything at once you're probably probably going to fail the first step is getting developers to care about the security of every line of code that they write developers job isn't just to write code it's to write running software that works well is reliable and is secure so you have to educate developers about the security implications of the

code that they write both internal and external developer training is great external developer training is great to give people a baseline you bring somebody in you schedule time on everybody's calendar um they go through a bunch of different types of security abilities how you identify them things like that and it's great as a way to just get that base of knowledge so that as a security person when you go back to your developers and you start saying cross-site request forgery token they know what you're talking about I think it's more important though to have internal security training that's happening on a daily and continuous basis that's just completely informal it's not something that you plan for

it's something that is that comes up naturally as security threats make their way towards your organization so if you see a security problem in the the code you go to the development team that wrote that and you don't just say hey bad job you made the security problem please fix it um you say this is how I identified the issue this is why it's bad this is what would happen if somebody else found the issue and this is how you fix it and this is how you can check to make sure that it's fixed um when you provide that holistic training and not just tell people hey you did something wrong um they're going to take your advice a lot more seriously

and it gets everybody to care about the process building a su successful product should be everybody's baby not just developers each team might be incentivized in slightly different ways um Dev wants to build things Ops wants to maintain stability and your develop and your security team wants to to break things but these teams aren't isolated they need to care about the goals of all of the other teams a good way to do this is to bring everybody into the same team by diversifying your scrum team bring somebody from the Ops Team and somebody from the security team into your scrum team and just let them sit in and listen and give advice device as necessary the next step is to implement

a firefighting team on a weekly rolling schedule members of the dev Ops and security team will work together to solve problems any issues that come up and they could be security issues they could be stability issues with the production uh server stack they could be UI or ux issues they could really be anything dealing with the product that one of those teams would normally deal with um as these teams work together they'll learn how the other teams work and how they handle these problems they'll see the natural overlap and be more understanding of the other team's needs decreasing the friction will help these teams be more comfortable and approaching each other if the teams aren't comfortable approaching each

other it's almost as if the other teams don't exist so what makes the security process so hard to manage a lot of it comes because between the imbalance of the number of developers and the number of security people the size of your security team is typically a lot smaller than the size of your devops team another thing is the security team might have responsibilities that spread significantly across the app the organization developers might have different jobs you might be a front-end developer a backend developer you might be part of the QA team but generally you're all concerned with building the product security typically has a lot of other responsibilities that might include physical security protecting the

staff from fishing which is pretty much impossible um monitoring or scanning server architecture or infrastructure responding to ongoing threats so are we getting hacked right now uh hopefully not and then potentially reviewing lines of code that the developers wrote for that day it's an unfair fight for security teams when tasked with protecting an an entire application from both the code and infrastructure levels how can we make their lives easier well with most things you improve the process similar to the QA process the security process is going to be a combination of automated and manual testing tools and methodologies first it starts with peer revie peer code reviews on your poll requests code that that's complex and

hard to read is hard to review and ultimately hard to fix hard code that's hard to fix makes people worry when they go and make changes so if you have find a security problem in the code and nobody can figure out how that code was written and works in the first place it's going to increase that time it takes you to fix that code and get that deployed to production once your development team becomes more security aware they also may be able to determine that there's a security problem in the code that somebody wrote before it even goes out which is great the next step is adding security tests into your test driven development you

can do basic checks that a lot of scanners would do inside of your unit tests these can be useful for testing things like making sure that input validation is in place checking for output en coding and various misconfigurations like cross-site request forgery tokens not working correctly you can also Implement automated dynamic or static code analysis tools that'll allow you to catch certain amount of lwh hanging fruit before it goes out into production these tools aren't going to catch everything but anything that can be caught automatically without somebody actually spending time looking at something um is definitely a win testing creates a precedent for stability and quality and integrating security into that same process creates a precedent

for minimizing vulnerabilities once your app is in pretty good shape you can start taking proactive controls to minifi minimiz security vulnerabilities some of these can be combined into an application layer intrusion detection system here are some examples of things that you might build into an app layer IDs checking for serers side input validation failure when the client side validation exists so we all know that you need to have server side validation to protect like things from um crossy scripting but why is the client side validation not catching something obviously somebody's trying to mess with you if your username field prohibits the use of a less than character a normal user isn't going to be able to get a

less than character into that field but anybody with a basic knowledge of burp Suite or zap could do that if you have a client side check in place that should be failing but it's only failing on the server side you should probably flag that person as somebody is's trying to mess with your application you can do similar things um with that for for validation of hidden Fields radio buttons drop- down menus or anything that has a pre-populated list so if you have a drop- down menu and you're expecting it to be a United stat states name Texas California New York and you're getting some sort of other value you know that somebody didn't select that from the drop- down menu it's just

not possible for a normal user under normal circumstances to select something that's not a state so when you see somebody doing that you should start watching a little bit more closely you can also check for just blatant SQL injection or cross-site scripting a lot of people when they start assessing your application are going to assume that something like this is not in place so if you see something like script alert one wait for delays all the basic scanner attacks just start flagging those people as well a be the best way to build these kind of tools is to use the tools that people are using to attack you to create a defense in this case the best defense is

really a great offense so using tools like dirbuster various cross sight scripting lists and start building in checks to look for things that come up from dur Buster so people browsing to admin login PHP or CGI bin or things that don't have a use within your application um but that scanners are going to look for just create those Honeypot uis and no normal user is going to browse to these because they don't exist in your application through the normal flow when you see people start hitting these you can start trying to ban them or you know take just look at them cautiously another way that you can start removing security vulnerabilities um without increasing the load too much

on your internal team is leveling the playing field with crowd sourcing offer a reward for freelance researchers if they're the first to find an issue in your your codebase or in your application automation doesn't replace having people look at your application because people are the new automation scanners can only go so far we know what they struggle with they're not very good at business logic flaws privilege traversal privilege escalation or Advanced attack vectors that require you to chain different parts of the application to get one good exploit working they also really can't pivot that well so they might be able to identify initial Ingress point but they're not going to be able to find all

the best issues they're great for finding reflected crosslite scripting SSL issues outdated versions and other vulnerabilities so you should definitely still use them but you really need people to take a look at your application because we know that humans are much smarter and they're the ones that actually write reports and Report breaches and with crowd sourcing it's now effective to use these types of people at scale compare this to me working 40 or 80 hours on an assessment at my previous job to 50 researchers that can each spend two or three hours there's just no way that I'm going to be able to compete with 50 or 100 people that can all spend a little bit of time working on

something this also frees up time for your internal security resources instead of having five people all going through your codebase and looking for issues just let the crowd go through your application and have those five people do a little bit of code review but spend most of their time looking at valid issues that are submitted by 100 a th000 or even 10,000 different people all across the world here's an example of a bug crowd customer that had been running an expensive web app scanner for a long time um you start out getting some good results over time as you fix things um your developers get a little bit more security Savvy and after a while you

just start getting clean results within 24 hours of running a program against the crowd somebody gained super admin access through a chained attack this was something that wasn't picked up by the scanner and that they had no idea it could even be possible from an external adversary they thought they were doing a good job writing code because their feedback loop wasn't smart enough to find issues your internal team isn't going to be able to find everything so the tools and the methods that they use to find security issues are the best way that they can learn about new problems in their code and fix them before somebody actually exploits them you should never assume that your code is

secure you should actually just assume that it's broken and let somebody test it find issues and try and fix as many of them as you can um once your code base reaches a certain size it's pretty much impossible for it to have no security problems but you just want to minimize them in both uh number of issues and the severity responding to to problems on a semi or a regular basis trains your fire team to respond and involve other larger teams as necessary this provides great training for your fire team because instead of just having issues that are coming from the internal team they're also going to be responding the external issues what happens when there's an

actual external issue where somebody is actively exploiting application your fire team will have kind of dealt with this already because they're going to be used to getting external reports um hopefully the reports that get them train for this is somebody who's friendly and is not going to actually exploit anything and that will provide great training another problem with traditional pen testing is it's just not people just don't use it frequently enough if you're getting one pen test a year and finding five to 10 vulnerabilities that's not going to make a team concerned about security year round it's something that's going to pop up once and then kind of go away for a while we ran a crowdsource pentest for a

company called instructure and they have a a learning management system called canvas and they found eight times the number of vulnerabilities that they had found in previous pent tests and these were pent tests that had been done by respectable companies for the last three or four years there were issues that had been missed by previous pentests and it's just impossible to find everything as a pentester um you can find some great stuff but there's probably something that you're going to leave behind so having more people look at it um increases the chance that somebody will find it having a lot of bugs is some of the best security training that your developers can get by integrating

crowdsource security into your build process and constantly getting feedback on your alpha or beta or pre pre-released candidates researchers worldwide will provide security feedback and allow your devops and security teams to remediate vulnerabilities as you start remediating vulnerabilities over time start tracking them figure out what types of security vulnerabilities or what parts of the application you're specifically you're especially weaken is there a certain part of the application that's just riddled with all kinds of security vulnerabilities or overall the apps pretty good except for reflected xss taking note of these things and aggregating them over time is going to help you focus areas of training for future training sessions and there you're it's something that you want to start tracking early

because there's always going to be bugs and people are already looking for these bugs if you give them permission they're going to look for it and they're hopefully they're going to disclose it for you but even if you don't give people permission people are already looking whether you have a responsible disclosure program or not so let's head them off at the PA and start a bug Bounty program because a lot of people are just going to send you details because they want to be helpful um we've heard of people that are submitting vulnerabilities through Facebook through security at through Twitter because they can't get a hold of somebody and they want to be helpful and they don't want

the company to be exploited they're just trying to reach out and genuinely help this company be more secure but not everybody's that way there are people out there that want to do so but only in exchange for money which is fine if the company's running a bug Bounty program and they've agreed to pay for it but if they don't have a responsible disclosure program or they say it's a program where we're just going to say thank you or send you swag or something like that um it's really not cool to to extort a company because you have a security vulnerability so one of the ways that you can stop this from happening is people are less likely to make demands

for money if you have an established program where they could already submit the same vulnerability and get paid for it pretty much every bug Bounty program works in a way where only the first person to submit a vulnerability gets paid for it so if somebody finds something they know that somebody else could have found it too they want to submit it for whatever the range is that you've accepted and they know that the chance of somebody else finding it is pretty high um in a lot of the programs that we run we see a ton of duplicates so most of the time you're not the first you're not the first person to find something here's an anecdote about an

actual extortion attempt um that was a prospect of ours they came to us and said hey we've got somebody from Eastern Europe they said give us €3,000 we have a critical vulnerability in your software and we're going to exploit it if you don't pay us this is not a great position to be in because either you need to get that information and and fix it before they can exploit it or you can call their Bluff and say either they don't have an issue or they wouldn't actually follow through with the attack neither of those are really where you want to be as a company so we came up with the idea of starting a bug Bounty

program and inviting them to it what we didn't tell them is they were the only person that got invited they were literally competing against themselves within 15 minutes they had submitted the vulnerability and the company did pay them out for it but they were able to pay them out for a rate that the company thought was fair based on the criticality um based on the criticality of the vulnerability after they took a look at it it was lower than what the attacker made it seem like and so they still paid them out they honored their agreement but they were able to assess it in a way that was fair to both the the the hacker in this case and to the

company um the company also learned a valuable lesson that this kind of thing could actually happened to them they stabilized the situation they got the bug fixed but they realized that the boogeyman is real they're now running an ongoing bug Bounty program with us because they don't want to be in the situation again they want responsible researchers from across the world to submit vulnerabilities and get paid for them for playing by the rules and being helpful they don't want to be ex extorted by people that found issues and decided that they should get paid a certain amount for them and this company does run a pretty lucrative bug Bounty program um which is actually gets them very good results

like I mentioned earlier 2010 was when bug Bounty programs really started to take off Google and Facebook created very loud and public bug Bounty programs and have paid out a ton of money to researchers which is awesome researchers really do get really do deserve to get paid well if you find a critical vulnerability in software that a ton of people are using because that isn't something that anybody can do it's actually a pretty rare skill set for somebody to be able to find a really good vulnerability in widely used software and whatever you're paying them is certainly less than it would be if they were to use this vulnerability in an irresponsible way or sell it on the

black or gray market for these types of things so I'm definitely in favor of rewarding researchers and paying them what they deserve based on the impact of the vulnerability as we saw kind of an uptick in PE sorry as we oh sorry um as we saw an uptick in the adoption of bug Bounty programs um companies like bug crowd joined to help facilitate that process um companies like Google and Facebook have a ton of resources available to run a bug Bounty program but not everybody has that ability and you want to run a bug Bounty program because bug Bounty programs are pretty awesome um they change the model from one based on efforts to results

because you're only paying out for valid vulnerabilities it also allows you to reduce the load on your internal security team because they can just go go through the valid submissions they don't actually have to spend time looking for all the different issues it's also pretty badass for your Dev Ops and security teams to be able to make the statement of come and hack us we think we're pretty secure um we know there's always going to be problems and you deserve to get paid for them and we'd love to hear about them but it's not just about being cost- effective or loud running a bug Bounty program is a bold statement and a well-run program is

a great way to show people that you take security seriously crowdsourcing has been proven across many Industries as a lowcost and costeffective way to get a desired outcome but is more than that it's also about leveling the playing field the economics of how we do security testing and defense are fundament fundamentally disadvantage to your disadvantage as a Defender if you think about what the attacker space look like there's different actors motivations and they have lots of time and lots of potential targets they get their gold by actually exploiting something not the time that they spend they put on their suit they put on their ski mask and they look for vulnerabilities this leads you to the

defender's Dilemma to be successful as a Defender what do you have to do you have to protect every asset every day from now until Infinity which is pretty much impossible as an attacker all you have to do is exploit something on one day and you've won for that day so you know I've been talking about bug bounties are so great crowdsource security is so great why isn't everybody running these things um it's because running a bug Bounty is hard when you start a bug Bounty you're going to get a lot of great information from people that genuinely want to help your organization be more secure but you're also opening up a fire hose of information from all around the world

there's a bunch of people that just want to get paid they want to get recognition without actually doing anything and this can be overwhelming if you're not ready for it so you got to plan ahead how do you run a successful program the mistake that pretty much everybody makes is they think that most of their time running a bug Bounty program is going to be spent on remediation I got done telling you um you know a couple slides ago that you're you're going to get eight times the amount of vulnerabilities as your previous pent test that sounds like a lot of work but in comparison to the number of invalid submissions you're going to get it's not

very much for every payout that you say is a valid vulnerability and is worth fixing um there's going to be about nine duplicates or in valid submissions so multiply however many you got from your last pent test will be conservative and say five that's how many valid vulnerabilities you're going to get and then multiply that number by almost 10 and that's how many total submissions you're going to get and you can't just ignore these submissions these are people that took time to write up a response look for an issue and just because it's a duplicate they they still spend a bunch of their time and effort writing these things up and they deserve a response and all those responses take

time even if you're using kind of canned responses it still takes a lot of effort to go through and say okay was this in scope yes or no is this something that we care about yes or no has somebody else submitted this yes or no and then responding to all these different people based on that kind of Choose Your Own Adventure of of vulnerability coming into your into your platform you're also potentially you also could get overwhelmed by just the number of valid submissions so make sure that initially you do have the resources available to deal with this um we had a a company that's in digital advertising and they wanted to assess the security

of their code base within 24 hours they had so many valid submissions that they just had to throw in the towel they're like this is great we love the work that you guys did we're glad that we know about these issues because we can fix them um but we need to take a timeout and actually deal with these and we'll come back and uh we'll let the crowd have another crack at it once we fix all this easy stuff another problem that a lot of people run into is that you are going to get out of scope submissions um you might have the best written Bounty brief in the world you might have the worst

people are still potentially going to go outside of the scope it might be on accident it might be on purpose but if you look at a submission and it's out of scope but valuable to you and you make a change based on that you should still pay out the researcher the golden rule is if you touch the code you should pay the bug um that doesn't mean that you should encourage this type of behavior and really you should do the opposite you should send a message say hey we are going to pay you out for this vulnerability it is helpful to us it is something that we care about but in the future please play Within the rules and

this will help your organization and it'll help any other organization that's also running a bug Bounty program because by convincing researchers to play by the rules this helps the organization and the researchers align expectations so a lot of this does fall within the within the organization's realm of things that they need to do um but kind of curating a list of well behaved and intelligent researchers is something that anybody in the bug Bounty space can help do so some of the important points for this is make sure that you give researchers what they need to be successful share with them any known issues or areas of the application that are completely just riddled with security vulnerabilities that you

already know about and don't want to pay out for as a organization you don't want to pay out for vulnerabilities that you already know about but researchers don't know what you know nobody's expecting you to open up your your hosted J solution to these researchers and say hey go through if a bug's already in there don't submit it that would be completely ridiculous and would be exposing way too much information to the outside world you don't need to take that kind of a stance but if you can share with them hey um you know we haven't gotten csrf tokens working correctly or we know we're vulnerable to xxc or something like that just let the

researchers know as much information as you can so they don't spend time identifying and writing up findings and spending time writing reports only to get the reply of hey we already knew about this sorry we're not going to pay you another thing that can be helpful is make it as easy as possible for researchers to sign up for your platform if you need to make small changes to whatever environment that they're testing in to allow them to do automatic signups or respond quickly to provisioning emails that they need to get from you to get signed up on your platform all that's going to be helpful to get to getting as many people looking at your platform as possible you want to

make the barrier to entry for them to look for your for vulnerabilities in your software as low as possible most importantly though you want to make it make it clear how much researchers will be compensated we frequently see clients start out with relatively low reward ranges so typically you'll have a low reward range of something like $25 and a high end of $250 or $500 because you just don't know how many problems there are with your code when you start you you can't afford to be paying out 10 grand like Google every time somebody finds a critical vulnerability if you're a small company you need to be realistic about what your security budget is and so you might say okay well

if somebody finds a really good vulnerability that should be worth a lot of money and it is but if you have to go back and get more budget because you ran through your your bug Bounty budget like that it's not going to benefit the researchers it's going to benefit one researcher and a bunch of these other people that have submitted vulnerabilities are either going to get paid out at a lower rate or they're not going to get paid out at all and that's really not fair to researchers so it's better to start with a lower reward range and make sure that everyone gets paid out at the amount that you agreed to than it is to start high run out of

money and then screw over researchers another thing that we see is um bug bounties are still cool enough where if you have a big reward range um some news is probably going to pick up the fact that you're running a bug Bounty program with an upper end of 10 grand 20 grand whatever um but if you change that 6 months later once the Press dies is down that's really saying that that the time that you were spending before is not as valuable as the time that you're spending now which isn't cool either you want the reward range to go in One Direction and that's up if you start low and with the hopes of clearing out a bunch of lwh hanging

fruit that's great a bunch of researchers are getting paid you're getting a better product and then over time as you get more comfortable and your internal security team um increases their security prowess then up that upper end to get researchers to come back and look for more issues when you've been running a bug Bounty program for for 6 months you know double the upper end payout and then at a year go ahead and double it again or something like that as a company the most important thing you can do when running a bug Browning program is so show the research Community we think you're valuable and we want to show our appreciation for your contributions over

time and we want them to go up so if we've had you looking at this for a year and you find a new issue that was hard to find before we want to pay you out more than we would have at the beginning not the opposite once your program gets kicked off your internal security team will see what kind of vulnerabilities attackers are able to find with little to no knowledge of your organization that's a lot different than an issue that gets reported by your internal security team that has an intimate knowledge of your architecture and codebase an external attacker is disadvantaged compared to an internal adversary AKA somebody part of your security team that is looking for

security problems um because they don't have that same insight knowledge the next thought will be well if this guy figured it out and he reported it who else could have figured it out and would they have reported it um bug bounties allow you to leverage the severity of a real security incident to get things done without the fuss of an actual security problem the captain of your fire team will appreciate just running a bunch of fire drills without having an actual fire to put out another example from a bug crowd customer is they were they're an online Marketplace place they ran a a Time boxed um crowdsource pentest and one of their favorite Parts besides learning

about a bunch of security vulnerabilities that they didn't know about was having the security and devops teams be able to look at vulnerabilities as they came in in real time they thought it was incredibly powerful to look at what people were submitting how they were looking at the application and from coming from a background of people that weren't security-minded they were learning all sorts of stuff about how people attack applications what kind of abilities can cause problems and it just gives you generally great insight into how good guys that think like bad guys work when you start a program you're always going to see a bump in the beginning there's a new level of resources effort and skill available to

provide security feedback for your application when you open up to the world that had that this wasn't available before and you're going to flush out a ton of security issues this goes back to starting with a reasonable reward pool in the beginning to make sure that you can handle this bandwidth of submissions and make sure that people are getting paid out at the rate that you agreed to then you start to see things tail off people start fixing a bunch of stuff and um the developers start learning from their mistakes then you see a slight increase and level off as researchers become more familiar with the code and as new vulnerabilities are introduced finally and this is the most important part is

you see things taper off as developers learn from attackers and start to think like the bad guys if you can get your development team to start thinking like security people you're going to be in a really good position as vulnerability reports come in it changes the way that developers think and improves their security prowess developers are generally pretty smart people and they really don't like making the mistake same mistake twice so if they get burned by a security problem they're going to remember that next time and they're going to make sure that they don't make that same mistake we see this pattern with almost every client that launches a bug Bounty program so how tight can you make that feedback

loop between devop SEC and the hackers nothing is closer than a head-to-head competition you've got your Dev Ops and SE Team all on the same page you're feeling pretty confident they're like okay we've been running a bug Bounty program for a while um we're responding to issues quickly and we haven't had a major security incident in a while let's try a gamified pentest so what you do is you create a pool that benefits your whole engineering team just make it big make it something that they want to go out and do um you know buy a bar tell them to invite their plus ones have a party go to an event whatever then replace the an existing pentest with a

time box bug Bounty program pay out money from that reward pool and whatever the hackers don't get the devop SEC team gets to keep and spend on whatever they agreed to in the beginning so that really incentivizes people to write good code make sure that the Ops team is patch in anything that is an external library or program and it's making sure that the security team is reviewing things as as best that they can and fixing any known issues because you know if something's a known issue in this instance you're still going to have to pay out because if you didn't fix it and you knew about it you definitely don't get to deserve to use that for your

party great things happen when you can tighten the security feedback loop between your engineers and what they consider to be the outside world so in conclusion bug bounties are cost effective and highly marketable but that's not the full story when you get a bug from outside the organization that hits the desk of the team that wrote that feature their mind will forever be changed in terms of what in terms of the way that they view security it allows your team to see the potential impact of an external attacker without the actual damage it allows people that have historically been Builders to see how Breakers think and most importantly it gets devops to believe in and defeat the

boogeyman all right question

questions so the question was um as far as touch the code pay the bug would you extend that to Partners so um let me clarify a little bit so basically if you're using something within your application that somebody identifies an issue in um but it's something that's managed by a third party should you as an organization be paying out for it is that the okay um I mean I personally think that those things should be excluded from the Bounty brief and like I said you know people will go out of scope um typically what we meant what I meant by that would be they went out of scope but we're still attacking something that would be considered to be

your resources um I in an ideal world you would forward that bug onto whatever that third party was and they would pay out for it um obviously that's not going to happen every time um and you know depending on what your budget is and what kind of organization you are I would be in favor of paying out for it if you can afford it but really take a hard line of be like you know we really appreciate that you found this bug in software that we depend on but this is like a one-time deal if you want to submit vulnerabilities to other people's software you're going to need to go through them um but we'll pay you out

for this one because it's something that we take seriously and we think security is cool I

think all right so the question is how do you recommend interacting with QA so um in in what

circumstance

gotcha so I think that anybody that you can make security-minded that's part of the build process is going to be beneficial so if you can get your development team your QA team your Ops Team and your security Team all security-minded and they're all looking for security problems and QA is running some amount of security testing as part of their process as long as you have the bandwidth to handle that on your QA team I think that it's a a benefit anybody

else I have information on what the average is for crowdsourced but not for traditional because that would require a bunch of people to be giving up those analy Antics um we typically so we rate things on a scale of priority 1 through five and we typically see things somewhere slightly below the P3 range so as the average so in between P3 and P4 um but we typically do see a handful of submissions that are in the P2 or P1 range one so P Priority One would be the most critical so we usually see at least a couple bugs that fall into the P1 or P2 category but most of the bugs do fall within the P3 or P4

category um so we kind of when we do a crowd source so it I guess that would be mostly directed oh I should repeat the question so the question was um do organizations fear that when you're engaging a crowd of researchers versus a typical um one or two pent testers that you're going to have a lower criticality of bugs that get submitted so are you just going to get a bunch of low priority bugs versus having a couple bugs that you actually care about and I think that this is probably most comparable to our time boxed pentest versus an ongoing bug Bounty program because in an ongoing bug Bounty program if you submit a more critical

vulnerability you're going to get paid out a lot more we have reward ranges that are anywhere from 100 to 5,000 so if you submit a P4 you're getting a 100 bucks if you submit a P1 you're getting 5,000 there's definitely an incentive in that instance for somebody to spend time looking for a P1 bug um we've also specific this is bug crowd specific so for our Flex program which is our replacement for a pentest um we do incentivize the most critical vulnerabilities so people who find the first second and third place critical vs get paid out at a much higher rate than anybody left over and to get invited to the lucrative private programs we do

curate a list of of researchers that have shown their ability to find those types of vulnerabilities in the past so within our platform as you find vulnerabilities or even if you find duplicates of good vulnerabilities your rank starts to increase which shows that you have the ability to identify high level issues over

time all right if there's no other questions I'm GNA [Applause] leave