Bug Bounty Hunters: Lessons From Darth Vader

All right, we're gonna get rolling. So, bug bounty hunters, lessons from Darth Vader. We've been doing a ton of research these days on vulnerabilities and bug bounty programs, and there's lots of different slices and view of the data, whether you're a vendor or a researcher, et cetera. But we're gonna try to make this one a little bit of fun. and go from there. So a little bit about risk-based security, you probably know us from our community offerings, OSVDB, DataLossDB. If you're into sort of conferences, there's this thing called SACOR where you can follow conferences and all that good stuff. And if you need any commercial stuff, you can come talk to us. All right, Star Wars. So who's a Star Wars fan in here?

All right, now who's a real Star Wars fan that if I ask you like questions, there we go, you're my guy. I'm gonna talk to you a fair amount of this, but we'll see if anyone else can help them out. Since it's being recorded, which is a little sketchy, Disney, Lucasfilm, this is all sort of fair use to prove a point to help us get better at security, all right? All right, so best episode. What's the best Star Wars episode? For the love of God, Empire. Why are we even debating this, right? I have kids now and I don't even tell them about Jar Jar and those first three. That just doesn't exist. Right? So Empire. So what we're going to

do is we're going to talk a little bit about bug bounty stuff and we're going to inter-engerman the Empire movie throughout and we're going to talk about stuff that happens. All right. So we're going to do a quick bug bounty overview to set the stage. All right? So first thing, how many people work for a company that have some sort of bug bounty program? All right, who knows what a bug bounty is? All right, so we got about half the room. So we'll do a little education here as well. All right, I saw maybe one fake hand about a company that runs a bug bounty program. Is there anyone that has a company that does

bug bounty programs? All right, anyone considering it? They've been hearing the buzz, so they're thinking it might make sense, maybe. Okay. What about any researchers? I know there's one. Researchers have submitted to a bug bounty program. There we go, three of you. All right, so we don't have a lot of numbers here. All right, so think about researcher motivation in the old school days, right? Effectively reporting vulnerabilities to vendors, it looked good on the resume, the CV, right? That was pretty much it. You got credited advisories, and that was about it when it came down to it. Unemployed people at the time that are researchers could use it to get a job, right? You could

do this cool thing, you say, oh, look, I know what I'm doing. I'd like a job, and make it profitable. Or people that had a job, they could use it to sort of promote themselves to get a better job. We still see this today, right? We still see researchers coming out with big vulnerabilities. Now they have fancy logos and names and all sorts of stuff to raise the visibility. And if you have a, you researchers, if you don't have a logo for your vuln, then it's not even worth disclosing anymore. It's not, no one cares. No one cares. All right. So there was really nothing altruistic about this, right? It was seriously sort of self-promotion,

if you will. Reporting to vendors back in the day was a pain in the ass. It really was. My whole three vulnerabilities that I ever reported in my long career were painful, right? They were either legal threats, right? Or you were just ignored. And so it was just very, very painful thing to do. So instead of sort of trying to talk and work with vendors, you would do alternatives. just publish it again, right, for those social sort of recognition. Maybe you would trade it with other people. You could use it maybe for offensive fun and profit, or you could just store it somewhere and say, hey, that was great, I found that. Or there was some money options available at the time, right? So

gray markets, you know, we're in DC, right? Three, four letter agencies, you could make some money there. Or you could go to the black market and you could make some good money. Some early bounties, right? So people, some vendors and some security companies finally realized that, hey, you know, we can incent people, we could reward them, and they'll give us this information. And so we started seeing this early on. August 2002, most people know iDefense, right, the VCP program. And I remember one of the first volumes that I dropped on a list somewhere after I did work with a vendor, iDefense wrote to me and said, you know you could have made some money by

giving this to us, right? So we started to see that sort of stuff going on. And then Mozilla's pretty well known in August 2004 for their bug bounty program, paying $500 for critical bugs, okay? So, that being said, and you're not allowed to answer, in what year was the first bug bounty program started? 1995. It wasn't him. I knew. You know, well hang on, so yes, October 1995. Not from you guys. Seriously, shut up.

Who started it? Who was it? Netscape. Oh man, you're killing me. Netscape. Oh, I miss Netscape. Does anyone miss Netscape? 20 year anniversary of that. So Netscape was the first one. Now what's interesting, so they launched this Netscape Bugs Bounty back in 95 to improve their process. Interestingly enough, their approach was pretty cool, right? They really wanted vulnerabilities reported in their latest beta. And so they wanted to give researchers incentive to find issues in the beta. before it went to production. Sort of sounds like what Microsoft just did here recently, doesn't it? Is that 20 years later? What's old is new again, right? So Netscape actually came out with this concept of let's try to secure our products through the use of bug

bounties before we ship them to people, before it gets out in the public. All right, full disclosure. 2000, 2008 timeframe, you could debate, it still sort of goes on. Battleground was sort of huge at this time on between vendors and researchers about how to disclose things, right? Researchers were still having problems getting vendors to respond, right? And at the end of the day, the perception was, true or not, that the only way to get a vendor to do something or fix something was if you dropped it on full disclosure, right? If you forced their hand. And there was evidence of this. You tell a vendor, they say thanks, They ignore you for a long time.

You drop it to a list. People go outraged. Next thing you know, the fix comes out pretty quick. So you would see people that were hardcore researchers that wanted to get things fixed. They were basically saying the only way, the right way, is full disclosure, right? And that was your importance of getting it dropped. Can't talk about bug bounty stuff without bug bounty contests like Pwn2Own, 2007 at a conference called CanSecWest. This is where you started seeing a little bit of money on the line, right? You could see MacBook Pros, $10,000. Bigger money came on the line in 2010, about $100,000 if you could break various sort of targets that they had, et cetera. And this competition was bringing lots of PR and growing cash incentives.

Alright, at the same conference, CANSEC West in 2009, this is where we started hearing researchers talking about no more free bugs. And you could sort of understand what they were saying, right? You're spending all this time, your own time, your personal time finding these problems. You report it to a vendor. And if they didn't treat you like garbage, they were just saying, thanks for that. And there was really no compensation for it, right? So it was like free QA testing, if you will. So we're not clear on how much impact this really did have on this whole sort of bounty history, if you will. But at the end of the day, it was very clear

and sparked the debate that there were some researchers that just basically said, we expect compensation for our work, right? All right, and now we're at present day, which you guys are really bad sample size because almost none of you raised your hands for anything. But bug bounties are all the rage. And we're seeing people that are talking about them nonstop. If they don't have one, they want to get into them, all that sort of stuff. All right, type of bug bounty programs and awards that are out there for those that aren't really clear on this yet. We've sort of grouped them into several categories. The first one we call company-run bug bounty programs. This is

where your company basically just puts out what you define as, what you're going to reward, and all this good stuff. And you're the one dealing with the researchers directly. So you're running the program on your own. What we call third-party bug bounty. programs like ZDI as well as iDefense. This is where a company you can report and work with them and they'll pay for it but it's not their own software, it's not their own websites. Now Competitions are still popping up all over the place, like we talked about. And then finally, what we call crowdsource programs. We'll talk more about this. We're talking about BugCrowd, HackerOne, CrowdCurity, SYNAC, and more. And we'll explain some more about this, but this is sort of using crowdsourced ways

to get people to look at your stuff. Bug bounty programs can be very, very simple. You can just sort of tell people you thank them, right? Like kudos, points, and a hall of fame, and appreciation sort of promise fame and glory, right? And there's some people that will definitely do that, and they're fine with that. But then it sort of ramps up into like tchotchkes, like prizes, t-shirts, mugs, maybe we'll pay to fly you to a conference, that sort of stuff. And at the end of the day, cash is king, right? So if you're really trying to incent people to look at your stuff, cash is what the big reward is. All right, so we're

going to talk a little bit about secure coding, standards, guidelines, and expectations. All right, so at this point, I think everyone can basically say that we expect the products that we purchase to be reliable, the function as expected, and to be safe and secure, right? That is a very strong word. It is a very strong word. If you think about it in terms of sort of tangible products, you get into a car, You turn the thing, you expect it to work, you expect it to go back and forth, function all the buttons. You expect if you get hit in the back of the car, it's not gonna blow up and you're not gonna die in fiery death, right? And so in the software world, whether you can argue

it or not, or the number of people that Internet of Things have killed yet, there is this sort of thought process that, hey, now that software is embedded in these things that can hurt people, right, we should expect them to be reliable as well. Why are we okay with this fact that just every, piece of software is insecure and that's just the way it is, right? And things don't work quite right, but that's just software, that's the way it is, all right? I think Internet of Things is gonna start changing things when the number goes from zero to people start getting hurt, when it's not just credit cards and intellectual property and that sort of

stuff, when it starts getting into human life and stuff, we're gonna start seeing much differences. But at the end of the day, like, We think security is critical. Everyone's here on your weekend to learn and engage in this stuff. It's about time that we start doing security, right? So expectations should be there. But, and this is an awful eye chart, and I don't want to hear a word about statistics sucking, but the bottom line is software is still shit. It is still awful. Whether you like the eye chart or not, and whether you think it's plus or minus, whatever percent off, we're seeing around 10,000 vulnerabilities a year in software, right? You saw in the

keynote today talking about vulnerabilities and security vendors products. I mean, my God, they can't get it right. So what do we expect of other people? And this has a blind spot in it too. This isn't even talking about cloud multi-tenant software as a service type stuff. This is straight software that you can download and install and it's still pretty crap, right? Another ugly eye chart just to call out that what we're seeing is sort of two of the most prevalent types, cross-site script being SQL injection. For the love of God, are we 10 years now and we still have these as some of the top ones, right? So what's going to change that 10 years

from now that I couldn't just take this slide deck, update some numbers and say the same damn stuff over and over, right? But we're not seeing sort of any indication that we're getting more secure code, better code, even though we have these expectations of it, right? So as security professionals, We try to provide some sort of clear expectations. We can argue about the word however we want, right? We try to say this is what we expect, whether it's through training, education, OWASP top 10, whatever's critical, top 20, whatever. We do assessments like we've talked about. We explain what's wrong and why it should be done differently. And you sort of would think by now we could have a bit more secure code. And if not, at least maybe cross-site

skipping and SQL ejection going away or on the downturn. But we're still finding that companies just aren't getting the job done internally, right? Even with clear expectations. All right, so this brings us to this guy. Finally.

We had to work for it. You had to get some real education in there. All right, so over to this guy, right? What did he want in the beginning of Empire? He just wanted to find the damn rebels. Where's the rebel base? Right? Is that so hard? Does he have to micromanage everything? He just wants to find this. Is it that hard to find? And I love the little Lego characters, right? So what does he do? He sends out a bunch of these droids all over the place and they're doing their thing, cruising around, sending pictures back. Should have been pretty easy. And what happens? We get a picture back. And his staff goes, hey, I think we've got something, right?

And then of course, one of the admirals comes over and goes, eh, I don't know about that. I want proof, not leads, right? If you remember, are you with me? All right, good. All right, so then, it's this guy, right? So Vader hears him talking and they say something's going on, so he walks his way over and he goes, hey, have you found something? And what does this guy say? He says, my lord, there are so many uncharted settlements. It could be smugglers, it could be pirates, it could be. And then finally we know this is sad Vader because at the end of the day his staff has let him down again, right? He's had to come in here and tell them straight up, he's had

to tell them, look, that's the system, let's go, right? Set your course for Hoth, we're gonna go there. And what just happens, right? He rolls in and wrecks stuff, it works well. But it takes Vader to get in there to get the job done. His internal team's letting him down. So, do we know who this is? What's this guy's name? If you're a real Star Wars fan, you're gonna tell me. Is it the guy who kills or the guy who replaces the guy? There's a lot of that. What's his name? What is it? No, it's close though. Starts with the P. There we go. He's actually captain here, so you're close. It's close, it's close. Who's this guy?

He's not dead yet. What is this guy's name? What's his name? Admiral what? Any other fans over here? You guys are... You're at a security counter. You don't know Star Wars. Admiral Ozzel. Alright. Alright, so what do we know about Ozzel? He wanted proof not leads. Right? He came out of Lightspeed too damn close to the system. Right? He's clumsy as stupid and he's failed for the last time. So what happens to him? We know what happens to him. He gets force choked. And now who do we have? You are in command, now Admiral Pierre. All right, so, Vader sets clear expectations. You screw up in his world, you're gonna get force choked. All right? So, developers.

And does anyone know where this is? The crazy Balmer, how many times did he say that in a row? It's all psychotic, right? So we tell developers what we expect. Are there any developers in the room, by the way? There we go. So this is for you. All right, time to force choke you. I'm not sure what else to do here. We've been telling you guys for a while. We're not seeing it get done, right? So you think we can force choke developers? No, they're hard to find. Good ones are hard to find. Maybe that's why we let them get away with the lack of security. Who knows, right? We can't force choke them. Alright,

so bug bounty programs, do they make a difference? Alright, if you guys can see this in the back or whatever, we got a chart here, I want to show you Shockwave Player, sort of a trend line for vulnerabilities, okay? So you can see from 2003 into 2008, Shockwave Player is an amazingly secure piece of software. Who agrees with me? There we go, we're starting to hear some things that maybe it wasn't there. Right, maybe people didn't find it. So look at this. In 2009, starts getting some focus. And the focus is, hey, we'll pay you for vulnerabilities. All of a sudden, wow, bones shoot up. It was really a great piece of software before, but all of

a sudden now just new vulnerabilities are here, right? They didn't exist. They didn't exist. It was just correlation. Exactly. So mid-2011, what do you think happens?

ZDI says we're all done paying for vulnerabilities. Software is inherently crap. And now all of a sudden it's completely secure again. There are lots of examples of this that you can look at, you can run numbers you want, you can fight with them about stats. But we definitely are seeing that money motivates people. Researchers look at things that either they're very interested in or they're getting paid to look at. Yes, sir.

Yeah, there's a fair amount of sort of does the information come out. Typically what we found from doing vulnerability work over the years is that eventually it sort of comes out and since we backfill it, it does help us to get a picture. But is it a clear picture? Of course we're not entirely sure, right?

There you go, too. All right, so I want to talk a little bit about company-run bug bounty programs. There's a lot of them out there, Facebook, Yahoo, PayPal, Google, et cetera. Again, as we've talked about, in almost all cases, reporting and coordination goes through that company and not intermediaries. It's pretty simple. You hopefully follow their rules. You find their stuff. You send it to them, and then you talk about whatnot. There's a huge number of these programs growing. We maintain our own list. We've been tracking the value of vulnerabilities and trying to determine what motivates researchers for a long, long time. Bugcrowd has a great listing as well down here at the bottom. But you can see there's, and it's probably way more than this

now, but well over 300 document it. 260 of them have some sort of reward. 165 are like the Hall of Fame, we'll give you kudos. And then 75 or so straight monetary money. You can't talk about bounty stuff without really talking about Google. They started bounties in 2010. They're really one of the more serious bounty vendors out there. They're a big reason we think that bounty programs took off. The Ponium 4 stuff was announcing $2.7 million in prizes.

August 2013, they talked about paying out more than 2 million for around 2,000 valid reports. Now they're doing bounties for other software, not just their own stuff. And they continue to push for bugs getting fixed at a much faster pace, right? So pretty good stuff there. They just last month, or September 2013 put out an update about Chrome specifically saying that they've already squashed over 700 Chrome bugs, they've rewarded 1.25 million, and now they've even announced that they're increasing payouts. It used to be, or now it's gonna be 500 to 15,000, where the maximum previously was five grand, right? So there's some money in there if you're finding the more critical stuff and you're also, they pay that higher range if you are providing exploit information in there, you

know, how to exploit or proof of concept stuff. They also had outliers where they would bump up to the 50,000 range even before the recent bump. Yep, yep. And now higher than that, so. So depending on sort of, they would always say case by case, if you did something really bad ass, they'd even give you more money. And so they just said recently, as an example here, they give a $30,000 one in August because it was a single vuln, but it was really, really impressive. And so you got them. They also have launched this thing called Project Zero. Anyone heard of Project Zero? Right? So to be determined how this, where this goes, but they've

created this sort of dream team of researchers where they're now just looking on their own to weed out bugs in popular software. Their statement from the project is, you should be able to use the web without fear that a criminal or state sponsor actor is exploiting software bugs that infect your computer. As of September, 48 bugs are already there. Yes, sir?

Yeah, so I haven't seen that from Google yet. We've asked a few times, but people are starting to be more and more comfortable with starting to share certain things. But asking for percentage of budget is probably not going to happen, right? From Google, there's a lot of zeros after that decimal.

So there are rules, we'll get into some of the crowdsource stuff, but there are typically rules of engagement where if you're selling it to one place, you typically can't do other things at the same time, right? So you can't sort of send it to 20 people and get them all to pay. So Facebook, I want to talk about Facebook a little bit. They've been doing some great work in here. So they reward, research obviously for anything that would compromise their community and platform. They were founded, the bug bounty program was founded in 2011 and this research is as of sort of early August. Over 1,500 bounties were rewarded from Facebook.

unique researchers were paid some sort of cash money, right? And they do have researchers in 79 countries and it's very interesting to start seeing where the, you know, the researchers are based, right? We've done a lot of research in, could you basically say, you know what, forget this daytime job and I'm just going to go be a bug bounty hunter? And it depends where you live, right? So, or it depends what sort of bugs you find. There's lots of rumors of big ones. We're not going to talk too much about brokers in this talk. There's lots of rumors that are not sort of validated about how much money you can make through those other gray

markets too. All right, so showing the money, average bounties in the low thousands for Facebook. $500 is their minimum, but they don't have a maximum. They sort of do the same things we talked about with Google. If they see some cool stuff, they'll reward it accordingly. Their largest bounty was about 33 grand.

and they have some more information out there, so it's pretty interesting. All right, a couple of things about Facebook, which if you guys are considering as your own company doing a program, you need to be careful with. Facebook came out with some nice stats, not sort of budget-wise, but they basically said that they paid out 1.5 million in 2013. And they gave a little bit of details, and they also share some stuff with us, but when you look at it, They received close to 15,000 submissions. And so that was a huge increase from the year before, right? 250% about. But what's concerning is only 687 of them were deemed actually valid reports to them and

eligible, right? So who are the math people in here? Right? It's like 4.65% valid.

So when you do some of these programs and you ask people to submit things to you, you're going to get some submissions. So you best have the team or a process in place or some sort of partner or something that's going to be able to handle this. Because when researchers send you stuff and then you don't reply to them, it's like back in the old days, now they just get really pissy and go out on Twitter and God knows what else they do, right? So you got to be careful on these submissions that are coming into you and understand the workload it's going to bring. All right, so... Now we're back to Star Wars again.

Was that a good break for you? So, Vader finds the rebels on Hoth, they flee, right? Are we with me? Remember the show? All right. All right, remember this guy? What is he looking for after they flee and he can't find? More specific.

He's not looking for Skywalker at this time.

No, Jar Jar Jar Binks is not the right answer. All right, so anyways, this guy, they're looking for the Falcon, right? He goes into an asteroid field, right? Do you remember? You guys are awful. You need to go watch this movie when you get back. It's ridiculous. So he's chasing him. He doesn't want to go in there. There's an asteroid field. He's saying, this is bad news, right? Remember all this? They even get this Minox thing eating on the falcon at the time, come on, right? Finally they have to leave, right? So this guy again, what do we know about him? He gave excuses about going into an asteroid field, about the risk, right? He

lost the falcon, couldn't find it, he just couldn't get the job done, right? So here again, we have internal resources failing Lord Vader, right? He kept saying, find the falcon. Should we force choke him? He's not gonna get force choked here. Can't force choke everyone, come on. Let's be honest. So what does he do? He says, we're gonna bring in the help. We're gonna bring in some help here. My internal team's not getting it done. We're gonna bring in the bounty hunters. So it's great his internal team's trying, but he keeps telling them, find the base, they're screwing up. Find the Falcon, they can't do it. So he brings them in and he says, hey look, if you find the Millennium Falcon, there's gonna be a

substantial reward. He's pretty vague about it too. He doesn't even say how much money, but everyone's on board, right? So, Admiral Pirat, what do you think he thinks about it? He says, we don't need that scum. All right, is this like sort of companies too? We don't need external people, security guys to help us. We could do this stuff. We're unbreakable. We got this. So he brings him in. All right, so here's some thoughts on it. Vader is in the middle of nowhere. God knows where he is next to an asteroid field. And all of a sudden, he gets all these bounty hunters just to show up, right? His team's not getting it done. Next thing you know, boom, they're there. And we're not...

His bounty must be good. Substantial means something is the definition for Vader, right? And we're not just talking about randoms that showed up. We're talking about some of the best people that have showed up. I mean, if he would have said, hey, I need help, and then this guy shows up, then you'd be thinking, hey, I don't know. But he's getting the best guy showing up, right? We got the best bounty hunter in the galaxy on this one. All right, so great. Internal team's failing. They're going to keep trying over here, but we're going to bring in some outside people to help us out, right? And so what happens? All of a sudden, because of

where the Falcon is, they've got to get the hell out of there because it's not really an asteroid like they thought it was. They start flying. All of a sudden, they found him again. Are you with me on the movie? All right, good. So they're looking for him. Then all of a sudden, guess what? What happens? gone again loses him again so what does this guy say do you know his name by the way that's okay hey no kidding oh you got it yeah i'll fix that for next time so what does he say he gets smart he goes you know just take accountability for this one right i shall assume full responsibility for losing them and apologize to

lord vader that go well by the way There it is. Right in front of them. Just like some of these open source software vulnerabilities that have been there for 20 plus years that anyone could have seen but didn't.

But your quote is right. He does say, no ship that small can have a cloaking device, right? That's a good one. All right. So, yeah. What do you think happens? Apology accepted. Captain Nita, right? So he gets force choked. I will say, this was an okay slide, but this one is so much better. And the fact that you can just seriously go on Google and put in something and next thing you know, someone's already got this for you, this is amazing. I agree, but that force choke is just great right there. Some lol, mean cat stuff. I love it. No sound effects. All right. So.

What then happens? We're back to needing bounty hunters yet again, right? They screwed up. They can't find it. Boba Fett's the only smart one to hang out there when they're in their little trash dump or whatever before they roll off. And he's the one that finds them, right? And again, you hear Vader at this point saying to Admiral Peter, don't fail me again, right? So he's trying to tell him what he wants, and he's being clear about it. So in all of that, Do we think that we can actually get bug bounty hunters to work on our software easily? Just show up randomly? Pay enough money. Maybe? All right. So this is where we start

getting into this crowdsourced bug bounty stuff, right? So I want to spend a little bit of time on how this works. If you're not a Google or a Facebook and you don't have millions of dollars and you don't think you can handle it, there are some companies out there now that might be able to help you. So the way that this stuff works, effectively your company would sign up with this service and offer bounties through their platform. The bounties are really open to all researchers that are signed up to that particular platform. Validation of bug submissions and bounty payments can be handled through the service. Typically what we're seeing right now is the validation part

isn't. It's still mostly a pass-through, but the sort of the payment and interaction with the researchers will be done through that platform. We are also starting to see a real blur now between what we would call bug bounties and pen testing, right? And so you're seeing this, we'll talk a little bit about with some of them, where it's not so much you're putting out here, you're just saying, hey, I want to do an engagement and they're only paying for results. So you that are in the pen testing world, it'll be very interesting to see if this is something that companies want to do instead of traditional engagements. All right, so Bug Crowd, founded in September

2012, they're out of San Francisco. They basically look for pretty much everything, web, mobile, Internet of Things, Embedded. They are one that just started this sort of thing, they call it Flex. And it says crowd source penetration testing now, not just the bug bounty stuff. Again, in the August time frame, there were about 23 public active programs, 170 of them, sort of various stages, and 57 companies had used some sort of a deal there. They had over 10,000 researchers signed up. So when you talk about sort of can you get access to people quickly, their answer is, hell yeah, you can get access to people quickly. Our question always is, are these people any damn good? Who the hell's doing this stuff, right? So out of the

10,000, how many Boba Fetts are out of that 10,000? Maybe not many, if at all. But it's very hard to tell, OK? Researchers around the world, 231 unique researchers have been paid some sort of money through their program. Over 1,000 bugs are paid since November 2012. Their average bounty amount $241. So we were showing some big money earlier. Through this sort of program, it's lower cost. So when you think about it, there is a potential for your company to pay lower price bounties and still get pretty good work, get lots of eyes looking at your websites or your software. They pay primarily through PayPal. rare exceptions, they'll do Bitcoin and all that other stuff. Their

process time, typically it's two to six weeks from the time that the bug is submitted until they paid. Their largest payout that they had at this time in August was $13,500. Again, it's a great map provided by BrugCrowd, just mapping out where all their researchers are located. So those 10,000, you can see sort of where they're grouped. And it's pretty interesting to see there for us. They also have this whole deal about kudos and karma and all that sort of stuff. So it makes sense for some companies. If you're a non-for-profit, but you care about security, you just don't have money for it, you can still sign up for the platform. And researchers have the ability to find and help you, and then they

just get these little points. So you don't even have to pay money through the platform. They'll also let you do shirts and all that sort of stuff. So you can still use the platform and just determine what you want to provide. Not everyone's a fan of kudos. Right? You can go out and see people saying, look, you know, I'm not going to waste any of my time unless there's cash. You could see this, right? Too many bounties with no bounty. And so it all just sort of depends on, again, motivating people. I will say there are some pretty well-funded VC security companies out there that are not rewarding money. They're just doing kudos. And I

think that's crap. If you've got some money, you should reward your researchers the best that you can. All right, hacker one. Found it in September 2013, San Francisco as well. They focus on response teams and they try to differentiate them themselves a little bit by saying that they're more of a platform that you can use and you can use their platform for absolute free and then you just determine if you again you want to pay people. 63 teams currently using the public one. They've got more private ones sort of a soft launch that they continue to use. Thousands of researchers registered over 800 researchers have been

submitted something valid that was either sort of on their recognition leading, you know, kudos board, hall of fame, or some sort of money, but they weren't willing to tell us how many unique researchers at the time. Close, a little over 1,300 bugs have been paid. Their average bounty is a little bit higher, $677 per bounty. Largest payout they had at the time there was 15 grand, multiple ones were awarded. Some of it through the internet bug bounty stuff. Another was from Yahoo. And then just here recently after the whole Shellshock bash thing, they rewarded like 20K after the fact. I'm not really sure how that whole just throw money after it works, but we'll see how that continues on. Yeah. Yeah. So this is the internet bug bounty

one. This is pretty cool. Again, they're similar to the Project Zero, the most popular software out there that we rely on. They're trying to put up money, which is funded by bigger companies to try to improve the software that we rely on, right? So it's pretty neat to see. You can see the internet bug bounty. Their platform's pretty cool. You can see the companies. If they choose to make it public, you can see sort of who's doing what, how many bugs are closing, and dollar amounts. All right, Crowdcurity. 2013, July, San Francisco, web application security, they've been focusing on Bitcoin as the starter. I think they've been talking about expanding more, but they're really talking about Bitcoin. Bitcoin, 45 active, 90 programs all time, 50 to 100

companies have used it. They also say they have a good amount of researchers, 1,300 researchers have signed up, 300 to 400 being active, and about 100 unique researchers have been paid money. 800 bugs have been paid out for them. But their average amount's $150, and if you look at sort of their levels here, they're a little bit lower than some of the other ones. So it'll remain to be seen if lower amounts of money will keep researchers away from there, or because they have, again, unique targets where people want to go after these Bitcoin targets. Their largest payout was $1,500. They also have a Hall of Fame deal. One of the things that they do

is sort of like weekly or whatever, they'll tweet about a researcher. What I like is this report quality. If you're in the vuln space and you track vulnerabilities, researchers are either really awesome or they just want to make you kill yourself because of the crappy reporting. And so it's pretty neat here to see that they are tracking the researchers and what they're providing into those companies. So we'll see how that pans out. All right, Cynac. SYNAC pretty hardcore says they're not a typical managed bug bounty program. These guys are really trying to differentiate their self again with this sort of pen testing angle, right? So they focus on application vulnerabilities across the web. They didn't want to share a whole lot of information with us, so

it's hard to tell what's really going on with SYNAC, hence why I want to talk to you later. So they run only paid engagements. And what ends up happening instead of if you're like a client and you want to engage someone to do a pen test and they say, here's the statement of work and here's the money, you're going to pay that no matter what they find. These guys are saying you can engage with them and then you only pay for what's being found. So it's sort of a different tune on pen testing. So we'll see how that goes. Unknown number of researchers, they wouldn't tell us how many unique folks. They say 40% of

researchers are US based with the remaining spread. They're a little bit different in the fact that every other program that I showed you thus far, you can just go sign up and you're in, right? So, I mean, I've no skills anymore at all. You know, I signed up and I was in and I could be rocking it. When you go to sign up with these guys, they want to interview you. They want your first, last real name. They want to do a video chat with me. So you're not going to find, you know, people that want to be sort of remain hidden or whatever using this program. No clue how many payouts. No clue on

the average bounty. They say most of their payouts are between $100,000 and $5,000, and they wouldn't tell us their largest. So we'll see where these guys go. All right. Bug bounty hunters, rules of engagement. All right. So does anyone remember? I'm looking at you again. Yes, sir.

Sure, so from a researcher standpoint wanting to get involved, there's a good talk that myself and Carson Irem did at DEF CON about researchers. We called it, Screw Being a Pen Tester, I Want to Be a Bug Bounty Hunter When I Grow Up. We went into in depth on that to give some tips. But the reality is, if you're interested in this stuff, you can sign up for them and just go check them out, right? That's the best thing. Sign up for each of them. Go look to see what's active. And it's a great way for people that are just trying to get into the security space. You can take a look at it, see

some of the targets, and give it a try. So I would do that path. And I'm happy to talk to you afterwards, too. All right, so Vader, right? We're going back to him again. He set some rules of engagements. What did he say? Well, that was the other way around. All right. He says, you're free to use what other methods are necessary, but I want them alive, no disintegration. That's what Vader's telling him, right? So he's trying to tell the bounty hunters, again, setting expectations. You can get them, but I want them alive, right? All right, so same thing with bug bounty programs. There are rules, and we expect a lot more rules to come in this, okay? It may not always be as clear as you expect, right?

So you want to understand what's considered valid. If you're trying to get into this stuff, and you spend, two, three weeks working on something and you submit it and they go, yep, not valid. Well, you've just had a bad couple of weeks and that wasn't really worth your time, right? So what are the restrictions, limitations? How are duplicate reports handled, right? That's one of the things where you've got 10,000 people that show up and all of a sudden you're working on it. Someone scoops you and sends it in five minutes before you, you don't get paid, right? The first one that gets in there. So there's some things you need to consider if you're going

to join this stuff. Welcome to race condition. All right, so it's something that you need to pay attention to, okay? Bounties can make, who's on security blue team in here? Keynote guy over here saying blue team's the way to go, right? Only a few people raising their hand. Bug bounty stuff can make your life hell as a blue teamer, right? Especially if you're talking about SaaS sort of live websites. How the hell do you know what's an attack versus a safe bounty thing going on, right? What happens if someone's out there trying to do a decent test, but the next thing they're flooding you, denial of service, they wipe your, there's a lot of things that can potentially go wrong, right? So

you do need clear rules of engagement when it comes to live sites, production, customer profiles, and all that. Do you see anyone standing up like, okay, here's

a test network, we'll be on it. Yes, it's already going on and you're leading me into, we've seen problems at Facebook with this, right? I don't know if anyone remembers this, I don't have enough time to go into details, but Zuckerberg's own timeline effectively hacked by a researcher, right? So basically he says, first, sorry for breaking your privacy and posting to your wall. I have no other choice. After the reports I sent to the Facebook team were pretty much ignored, right? So I'm going to...

It wasn't a vulnerability. They didn't want to deal with it. So anyway, so yeah, so he posts and there's this sort of longer story about Friend and Zuckerberg and all that sort of stuff. But effectively... There was a miscommunication with the Facebook team and they didn't think it was an issue and he wasn't explaining it clear, I think, is their take on it. So Facebook fixed it, but they basically said, we're not paying you. You broke the rules, right? You didn't follow the rules, which is don't test on customers' live profiles. No, no. They created a Kickstarter to pay them. And look at that. You want to give this part? I'd love to give this part. So what does the security community do? We decide to get

the own bounty for him, right? You can talk to Space Rogue later about that one. But effectively, I mean, hell, they raised like 13 grand for the guy. So anyways, I don't know if the moral of the story there is good or bad, but you gotta be careful, right? And when it comes down to it too, If anyone thinks that legal threats still don't happen, right, they're wrong. I mean, you can see them, they're tracked. Brian has them tracked. So be careful, right? So if you're just out there messing around and you're not following, you think you're being a good bounty hunter, you know, you could get yourself in trouble. The other thing, too, is about disclosure, sort of a question about multiple places here and there. So if

I find a bug and then I give it to, say, Google, for example, and then I just drop it on them, will they pay me? You know, Google is sort of one of the ones that they might. They basically said, hey, look, we really want you to work with us coordinate we're gonna give you money here you should at least have some courtesy and work with us but if you go against the principal it probably doesn't qualify but maybe it does a case-by-case basis I think there's a lot more folks I don't know what's going on over there there's a lot more places yeah there's a lot more places that would basically say no right we've told you what the rules engagements are you're not following along we're

not gonna give you money for this you've screwed up Yes, that's out of scope. Out of scope. Instead of just knowing that's one of the things we wanted to look at. Again, people will, for these bug bounty programs, say what's in scope. So you may find something absolutely critical, but if it's not in scope, it might not qualify. So there are some concerns there. All right, so he's been giving me the warning that I need to start wrapping things up. So we're going to go into wrap-up mode. So bug bounty stuff, are we there yet? So Vader relied pretty heavily on bounty hunters, pretty much, right? I mean, at the end of the day, his internal team was not getting it done. And the reality is without Boba Fett

being there, he wouldn't have got what he wanted at the end, right? So he needed those resources when the internal team just wasn't cutting it. I'm under the impression that bug bounty programs work, and they're a good thing, OK? It provides a clear way to engage with researchers, right? I mean, so many people will find something. There are people out there that want to help you, and if you don't tell them how to engage, then it just gets ugly. Bug Bounty, they incent researchers to look for your stuff. And you can go from kudos, start there if you want, and work your way up to cash. And it allows for you to really define and control that disclosure process. And in many cases, you could say it

could reduce a breach potentially, right? So you can control it, you can get it in, you're not going to get something dropped on you on a weekend while you're trying to be here at this conference, right? So it's a good thing to define how to coordinate. And then, cruise scrutiny on the projects, on your stuff, it's a good thing, right? So a lot of people get worried. They say, I'm a little concerned if I do this, it's just going to bring them all after me. Well, we've seen that with Facebook. It's a potential, right? And so you've got to be prepared for it. But it is a good thing with more eyes looking at your

stuff being incented to look at it. But what do bug bounties? They do not do, right? They do not replace solid SDL process during development. They do not replace internal QA. Believe it or not, they don't replace external consultants and third-party confirmation. And at the end of the day, they do not lower cost of correcting defects for your company or customers. Because again, you want stuff fixed earlier in the cycle, not when you ship it out and people are relying on it and using it. Okay, so this isn't a sort of yeah, I'll just forget about all the security stuff and we'll just bring in the bug bounty hunters, right? So we wanna make sure

we're being clear on that. All right, couple more things. This bug bounty stuff seems pretty damn easy, doesn't it? Let's just throw one up there and it'll be perfect. What do you think? Maybe not. People know about the Yahoo HD Bridge case? So effectively, HD Bridge finds several vulnerabilities, cross-site scripting in Yahoo. And Yahoo, believe it or not, and you gotta feel for them a little bit, the guy that was dealing with it He was actually trying to be nice, and he wanted to give out a shirt as a thank you. But the easiest way to do it was just to give this $12, 50-cent voucher for the little trinket in the store. And so, as Karsten would say, after I spend all my time on

that, you give me $12.50, that's fucking offensive, right? I mean, that's ridiculous. So you need to be slightly careful here. It's better probably to send that shirt or say thank you than to say, oh, all of your work is worth $12, right? But to be very clear, it was just one security guy hanging Yep, yep, so this guy was trying to do the right thing, but it's absolutely just, it's a recipe for bad press and that's what they got, right? So in November of 2013, they put out a proper program, anywhere paying from 150 to 15K and those cross-site scripting vulns ended up a thousand bucks. So you just wanna be careful about it. There's

a big difference between bounty and extortion. Does everyone understand that, right? You see this guy, Probable Onion up there, tweeting at someone, you have 12 hours to fix the vulnerability in your system, otherwise I'm gonna take control of it, you've been warned. And then this friendly, and then, and then this company comes back on, Ooh, we'd be happy to bug bounty it. That's a little different, right? And so of course, probable onion staying true to himself says here, you've been warned, but email me. We'll talk about some money, right? Now, This guy had been doing a bunch of other stuff, if you're aware of it. He's a teen that's been arrested. He was swatting people.

He was swatting Krebs and doing all kinds of stuff. He even got to the point where he basically put out on Twitter saying, hey, if you want me to swat anyone, let me know. And if you don't know what swatting is, that's when you basically call and say, Brian Martin at this address over here has drugs and there's a hostage situation. You should send help. And then the SWAT team rolls in on him and wakes him up and creates a really nasty day for him. So hi, Brian. Thanks for using me as an example. You're very welcome. All right, the future of bug bounties. More and more companies are gonna jump on board on this

stuff, right? There's a lot of people that are jumping on already, and I think as people get more comfortable with it and they see the value of it and how to do it, they're gonna jump onto it. We believe that karma and kudos points, that's just gonna become irrelevant. Cash is gonna be king, and researchers are gonna follow the money. And they may be playing along with $300 now, $600 there, some of the bigger dollar amounts, people's eyes open up. But when you talk to some of these brokers that are out there, again, I'm not justifying or saying that their numbers are right. But when they're saying things like a single vuln can get you

a couple hundred thousand dollars, one claimed over a million dollars for one vuln. If money like that does exist, then you're not going to get kudos and karma points and 500 bucks and that sort of stuff. So we do think that money will eventually talk. And it does have a fad sort of potential, if you will, that can wear off that they'll continue to exist. But then we hope that sort of companies will continue to understand that investing internally really is the right way to go. So at the end of the day, what do you guys think? Was Peart correct? We don't need the scum? You think we need bug bounty hunters? Maybe? We need

scum? This kind of scum?

Or as much as I like Boba Fett, is it gonna go the way of the Sarlacc pit, right? Is he going down? So in the end, again, we think follow the money. I'm low on time, but I definitely want to say thanks to a bunch of people. Carson, Irem, and I spent a lot of time on this. Brian spent a lot of time. Katie from HackerOne, Nate from Facebook, CrowdCurity guys, Marissa and Casey from BugCrowd, and all the bug bounty hunters out there. Thanks for doing it. And thanks for your time. I appreciate it.

Yes, sir. So back in the old days, which you didn't go into the really old, old school in your talk, but back in the old days, we did When we had vulnerabilities and we found them, we didn't do it for money because there was no money. We didn't do it for t-shirts because they didn't give us t-shirts. And we didn't do it for credit because there was no credit. So we did it to get shit fixed. That was our goal. We wanted stuff to get repaired. So if you're a researcher today and you sell your bug, what control do you now have over that bug and can you ensure that it actually gets fixed if

that's your motivation? And I hope that's somebody's motivation still today. Thank you.

I don't have to repeat the question, thank God. Yeah, so again, it just depends, right? I mean, what he's saying is true. Some people, again, thought the right way to get things done, and that was their motivation. You've got to read the rules of the bug bounty program. Some of the company ones, basically, you're giving up control, and that's that, and it could be considered hush money, right? And I don't think that's the intention of it, but you're not going to be in the loop demanding a fix. Some other ones, like HackerOne and some of the Google ones, to our knowledge, they are going to fix them. And you still sort of retain some level

of involvement. And you're not just giving straight up everything and over it. So if you're into that, you just got to read the details. Whether or not bug bounty programs just turned out to be show me the money and no one cares, we'll see. We don't know yet. Yes, sir.

Yeah, so each of the programs has what they try to put pretty clear legal sort of scope around it. So if you follow the rules of what the bug bounty program says, you're in pretty good shape. But each one of them are very different. But they'll try to say what you can do, what you can't do, and then again to this whole sort of disclosure thing, if you're turning over the rights to the information, then you are not allowed to talk about it sort of ever again. But if you're not following the rules and you go off track and you start attacking something that you shouldn't have, you can get yourself in trouble. So you've

got to be careful. Yes, sir.

Yeah, so at the... ZDI and IDefense figured this out. Some people were finding a bug, selling it to both of them because they were competitors and wouldn't talk a lot. But eventually, employees from one went to the other and vice versa, they started looking at the history and they're like, holy shit, we've both been getting ripped off all this time. So it does happen. It comes out eventually, right? And so at the end of the day, you may be able to get away with it for a bit, but at some point it comes out. And if you're trying to do this over for quite some time, it's not something you want to do. It's not

limited. You could do it up front, but if you do it for a while, it's going to be caught. Yeah, exactly. Even in the vulnerability database world, we track researchers and if they're going to be a pain in the ass for us. And as most, what's the confidence level of if this guy tells us their problem, should we trust them? Or has it been the last 10 reports that have been crappy? So they're tracking this as well. So yeah, you get away with it for a little bit, but at some point it's going to come out that it's going multiple places.

Yeah, you can do that. But again, most of the brokers are very similar, right? Once you give it to them, they're going to say, have you given this to anyone? They want to work with you and they want to know. If someone's going to pay, again, rumored amounts of hundreds of thousands of dollars, the last thing they want to know is you've already given it to someone else. So you can blow that channel. You might be able to do it for a bit, but at some point, it's come back.

And again, this is an immature space. We're going to see how it goes. Yes, sir. How do we, as researchers, trust the . Yeah. How do you trust them ? Yeah, so I mean. . It's a great question, right? Who am I to say that someone else isn't trustworthy? But you've heard it over and over about do you trust formal criminals that are now selling, you know, vulnerabilities and all this stuff. And at the end of the day, it's a risk on your side. You know, you could go to a Google directly and you know they're going to be pretty stand up trustworthy about the process, but you're only going to get so much. And if you're chasing these big dollars,

then you're starting to deal with maybe some seedy characters, who knows, right? If that's your thing and you're happy to sell into the gray market or the black market, then maybe the brokers are the best way for you. So it's a risk, but it might have a higher reward if their numbers are accurate. But again, we just haven't seen proof of these massive numbers. And in some cases, because first of all, you probably shouldn't know who it's being sold to. Second, you probably don't want to know who it's being sold to. And they don't want anyone else to know either. Anything else? All right. Well, cheers for taking the time at the end of the

day. I appreciate it. If you have questions, we'll be around. Thanks.