Lessons from Building a Human Firewall

Okay, we will start on stage two and uh Allar will be talking about unleashing the crowd. Please

[Music]

[Music] Hello everyone, my name is Allar Lok And I wanted to ask you a question. Does any is anybody else feeling it? Like that relentless pressure as your product attack surface is expanding but your security efforts just can't keep up. It seems like a impossible task especially considering limited resources. But um what if we didn't have to fight it alone? What if we could uh leverage a global set of allies, help us secure our defense? That's what this talk is all about. By the way, has anybody heard of Vault? Well, if you're a stone and then you maybe maybe probably yes, but uh if not, then here's the gist. Basically, it's a platform for mobility

and delivery service. We have five core products. ride, rental, car sharing, aka drive, delivery, and market, which is grocery delivery. So, but before we begin, let's see where we are right now. Can I get a show of hands? How many of you are working in cyber security? Okay, see a lot of hands. That's great to see. So, I've come to the right place then. So, here's a little bit about me. I'm a cyber security engineer in Bolt and I work for a product security team. I mainly work with the vulnerability management and I'm here to share our lessons from the trenches. So discussing about our experience with crowdsource security and also most importantly I have a dog named Millie.

So let's start with the problem and try to understand where does crowdsource security fit into the security toolbox. So the problem working in a fast growing company means that everything is constantly changing. There's reorgs, new features, new endpoints, new attack vectors. This means that the attack surface is constantly growing and security teams are always left playing catch-up. So, okay, that kind of sucks. But surely that means that the teams responsible for keeping these systems secure are also expanding at the same pace right? Unfortunately, no. The reality is that security teams have to make do with a lot of limited resources. So for example, our product security team is very small and we like to think in the

long term. So we want to build scalable systems which help our future move forwards. But that means that we lack the capacity for deep and narrow focus which is required for uh for every application and the service. So we're constantly asking what is the most important um and feasible security problem that I am capable of solving right now. And the sub question would be is doing activity X better than activity Y to solve our problem and this is like a constant cycle. So we got to ruthlessly re-evaluate and pick our battles. And what happens if we have constantly growing attack surface and limited resources? We get blind spots. And these blind spots are constantly growing

by these threats. So what can we do? Well, we have let's start with uh option A. So option A, shifting left. We could delegate security testing to feature dev team. After all, they are the most uh knowledgeable about the feature and it probably means means that uh this is really efficient and um helps us with saving with costs. So, but um it's clear that the feature dev team is uh really adept at uh usability and functionality and performance of the feature. But uh is it the same with security testing though? Um maybe maybe not because most uh feature development teams lack specialized security expertise which means they need a deep understanding of uh vulnerabilities and also how to

exploit these in a given environment. Also testing is mainly focused on intended functionality and this means that the security testing is often uh you know kind of forgotten or just uh dep prioritized and also time and resource constraints. So as we already mentioned uh feature dev team's main goal is to create a usable feature or application within a reasonable amount of time that fulfills all business requirements. That's the main goal. But the attackers are not limited by this deadline. Not at all. So they have all the time in the world to explore all of the uh the whole ecosystem and their goal is to find and exploit vulnerabilities to subvert the program uh applications intended purpose

and uh finally there's limited scope and tooling. So there's a limited amount of time to do the features and as we talked security testing usually does not belong there. uh about tooling. Security tooling uh requires quite a lot of uh expertise and also yeah knowledge about uh how to use them and the vulnerabilities and also they often require a business license which means that uh they need to get team and everyone needs to be on board or else it will be considered a non-essential cost and um meaning that the feature dev team can't use it Okay. So, what else can we do? Well, option B, let's try hiring internal pentesters. Let's see where that gets us.

So, an internal pentester will develop a deep and continuous knowledge base which will have very strong understanding of company systems and architecture which is great. Also it uh creates a seamless collaboration between the developer team and the information security team and the strategic focus. So the internal pentester will develop a strategic focus which helps identify systemic weaknesses. But uh there are some cons. First of all, it's a high cost because hiring a full-time talented employee is often more expensive than just uh doing a project based external pen testing. Then it is not a scalable solution yet again because of the high costs. uh and also the rep um results are very dependent on the subject's skill set.

So let's uh discover some more options. We already briefly touched on pentest providers. Let's see let's see about that. So, pentest providers have highly expert um specialized expertise uh when it comes to testing and also they provide an independent assessment which is fantastic. Um also they're available on demand of course so you can just schedule when you want this application to be tested and they will do it. Uh but the cons are that it's time boxed. So again there's a limit uh limitation of time which means that the testers are often incentivized to find something rather than do deep exploratory testing which is very time consuming. So also they have lacking contextual knowledge of the whole system. uh which

means that uh they they have a hard time finding complex chains of vulnerabilities and therefore uh it is hard to demonstrate for for example impact and also they're restricted to a very specific scope and they have to work within this scope uh or else they will breach the contract if they go out of this. So I I guess this uh means that they have to be pretty careful. So it's like they can't do what they want and also it offers a just a brief snapshot in time. So and what's the last option? Let's try using the crowd enter crowd source security. So crowd source security is has various types. So there's VDP or vulnerability disclosure program, bug bounty program

aka BBP and crowdsource pen testing. We at Bolt have implemented it through bug bounty program which uh gives us access to global and diverse expertise uh of skilled researchers which uh that we get like unique attacker mindsets from all over the world. It is continuous and always on. So yeah, basically 24/7, no time constraints. Also, it's very cost effective because you you only pay per per results. It's a very costefficient model in that sense. And it is rather scalable because if you add more features then you can just invite more people in testing and uh that kind of uh evens it out. So uh in very simple terms how does a bug bounty program work for the company. So a

company signs up to be u bug bounty provider uh define defines the engagement rules scope targets rewards launch the program gets vetted hackers to start testing the engagement uh findings will be first triaged by park bounty program provider and if it's passed then it's uh passed on to the company for the second level triage and if it's if it fails then it's not redirected to and the valid findings will be eligible for a payout from a company's price pool and uh delegated to the responsible development team. Pretty simple. So where does a bulk bounty program sits in the reporting flow? It's uh right here. And uh important bit is that it has to be before your ticket

management system uh to like uh ensure like uh logical flow of information. So yeah, you get the input, you get first level triage and if it passes, it gets second level triage. Nice. So let's talk uh our strategic integration. So let's say you find a vis issue. What do you do? You Google uh vulnerability report vault, right? And first link that pops up public vulnerability reporting policy page. You skim through that yada yada yada and you find security email. So you message the security email. What happens then? This is a manual decision-making point. So depending on a variety of factors such as time of day, workloads, perceived priority, it will either get pushed to Jira or ticket management system or

directly to product security team without uh bypassing ticket management system. Sometimes it gets added back in, sometimes not. And also important thing to note if you have to do this manual decision uh a lot every time when your organization gets bigger you will get a constant volume of these submissions and it will become a bottleneck really quickly. Okay. So let's say we started our crowdsource security program. We set the public vulnerability reporting policy. We set up a security email and a private bug bounty program with B routes. So this is how the setup looks uh on this um on the bounty program provider side. We set up a single uh simple engagement brief. We expect no format uh concrete uh

formatting from the bug bounty reports and there's no clear process for handling reports. So this is how this will kind of look like. So yet again you're here, you found the vulnerability, you go to security email and you have to make this manual decision again which uh as we um described can uh the outcome may vary uh significantly. And now there are even more decisions to make because you can you have three options basically now you can create a ticket to Jira or you can invite researcher to your program. So first that's the first decision or there's a third option as well. It can go secretly directly to product security team. So this is obviously not a very

logical flow. And also there's a second manual decision-making point. So a lot of cognitive overload is possible and you have to do this every time. So it's it's a lot of uh wasted uh decisions. What are the results of this? So as I discussed there there is a lot of operational overhead which means that uh making optimal decisions is rather difficult and it significantly contributes to the decision fatigue. So that makes it hard to prioritize because there is no clear flow of information which uh causes confusion uh and it becomes really apparent when you have to start tackling the backlog. Believe me, uh there's also inconsistent report quality because there are so many different uh sources of input and

there's no expect expected format. So that's a recipe for disaster. And finally, vulnerability databases are not in sync. So since uh you had to make so many manual decisions uh f crowd might have more submissions or differently worded and your jira might have differently worded or not uh documented at all. So the learnings are we have to automate to remove overhead. We have to improve the funnel. We have to incentivize good formatting because this is a win-win-win scenario because if you create a good format then you teach the security researcher to for u how to submit good reports in uh and increase their chances of a payout on our bulk bounty program pro uh provider side. It is good because it's

easier to triage and for us it means that less noise gets through. So yeah, all three parties win and uh the fourth learning is that we consult with uh our bug bounty program and improve the scope as needed. So identify and implement actionable steps to better identify duplicates etc. So after applying the learnings uh we make the changes which is to revamp the engagement brief. We turn the private bug bounty program to public. We set up a security email autoresponder and we set up autosyncing in bug bounty provider. So this is how the uh setup v1 looked like and this is how setup v2 looks like. As you see it is much more logical now

because autoresponder redirects to bark route and uh so everything will get go there means that we leverage the their uh program as much as possible and if it uh passes the first level try the triage then it will be to Jira which means there's a much more logical flow of information results Well, significantly reduced overheads. It's easier to make optimal decisions and there's less decision fatigue. And there's also higher higher signal to noise ratio because almost every triaged uh report will now include a working proof of concept. And higher quality reports means it's easier to triage and Yeah, already covered this. There's a single up-to-date vulnerability report database which is uh great. We can be

quite sure that it's all it contains all valid issues because it's always synced. So roles of these uh different methods in our security team toolbox shifting left finds and fixes lowhanging fruits before they ever reach production. This is a continuous work in progress as we're providing and building the tooling that they can use. So internal pentester provides continuous and deep inhouse knowledge and we hired it but the responsibilities responsibilities changed because we were building stuff. External pent tester delivers an unbiased audit report of a specific part of the app. We have that and the bug bounty program which leverages global hacker community to identify vulnerabilities composing uh of complex attack change which we also now uh implemented.

So let's talk some triumphs challenges and intelligence. So here are the triumphs we have uh found some quite interesting findings. Basically the first one is about how to get the free pass. So get free rights which is basically which uh happened because the security researcher dis discovered that he could manipulate HTTP headers and uh change the price before buying the pass. And of course our server was like oh that's fine. There's no validation. Let's do it. Then the second issue there's a payment bypass which allowed for free food which was uh basically tied to our uh customer support refunding system which uh happened where if you made a purchase and it was delivered to you,

you could then ask for a refund, remove one item from the list and uh it thinks that it uh takes that one item off the list but actually it provided a complete refund. So that's fun. And uh also there's a oneclick account takeover which was um pretty complex attack chain. Basically, there was a deep link uh which uh could contain arbitrary payloads and the security researcher was able to find out the uh JSON payload which basically created a fake window within the app which looks very legitimate and it contained a CTA button and there's no course policy. So basically you could send a post request to attacker C2 server and get like authorization headers which is not ideal

and but this also is a fantastic find by this security researcher and a perfect example of when Bounty program can provide you value. And the last one is unlimited remote scooter ringing which basically means that you could make any scooter in the world ring at any point in time for unlimited time um unlimited time and there's a working proof of concept to show you that. So I print to you present to you a symphony of the scooters.

[Music]

Yeah. So normally there should be like a free ring uh limit but since the endpoint was suddenly changed the rate limit was forgotten and that's the result. So we discussed triumphs let's uh talk about the challenges. So it is significantly reduced but there is still significant amount of noise. Uh which means that non-issues are portrayed to be really significant and issues reported to us are instead caused by third party uh third party providers. So just a few examples. There's also inevitable challenges of multiparting party triaging. So on our B bounty provider side there's like a lot of security researchers who constantly switch and uh so it means that uh at any point in time there's a different

triager and that means that the results can vary significantly. There's inconsistent report volume uh which means which can catch triaging teams off guard if they're not prepared for it. And finally there are unpredictable costs. Now I know we discussed that uh it is these are low cost. It's a it's a positive but uh you have to consider that the price pools which are made are usually created uh on the amounts that the average payout is done in some certain amount of time and most findings are either low or medium. So they have much smaller prices uh rewards. But if you happen to have a couple high or critical findings which have like payouts that are three to five or even more times

larger then it can like just drain the price pool instantly. Okay, let's talk a little bit about actionable intelligence. So how to go from findings to strategy? You have to start and documenting and grouping your findings. So you you can use this data to identify systemic weaknesses. You can use grouping per vulnerability type, business vertical, service, team or whatever makes sense to you. Then you have to evaluate and prioritize. You have to identify the most pressing but yet solvable issue at this point in time and you have to create a plan and take action. So some examples of action would be in case there are a lot of web vulnerabilities you might provide training to the developers on OAS top 10

for example or make a training for secure software development life cycle. You can also create some tooling which uh devs use by default to eliminate uh any kind of friction and also eliminate classes of problems at the same time such as for example if you have hard hardcoded secrets you can use like a secret scanner such as truffle hog which would run on every PR and would uh block if it contains secrets. Let's uh discuss a little bit about how to get started in uh when you consider a bug bounty program. First things first, choosing a vendor. As always, you should do your research and you should connect all of the Bounty provider vendors who you know uh which

interest you and ask about various topics such as program management. For example, do you have a private and public or just private or just public? Also, how does your platform handle triage and how does the communication with relevant parties act on your platform? There's also you should also ask about research community. How do you vet and make the sure that the security researchers are legit and not like some black hats? You should also ask about possible integrations and reporting capabilities such as what native uh integrations you have like do you have integrations to Jira or messaging applications uh web hook capabilities all that sort of stuff and also you should ask of course about security and confident confidentiality

of data which means how do you ensure that my data on your platform is secure. How do you do that? And what procedures are in place that help you protect this data? And also, of course, you should consider pricing. So on top of bounties, you should consider what kind of other fees there are associated with the platform. And also you should ask uh how can you help manage our costs so that it won't go out of control. So and you choose what works for you. For us it turned out to be bar crowd. So how do you decide between private and public uh engagements? Generally, if you're starting out, go with private because you have more

control over who gets access and can control the amount of participants in your in your engagement. Also, there are less submissions. So, it means that you won't get instantly overwhelmed by a sudden uh sudden influx of uh reports coming in. And you should consider making the engagement public if you've already ran the private program for a while successfully. You are ready for a large increase in submissions and you have confirmed with the bug bounty provider that this is the right way forward. So some tips for success from our experience that is continuously work with your provider and highlight issues. So that is the best way to make sure that these issues are fixed. You should create uh detailed terms of

service to enforce when needed. meaning that these security researchers sometimes go out of hand and uh start uh typing in all caps and uh demanding everything. So if you have clear terms of service, you can enforce it easily. Also you should keep an updated list of in scope and out of scope items because these bulk bounty providers they will check against it every submission. So it will help them with triaging a lot. You should also simplify your reporting flow as much as possible because this will uh this will uh remove the constant decisions you have to make so you can make higher quality decisions overall. You should also create an operational checklist to criteria uh of criteria to check because it saves

you time when triaging a lot a lot of time and also it helps keep you sane if you have a large backlog of items which you have to go through. So some concluding thoughts. Security works best in layers and consider this a bug bounty program is not the replacement for other methods. But you you you can also you don't have to fight alone. You must know that because and consider a uh crowdsource security as a valuable tool in your security toolbox. crowds finds what others miss because it's a global global mind. Uh you get very different uh unique attacker mindsets which uh means that uh they are able to find very different types of vulnerabilities.

Sorry. And also crowd helps you transform your vulnerability management by creating data through vulnerability reports which help identify systemic weaknesses and also help shift from the defense from reactive burden to proactive efforts. And also remember a bug bounty program can be a very very powerful tool if it is properly managed because you have to realize it is a continuous process because as your attack surface is constantly changing and evolving, so too must your park bounty provider because uh you have to keep up to date all of these engagement briefs. uh terms of service, in scope, out of scope, all that stuff. Also, it is a huge win if you incentivize good reporting. It saves so much time

for all everyone. So you have to do it really because otherwise this triaging will be very inconsistent and uh you won't make optimal decisions probably and also you should explore and implement the all of the useful capabilities that the provider offers which is right for your program. You should consult with your provider and highlight problems and discuss how they can help you fix these problems. You should understand all of the features, integrations, and services that the provider offers to understand what works for you and what doesn't. And basically just find what works for you and implement it. Because remember, it's not just about finding bugs. It's about building a human firewall. Thank you. Thank you for listening.

[Applause] If you have questions, we will pass the microphone. >> So, are there any questions by the way? >> Yes. Can you hear me? Yes, >> I can hear myself as well. Uh, so would you say this BBP is um is viable should be considered only by large enterprises such as Bolt or if you know I have two three applications that I'm developing and hosting uh would that be an option for me as well or is it like too much of a hustle to take up? >> Mhm. Okay, that's a great question. I would say that a bug bounty program makes sense if your attack surface is big enough and uh you can't feasibly cover everything that you're trying to

put out there. So if it's only a couple of applications and if you know all of the endpoints and uh and you know your system and if it's a small thing then I don't think it makes a lot a lot of sense because also as I mentioned the bug bounty uh program is a living thing and it has to be taken care of. So because if it's neglected then yeah it will not produce uh great results and uh also it is important that uh your company is relatively big because these uh researchers that get on it they there is more chance that they will come onto your uh program if they are aware of your service, right?

So, if it's a small thing and uh it's not really applicable for them, then maybe it doesn't make sense. I hope that answers your question.

1212 can you hear me? >> Yes. Yes. >> Uh so it's a quite a simple question but how many people from your side is actually like managing all of this program because I assume the board is like has a lot of people in security and you maybe you have full-time employees who are actually maintaining uh like managing tickets and all that stuff. >> So how many people it usually takes to run this program effectively? >> Okay. So our product security team consists of only four people and my responsibility is the crowdsourced vulnerability management the communications with the provider uh look for improvements and also solve the operational overhead. So it's on me basically. I don't want to delegate to

other others if uh if I feel like there are more improvements to be made.

Anything else? Any questions? >> If we don't have any more questions, then let's make another round of applause to Allah. [Applause] Thank you.