BSidesSF 2023 - How do you trust your open source software? (Naveen Srinivasan, Brian Russell)

how do you trust open source software um introducing open source scorecard hi I'm Naveen srinivasan um I'm excited to be here at besides SF this year um I'm all about open source and supply chain security I'm a contributor I contribute to a few open source projects um and I'm one of the maintainers of scorecard Brian I'm Brian Russell I'm a product manager on Google's open source security team and I work a lot with openss scorecards as well with Naveen so before we get into the the meat of the presentation we can start with just a fundamental question of really what is Trust you know we can look at a formal dictionary definition I tend to think of

it as at what level can you feel comfortable about something that you can rest easy kind of knowing that that things are safe and taken care of when we think about it in terms of how you build trust we're going to look at in two different ways there's kind of the ideal gradual way of doing that that's where we can really get to know you know a person or an organization over time by doing that you start to get to know them you can really just directly see what's working are they doing things that seem safe are they doing things that aren't the issue with that is that scalability is really difficult especially if you

start thinking about in terms of Open Source software and all the dependencies that go in there's just no way you know you as one person or even a team of people can go out and just meet everybody that basically builds all those dependencies so let's think about how we could jump start that if we could just find a way to get to know someone to be able to build some trust up front one way to do that is really just looking for evidence of how things are operating when it comes to a project what kind of best practices are they employing and when we start to do this approach and look at it from an evidence-based

perspective that's something that we can do with really high scalability we can go out and we can collect that evidence repeatably for all these different projects that we're interested in so thinking a little bit more in terms of software what kinds of best practices really start to matter and you could ask yourself things like if I had a couple of projects how would you feel if you know one of them was conducting code reviews and another one wasn't or one of them has unfixed vulnerabilities or one cares about permissioning things in a way that that really makes sense for an open source project and whether or not there's activity or if this project is is just sitting there without

a lot of people still watching it so these are the kinds of of best practices that you could start to look at project by project now if we think about building that trust in the context of a very specific problem we start with this block diagram in this block diagram really represents what modern digital infrastructure looks like when you pull back that first layer you have a lot of components some are big some are small some are really well maintained others might have one critical point of failure what you don't really have when you just look at it like this is a lot of insight into what's going on only you know what really is is important for your

particular system so what if we start thinking about evidence and what if we start thinking about what if we build some insight Within These different dependencies what if we started being able to look at it in a way where you could see that some dependencies are really in great shape and other dependencies really have a lot of work to do and you know some are doing a pretty good job but but have some room for improvement it's that kind of insight that we want to think about tools that we could use to really start building that and that is really what the open ssf scorecard project was created to do is to help really collect best practices

or evidence thereof in one place such that you can evaluate projects really quickly so if you want to take a look uh at a scorecard right now you can go to this site called depths.dev slight disclaimer since I'm at Google depths.dev is a Google project but they are also the biggest consumer of all of our scorecards data and at the moment they have the main ux to look at scorecards so if you navigate there I encourage you to just think of a project that you use and if you don't have one off the top of your head you could just look at the scorecard project itself uh do a search for it click on the project

piece if we have scorecard data for it you should see it on that project page and you can start to look at within those set of best practices or checks as we call them you know what's going well on this project and and what could have room for improvement so to give an example of you know one particular check each of these checks have roughly the same kind of structure you have a title of what it is a description of what it is and a risk level associated with it so in the case of binary artifacts this is a case where you have binaries actually checked into a source repo which means that you can't actually see

the code behind them and even if the code is supposed to be in the repo too there's not that strong link between the two so what scorecard does is looks for evidence of that and then for each binary that it finds it just in kind of an Olympic scoring kind of way starts subtracting from 10 a point for each binary that's found and that's really what you would see across these different checks this same kind of basic formula of this is a best practice and then there's a scoring mechanism under the hood if you look on the project page you can start to get more details on on how all of those are counted and what what really each check

means so at this point you might be wondering you know can I use core cards if you have a private repo versus a public repo the answer is yes scorecards comes in a few different flavors but essentially what you can do is run it as a CLI run as a GitHub action or consume the public scorecard data that's already there what if you use git but not GitHub you can use it in a slightly limited way but those checks are there and you'll be able to run like the CLI tool to do that and for fans of gitlab out there if you have that stay tuned that's something that we're working on right now so a brief note on just

what really supply chain security is looking like like why scorecards really came about and then we'll we'll go into some actual demos and and kind of go into a little bit more of the mechanics uh the long story short um we could pull up more stats than just these but open source software consumption is growing it already accounts for a really big chunk of most people's infrastructure or the code that they're shipping and that volume is only increasing over time uh after this presentation is available you know you can look a little bit more at what the Linux Foundation found what's behind these numbers and what sonotype published in their their annual state of software supply chain support

but what you see is strong growth and with that growth unfortunately you also see a steep increase in supply chain attacks themselves so this has become an increasing problem and something that more and more people are starting to look at uh this graph comes from sonotype's annual state of the software supply chain report uh if you have a chance take a look at it it will go a lot more in depth into both the problem itself but also look at how different tools can be used to help mitigate that and it takes a special look at its scorecards in particular as well so with that uh we'll get to some demos and I'll hand it back to Naveen

thanks Brian um last year we launched scorecards API um and I want to take this opportunity go discuss about the API why we built it what is the reason and all that the API is available at api.tickets.com Dev one of the critical things which we made a decision was to provide the data that scorecard already collects um it collects 1.2 million GitHub scans every week and all of the data is available as bigquery but we spoke to a few customers and all of them wanted some way of getting it using standard HTTP so that's one of the critical reasons we build the API this API does not require any tokens so anybody should be able to utilize this

API um we also made the API predictable so you can hear in the in the call command you can see it's kubernetes communities that you should be able to replace with any of the hopefully the 1.2 million GitHub scans uh get up scans if you do the question then comes what if a repository is not being scanned we will have answers for that as we proceed further but this is the this is the way this is one of the reasons which we build this API it's a standard rest API um it does get and post but uh right now I'm going to focus on get but I will uh as the demo goes on we will show why we

use post 4 and how we're using this like I mentioned um what is this API serving it's still it's serving 1.2 million um GitHub weekly scans so scorecard team runs um scorecard on 1.2 million get a repositories collect the data and when you hit the API you get those scans so essentially you can make decisions based on that um we launched this API late fall last year we are seeing about 200 000 requests uh not too much but this is this is running as public server so anybody should be able to utilize this who is using the API we don't know who's using the API so if you're using the API please come and talk to us we would love

to hear uh for what you're using for here in this example uh we we saw somebody open an issue they were trying to ask without do we have any weight limiting right now we don't enforce any rate limiting um explicitly but don't deed ask this so what can we do with this API um what you see here right now is um is a pie chart diagram of specifically of one of the checks called code review code review is one of the the most critical checks in the state of supply chain security report what we decided is how cool will it be to use the API to decide how the top thousand critical projects are I'll talk about how what

this critical thousand critical projects are so we pulled the top thousand critical projects use the API pull the data of them and we wanted to see how they are performing and we can see in this but most of them are doing well but a few of them need some help um coming back to specifically critical projects there's another open ssf project called criticality score it's specifically measures the criticality of Any Given Project based on an algorithm Rob Pike built um and the cool thing about that is you can use you can use it for your own needs all this data is available free and um you should be able to tweak your criticality based on your own parameters

it has conf it has configuration to go utilize that so we use that to go build this essentially using the API okay enough with all the dark can you show me the code please uh I don't want it bored with code over here so the code is uh on my personal GitHub repo so you should be able to clone it uh use this it's written go um scorecards written go I'm familiar with go but you should be able to utilize any modern programming language to pull this it's a pretty simple project let's talk about the usefulness of this API I am going to be the contributor and I will be a security maintainer for a project

I want to submit a new PR with the dependency Brian I mean that sounds pretty pretty reasonable uh but you know is this dependency really healthy what does scorecard be helpful Brian you know if I knew what that was I might be able to tell you have you checked out the scorecards apostrophe Brian well uh uh you know now that I have uh this is great in fact I think I'm going to use this API for checking all future dependencies in fact I think I might add it to my CI CD workflow tool and also it's just so easy to use makes me wonder why doesn't everyone use this at this point thanks Brian um so we decided to build this as

um another get up action it's still a work in progress feature which we are working on what you can see over here is if you add any new dependency it'll come back and give you a score and that can help you decide whether you want to add or whatever you want to do reject the PR but it does not but this is this is what we are trying to work towards and the whole idea behind this is you should be able to utilize the API to make your own decisions and that's the whole idea behind this

um next is we also launched last year an open ssf scorecards badge what is the idea of the badges the badges is for the project repository owners to showcase how what are the score for their repositories it's for consumers to come and see that that's the basic idea behind this this in turn uses the API so if you run these scorecards get up action it uses the API to post the data so that's why if you if you if you bring up a new repository and their repository does not we are not scanning it you should be able to use the GitHub action install it run it and you should be able to get start getting

your scores so that's another way to get scores uh populated into scorecards API so so we want to take a take an example and Brian was we don't want to fumble in front of everybody to install the demo Brian was thankful enough to help us uh make this so we just pre-ken recorded it so hopefully it should should show us how it's not pretty hard so if you go to school class repositories specifically scroll down to the detection of how to how to get the badge we have we have given it uh something that you can copy it so you should be able to navigate to your readme and paste it and change only two things the the owner and the

repository the moment you change that and commit it commit that in you should be able to see those badges for the Repository like I mentioned the critical thing for this is the data should be available in the scorecards API

if I only have 30 seconds go to depths.dev that should be able to provide you scorecard's data if you have five minutes run the scorecard's cry if you have 15 minutes install the GitHub action add a batch to your Repository just to know if I if you use a private repository if you don't want to share your data we have that option not to share so you don't have to share your data with scorecards API if you have more than a day include incorporate the API into your workflow so to close this out I just want to say a few words about the organization in which this scorecard is being developed the open ssf has a number of different

projects but it's an organization that is really meant to bring a whole bunch of different people together that have a vested interest in trying to make open source software more secure so it has a variety of different members that have come together under it to really work towards making open source you know as safe as any software as you could have so participation is very open uh there are a number of different ways to really engage with the openssf and see not just the scorecards project but a number of other projects that are really critical towards skier towards securing open source software uh some of them are here uh there is also kind of the one

catch-all page which is openssf.org get involved where you can get all those links on the last slide so I think we're at the close at this point uh and I just like to thank you all for coming I know especially into the day talk I appreciate everybody that that really came out uh at that point we'll go to questions

great thank you guys let's go into q a mode do I have any questions out here I'll be on my way

you I'll meet you thank you Sako is here my question is around first of all amazing job like we are going through this dependency hell I would say but are you having roadmap also some automation around updating let's say to new releases and so forth so it's a great question thank you in terms of kind of I I think you're asking about are there ways to automate fixes based on the findings of scorecards so that would definitely be an extension of where it's at today uh at the moment we're still very focused on detection and making people aware so that they can go in and make the fixes it's not to say we went ever enter uh

into that but um in the moment it's it's not on the immediate roadmap go ahead thank you thank you great um we have five more minutes does anyone else have any questions coming around I'm going to go up and then down to you if that's okay

coming right at you um the work in progress feature you mentioned to post on PR as many new dependencies introduced seems really awesome and will work really well for those of us who work at companies where there's a massive mono repo things like that how can we get updates about when this is introduced and I'm also really interested in the um you know the point about when we do dependency version upgrades are we downgrading actually perhaps security wise can we know that via a PR comment absolutely um that's a that's a very valid that's a very valid concern um you should be able to I can go back to that I'm sorry um the reason I'm going back is you you

should be able to follow this PR thread and you should be able to um I'm the pr owner um so hopefully in the next few weeks I should be able to merge this in and get it working but if you follow this PR thread we will do that and also like what Brian mentioned if you follow the open ssf in from Twitter to everything there will be a a release with that being in a blog post you also you should be able to utilize that information hope that answers your question he coming down to our next question

yeah so um what was the process for coming up with all the things on the sort card and how do you see that changing in the future if that will evolve d so it's a great question you know when you have a whole bunch of different best practices who's really to say you know these are the right best practices uh initially we started out with just things where a broad group of people who was working on this project universally said you know yes that would roughly increase my trust in a project I don't think there was uh a super rigorous methodology it also predates my time of working on the project but the starter set was really just you know

people who had worked in the space for a while had been thinking about the problem for a while um you know came together and basically said I think this is a good set if we if we knew these things that would help influence decisions about whether or not we would want to use this project I hope that answers the question do I see any further questions coming over

there you are just as a follow-up tool I was asked what is the process for adding a new you know line item so oh that's a config that's a config setting for that um you mean for the the oh I think he means checks oh and you a new track Oh Oh you mean a new metric so um for that you gotta you got to create an issue we have a discussion and then we go about doing it so it's like any other new features on any other open prior open source project that you want to do you come back and specifically give us why how and and then you should be able to go ahead and get that working

yeah the short answer is come talk to us go to the get involved page and and you know join meetings open PR's um all those will start the right discussions we have time for one more question okay well thank you guys for being here today on behalf of b-sides SF and doyan sec I would like to present you with these gifts from our sponsor doinsek and thank you so much guys for wrapping it up today thank you let's give him a round of applause thank you