Gamify security best practices to scalably improve engineering culture

my name is David I'm an engineer on the security engineering team at chime and I've been here for almost three years today I want to talk to you about monocle which is an internal rails app that we've built at chime which helps us get everybody to do the right thing for security specifically engineers so we're going to look at some of the scaling challenges we've had at chime First for security specifically how we approach those in terms of gamification education and appreciation of people doing hard work or work that's typically not appreciated and some of the results we got from leaders and among engineers and how you could get started at home or at work I should say

in an easier way than what you're seeing here like you know I spent as you heard a few years here at Chimes so this is definitely not something you finish in a weekend so in terms of our challenges at chime we recently tripled our engineering team so that's a lot of people to learn things we've created a lot of new services and with every new service there's different Baseline security configurations that you need to set up so we've filled those and we've also completed a number of audits of PCI socks things like that and as you can imagine those were somewhat stressful at times raise your hand if you've ever had a painful audit okay wow that's like 95 of the audience

so we want our teams to focus on security I think we can all agree on that but how do the teams actually learn security best practices you can't just rely on them you know reading things online and teaching themselves so also how do we get everybody to secure their services without burning ourselves out with all of the jira tickets that we create and Bernard TPMS out uh technical program managers out with all of the pinging that they have to do to get people to address the tickets um and then also how do we even know whether our services are secure in the first place uh you know there's a lot of security tools that we're all using and it's hard to figure

out uh you need something to fuse all that information together so I want to give you a little illustration of what it's like without monocle or a tool like it um so hi this is David I'm from the security team can you please resolve these 23 vulnerabilities in your service so raise your hand if you've sent a message like that okay nice um yeah and you can imagine that this kind of uh message might make the engineer receiving it feel sad um some of the things going through their head are like I didn't know I was supposed to resolve these vulnerabilities like why didn't somebody tell me about this before I got in trouble so

so to try and address that one thing we do is we send a message to their team Channel and we say hey your score dipped below a b because we're giving everybody grades but or not the people but they're repositories a grade between an A and an f and we want everybody to be above a beat if they're a production service so we send them a message looks like that's big enough on the screen and we tell them hey do this and this to improve your grade they can click through and learn more about it when they when they're on their GitHub repository itself we are showing their grade on the readme and the reason we're doing that

is because people spend a lot of time on GitHub and looking at their own readme's they'll notice oh my score is not good I should make that go up um and so the majority of our teams are really proactive about this which we love and we don't even have to ask them to um follow the rules so here is what it looks like once you click through the badge you can see a number of the different rules score factors that we are grading people on and each one has some points associated with it this is recalculated every night based on API calls to different places and you can click to expand an item and learn what you specifically need to do

to fix that item and uh some of the key items that reduce our audit workload are approved base images these are getting rebuilt every week or so to make sure that the vulnerabilities that are patched are the patches are pulled in we make sure that everybody has at least one pull request approval you know making sure you don't have like a single uh actor threat type thing but also making sure there's no bugs and things we want people to resolve the vulnerabilities that we find so if you've got higher criticals we're telling you hey please fix those and we want to make sure that everybody has tests and passes those tests before they merge uh

it seems like a basic thing um but you know some repos don't have that so we show a good amount of information to leaders as well about how their how our security posture is so this is a screenshot of the home page of the app and you can see a breakdown of how we're doing in terms of who how many people are above a b and this screenshot roughly this is a fake one but gets presented at our monthly engineering review meetings if you don't have one of those at your company you should maybe tell people have one and then you can join it and present security stuff basically the attendees of that are like all the

engineering managers and Engineering leaders up to the CTO so on the right we've got a breakdown of which score factors are passing or not and that's really helpful because sometimes you've got a new initiative you want to make sure for example that code climate is enabled for everybody or let's say that we've got no security vulnerabilities for GitHub security alerts then you can easily click on whichever bar and find all the repos that are not doing that thing and then just go track those people down um that's pretty helpful so we've also built this thing called risk advisor which uh evaluates every pull request to make sure people are following our like four most important rules

some of those are I already mentioned but the the important one that's not mentioned is dependency confusion uh that's when you accidentally downloaded a dependency from the internet instead of your private version of a gem or dependency so the way this looks is on your pull requests you see a status it's kind of small sorry but basically it's showing you yes you're good to merge and if you click through you can see the rules that were evaluated and see that your ready to merge but let's say you're failing some rules then this is what you might see as an engineer on the page you know a little red X on your pull request and let's say they didn't notice that

and they clicked merge anyway we're not blocking the pull requests because we want people to still be able to get their work done um so that means that they might merge without passing the rule and if they do the monocle will detect that and automatically create a jira ticket to track it so it'll also send a message to their team channel so that the whole team can just be aware of something that happened that wasn't supposed to and help resolve it um we found this really helpful um you know with jira you can query all of the past forever's worth of Juror tickets and then very easily satisfy your auditing requirements um so we realized that even though

arkadi here who's giving a talk later today gives a really interesting onboarding session sometimes employees Engineers will forget that monocle even exists and so we wanted to create more awareness for monocle and so one of the ways we did that is we came up with the idea of sending scorecards and we send these scorecards to slack channels and it shares some basic information about the repo and how they're doing and we noticed that teams celebrate that they have good scores and that's really nice because a lot of us probably have co-workers who are doing important work like keeping all the packages up to date but they're not really appreciated because it's kind of really annoying work and it's not you

know as cool as building a new feature let's say um a quick note about these uh slack channels that we've got we've found it extremely helpful to create a convention-based slack Channel naming scheme so hashtag GH repo name and uh when we we have it set up so that monocle will create the channels for us and add the top contributors but that's not really necessary you can just do this manually um and you know all of your tooling can just send messages to those channels without having to figure out the name you don't need an ownership directory or a spreadsheet or anything so uh it would be very bad if our members lost trust in us and we lost the

financial data that they've trusted with us um so does gamification work does it work for us to give grades to all these repos and I would say yes it really helps us get a view of how is engineering doing how is security doing and are people following best practices and culture guidelines um so we've seen that leaders can see what our security posture is Engineers are not overwhelmed by having to check five different security tools probably we don't even give them access to all of those tools in the first place um and we're saving about 2 000 Engineers 2 000 engineering hours a year on auditing um that's time that they would have been spending trying to use github's

improving but maybe not perfect search capabilities um and clicking through just manually to check stuff so this is a graph the blue is how many score factors we've been tracking and that's been going up over time and the green is how many repos have above a beat and both of those have been going up and to the right which is perfect so we're safeguarding member data people are improving security and leaders can see the numbers going up we've also found that it's much easier to roll out new security initiatives so every time we've got a security initiative the all the structure for notifying people and delivering the words to people so that the instructions all of that is already

there with monocle so we just need to focus on what are the instructions and then following up on the few teams that don't already uh follow our instructions on their own um and uh just briefly on that our TP GM's technical program managers are just very happy with this as well they have a lot less spreadsheets that they have to track and they don't have to Ping all of these engineers and Engineering managers um so if you are thinking about where to start for yourself think about where where does security intersect with engineering and uh start simple it could just be a Cron job and it gets some basic information out of the GitHub API and then it sends

slack notifications to teams and maybe you have another Channel where it sends reports There's Something Beautiful about sending slack messages which is that you have a chance of critical hit or whatever you want to call it basically an engineer might have five minutes free a junior engineer or just anybody and then they see the message and they fix it and it's done you didn't have to make a jira ticket and track it and all of that tracking can be more expensive than actually fixing the item for in some cases um I would also recommend checking out ossf's All-Star and more generally backstage as uh terms of solutions that you could try setting up at your company

um here's a little photo of the architecture so everybody is visiting the rails app the rails as excuse me the scores recalculate every night as they talk to our different data sources like GitHub stack rocks jira Circle CI and then that gets stored into postgres this is roughly how risk advisor Works where subscribed to webhook events from GitHub and we validate that they came from GitHub and then we forward information on to open policy agent which runs these Rigo rules and tell us whether or not the person is passing the the rules that we've written and you know based on if they pass them or not we create a ticket notify people and update the GitHub UI

we are authoring those rules inside of the monocle repo which you know makes it simple as well um talking a little bit more of the philosophy of where you start um it's a good idea to find Engineers where they already are so ideally that might be in their editor because it's the cheapest place for someone to fix something we haven't really gotten to that yet um next would be GitHub they're readmes and pull request statuses that's what Engineers are looking at all the time and in slack because at least for our company that's where everyone is hanging out all the time they're not looking at email so much or at least me um and you want to give them

instructions that don't include security jargon so ideally small items that they can follow and if you can help them celebrate their investment in security with their team that's what the scorecard is about and these messages to the team channels help with that as well so if a tree falls and no one hears it did it make a sound uh yes your security is definitely better after these things have happened and so showing everybody that they did happen is a really nice thing um so yeah uh we can move on to questions in a second but if you have something you want to ask us you can send us an email or mention me on Mastodon and probably someone's going to

ask this but yes we do want to open source monocle but we just haven't found the time yet hopefully this year we'll see um and chime is looking for an engineer to work on data security so consider reaching out this is definitely my favorite favorite job ever thanks [Applause] foreign