BSidesSF 2023 - Tracking Meaningful Security Product Metrics (Leif Dreizler)

all right everyone let's make some noise and then be quiet for our speaker leaf I'm Leaf I'm currently taking some time off in between jobs but most recently I was uh senior engineering manager at twilio segment for the security features team um currently the co-host of the hit podcast uh 404 security not found which has been described as friends at a conference talking about security stuff I'm not sure if that was a dig or a compliment I took it as a compliment but if that sounds like something you want to use to catch up on some news we we do one a month and I'm also a startup advisor and investor so if you you know have a early

stage startup I'd love to hear about what you're working on you might be asking what's next I'm joining sem grep in about a month so if you uh use sem grep and want to give me some some feedback on the product happy to take that if you don't use it go buy some sem grep make my my options worth something so um uh the agenda for today I'm going to give a little bit of context about what our team's up to I think that's helpful just so that you know the types of things we're creating metrics about I'm gonna try to sell you on why you should be doing metrics at your company although based on your attendance I'm

gonna hope that you're at least kind of bought into that already um gonna kind of walk through a project that we did to really upgrade uh our our own use of metrics and then wrap things up and maybe have time for questions although when I was running through the presentation it was right about time so you might have to harass me in person afterwards um so most of what our team builds is stuff in the authentication realm so uh the whole like login logout flow is all stuff that we work on we also work on an internal application called access service which is something that you can use to get time based peer-reviewed access to things there's some commercial

tools out there that do this now if you're familiar with OCTA IGA Crosswire opal and I think it's worth keeping in mind that segment is a business that sells to other businesses which I think makes security a lot easier if your customers get mad when you do a bad job at something the business tends to try to do a good job at that thing and so I think that investing in this stuff is is a little bit easier when you work at a B2B company so uh yeah based on your attendance I'm going to assume that you're at least kind of bought into metrics but I think that these are really important because a lot of security teams really struggle

to quantify and demonstrate the value that they bring to their org and I think a lot of that boils down to other parts of the business not knowing enough about your team which you can fix through internal sales to an extent and then also security just not understanding enough about the folks that they're supposed to be working with I think that everybody here has probably heard about a security team that's not giving practical recommendations or stuff that Engineers just like know won't work and you really want to minimize that because anytime you do that people are less likely to work with work with you in the future um so the right metrics Empower teams to

communicate cross-functionally it helps educate other departments about how things are going what things are important what things aren't important and even if you're not a customer facing team the way that we are or like a directly customer facing team I'd encourage you to get creative about how you can tie things to customer value I think one thing that I thought was very clever that our GRC team did is they started making our sales team enter in the contract value before they'd give them a sock 2 report and this wasn't meant to be like a hurdler or any anything like there wasn't I don't know that there's really a minimum bar but it's just at the end of the quarter you

can say hey we helped out with x million dollars worth of deals um because they wanted this this sock 2 report that we had um so I'm going to talk a little bit about access service again this is an internal app that that we built a couple years ago and I think the metrics can also help you justify spending time on internal stuff or help you rationalize to other teams why it isn't a good use of time right now um without metrics you might make a statement like this at the end of the quarter or the end of the month or whatever however long you spend on this like hey this thing felt slow We sped it

up and now it's fast um but if you have metrics on how slow something actually is you can make this a lot more impactful by you know doing a little bit of research saying okay hey uh the web standards recommend a page load in one second before people start feeling slowed down it's an internal app so maybe like two to three seconds is okay but like five is is definitely too long um and then you can do a calculation you know I use calculation very Loosely but really what I'm what you're doing is trying to figure out like the effort to value and so if you think hey it's going to take us two weeks to speed this thing

up um and we can get a 50 reduction in in loading times like maybe that's good maybe maybe it's not but you at least have something that you can go back to other teams internally and say hey this is how long it's going to take us to do this thing this is a good use or a bad use of our time right now um so creating a a solid metrics program is pretty time consuming this was a pretty solid investment in our team our by our team this this past year it was something that two Engineers spent um like close to a quarter working on uh obviously Engineers work on plenty of other things it's not just three months

of straight work on something but um this is a pretty big investment and like any meaningful size project you should be asking is this the right thing to do right now um your company or your team probably has no shortage of of good ideas and it's really about figuring out if it makes sense but this was something we we said we wanted to do at the beginning of the year so we had already kind of like put a um you know some time allocated for it but we ran into a few issues that I think we probably could have avoided if we had had done this work a little bit earlier we had an internal app that

crashed during a big migration and there wasn't really any warning we just looked at the logs afterwards and we're like oh hit 100 utilize 100 utilization and crashed maybe a warning at like 80 that something bad was going to happen would have been would have been helpful um so I break things down into two different groups of metrics Health metrics and product metrics um Health metrics are things that help you answer is what we maintain dependable and will we know when something goes wrong um I think security engineering teams should really try to be as similar to their end counterparts as possible and so you should just don't reinvent this wheel like just go to your Edge team and

figure out what they're doing they're probably better at running reliable Services than you anyway just steal their homework um and so some things that I think like broadly fit into this category and like there's probably others too is error handling uh this is pretty pretty self-explanatory I think but just to give you some context on like what we actually fixed we had some things that would send errors to Sentry but then those were inconsistently sent to slack we had some services that would just send stuff to slack without going through Sentry and then you lose out on all the benefits of Sentry we also had stuff going to a bunch of different channels which is annoying because then

people have like more things to look at it's nice just say hey it you know if it's important it just goes to this one channel um Sentry also has scheduled task monitoring so we have a lot of jobs that just run like once or twice a day and you're just going to assume that those are going to keep working until they don't and so Sentry's great because they have the ability to tell you like hey this thing didn't run because a lot of times if this stuff doesn't run and it doesn't even start to run you won't get any errors so you won't know that it didn't run we used to use datadog for this but we actually found Sentry to be

more reliable for monitoring when our stuff was unreliable um SLI is this this is like a really broad category but you can think of like latency to an external service is something that might fall into slis um application performance monitoring this is something that allowed us to figure out like what um what was actually happening in uh in Access service like why it was so slow I also these slides are on sketch so feel free to keep taking pictures but they're also up there if you want to just use the stuff that's up there um service best practices these aren't really metrics at all this is just kind of like in the same broad theme this is

having readme's that are easy to get someone up and running it's links to dashboards it's claiming the service in backstage which is what we use to track like who owns what I think of this as just the things that you do to be a good citizen in your uh in your organization so some examples of what we might track for something like off-board Service uh off-board service is a pretty simple service it runs once a day it boots everyone out of the segment application once they've been deactivated in Octa and it prevents ex-employees from accessing their accounts so for the health metrics does Sentry think this thing ran were there any errors if so what were they

how long did it take to run how many calls did it make to the OCTA API and is this service something that you know if somebody needed to make some changes they're not going to get bogged down and like fixing a bunch of unrelated nonsense and just like have a really hard time getting the service up and running switching over to product metrics I think this is the fun part you get to put on your product manager hat and think about why the stuff that you're building is valuable I think there's also a lot more art on the the product metric side than the health metric side those are much more science is your thing working or not

um but I think everything that your team builds should have at least one product metric if you can't come up with one reason why the thing that you're building is important it's probably not a good thing to build and if you've already built it you should maybe retire it um I learned a lot about this from a former product manager Rachel who I gave a presentation with last year um this I would say this presentation is like complementary it's it's not a rehash of last time but if you want to check this one out uh this one's on YouTube on the uh the b-sides page yeah it's actually the same brand and I have matching shorts for both of these

but I didn't know if people wanted that so I I wore pants all right I'm I'm keeping that uh next time I'll wear the matching tops and bottoms um so product product metrics um without any metrics you could probably say something like this at the end of the you know however long it takes you to build this we built off-board service it remove X employees simple um but if you throw even like a little bit of data in it it sounds a lot better um and so you can say hey not only does this help retain customer trust it helps maintain our compliance with our internal policies it helps with our sock too and in the first 30 days it

off-boarded 60x employees 48 of which were missed by previous manual reviews so you can draw some comparisons to the before State once you have this kind of information um for product metrics there's plenty of other things that could be a product metric these are just some things that I think are worth at least considering um if you can tie something to customer Revenue uh that's always a good one um if that's a little bit more difficult just having some data on like how many customers are using this either just raw numbers or percentage for internal tools it's good to try to capture um how many people are actually using something in a given week or how many

times a tool completes something that could be a bad thing avoided or a good thing encouraged you can also always try to tie things to team goals or or goals or company goals those are good because I would say generally those are things that other people should at least be kind of familiar with and so if you can say hey we did this thing that supports this other thing that the company said is important that is kind of your job so that's a good thing to to try to do for your metrics as well um some product metrics from SSO let's just say you work at a company that has 100 Mil in ARR you might be able to

track that 30 of your customers have SSO enabled um and you might find it somewhat surprising that so much more revenue is tied to something rather than the percentage of users but for something like SSO that actually makes a lot of sense because your bigger customers are the ones that are more likely to pay the SSO tax and so they are naturally going to shift this and then it gets even more uh the comparison grows when you talk about something like scam because typically you have to have SSO enabled anyway um if you haven't heard of scim and you work at a B2B Company please convince somebody to build it it makes corpse and it's jobs a lot easier it basically

allows you to do provisioning and deprovisioning and group mapping and a bunch of stuff so uh getting started uh hopefully you have a list of what your team owns if not that's okay but that's definitely Step Zero is figure out what you want to start tracking things about and then figure out what the current state of these are like maybe you have some metrics maybe you don't that's fine too and then you should have some idea of where you're trying to go you're obviously going to figure stuff out along the way which is great but just have have some path that you can plot from A to B this is an oversimplified version of something that we made pretty early it's

just a list of what we own and then like what the state things are in um more tiles and datadog or you know whatever you're using isn't always better I would say that it can be overwhelming when you have a bunch of useless data and so more or less is more here I think that you want to have the right stuff tracked rather than just like a bunch of stuff so for our implementation we have all of our services send stuff to Center your datadog and then we send stuff to Slack we also have an internal logging system that you know is used just kind of like for debugging stuff so that's kind of unrelated to this but also part of the

same project on the metric side we're very fortunate to work at a company that has very smart uh data engineers and analytics folks that have set up a bunch of this foundation for us and so we didn't have to figure out how to tie something to a customer account in Salesforce and get the ARR like there was already patterns in place where we could say hey this workspace is related to this amount of money um but I think that getting good with business intelligence software is pretty valuable for for engineers whether that's a security engineer or software engineer or whatever because it allows you to answer a lot of interesting questions about your business you can also use bi tools to augment

internal product functionality so we do this for for Access service which um as I mentioned earlier it's something where you can say hey I need access to this thing for a certain amount of time like a day or a week somebody else approves you and then you get access and then it expires but we often get asked questions that we don't want to build something in the product to answer this question we can just refer to somebody to a dashboard so somebody might ask like what are all the tiles that this person is an approver for they're going on Parental leave and we want to make sure that we have coverage for them and you know if we're

a commercial product we would have to write some database queries add some new routes put some stuff in the UI write tests but you can just skip all that and just write some queries and create a dashboard in Snowflake and you know it's fine if you make an internal person go somewhere else um so some product metrics that you might or that we thought about for Access service just to give some more examples of what an internal app like this might care about tiles with less than three approvers so if this is an OCTA app or a group that only has one approver and that person leaves that can be annoying to people that need access to that thing

um more generally if you have an internal tool trying to figure out how many people are using this um if someone's like hey why are we spending so much time on this and you can say hey a third of our company uses this every week that's a lot easier to justify spending time on that um something like median approval time allows us to do a contrast between the before and after it used to take hours or maybe a day to get access to something with access service it's maybe like three or four minutes in most cases and then the most active users are a great pool of people to get feedback from so if you want to build something

new or you want to change something like hit those people up because they're the people that probably care the most about what your app does um so let's assume you you love these ideas you want to do this at your own company you should take stock of like where your metrics are at before maybe these are all zeros which is totally fine that's gonna make you look really good at the end of the project um but you should have some metrics about your metrics so you can say Hey you know just like any other project this was a lot of work we spent a lot of time on it um you know this is kind of the the

before and after and this does get like a little bit meta of like hey we you know we put metrics on your metrics but this is just a you can just think of this as a project and your projects probably want to have metrics this one just happens to also be about metrics um pause for photos um so once you once you do this you should share your work there's probably a lot of other teams that want something like this at your company and some of the things that we did to make this easier every single snow site dashboard Sal can attest to this he built a lot of them has comments and so anybody else from

security or wherever they can go in and they can see oh this is why they did that crazy join that I don't understand let me just copy that because I want the same thing um you should have links to all this stuff in your readme's make it easy for people to find this stuff and then present it internally nobody's going to know that you did this stuff if you don't tell them that you did it and this is a good way to you know to practice a lot of things but it's also a good way just to make people aware of what's available to them and say like hey go go steal our queries they're also a great

thing to put in promo packets I think that a lot of managers get caught off guard when people start asking them questions about impact and stuff like that and it's a lot easier if you can just say like hey we made this thing 25 better 25 faster whatever if you have the data at least that's been my experience during promo and like calibration time is I feel like the the managers that have metrics usually do better um this is also something that you should start thinking about during project planning so we actually have a section in our template that's like what are you going to track about this thing and you don't even necessarily have to

have a goal like you don't have to say like Hey we're going to make this thing 10 better whatever I think it's good if you do but just having the raw data so that at some point in the future when you want to make improvements you don't have to spend a month collecting data you just have the data about like the usage and things like that you need to include time for this in your project plan if you don't somebody's just going to say hey the thing's built move on to the next thing unfortunately this stuff has to happen at the end because otherwise there's no data to build the dashboards and so it can be put into an awkward spot but I

think if you put this as part of your project plan it makes it easier to defend the time that it takes to do this um one last example this was something that um we had an intern two summers ago uh she did a fantastic job on this this project the goal was to get more people using MFA um more more customers specifically using MFA and the way that we targeted them is we went after the workspace owners and got the workspace owners to force MFA for their workspace because then that just gets everybody um and without any metrics this still would have been pretty good especially for an intern like hey successfully completed this project got a lot of

people to enroll but because we had metrics of how many people were using MFA before we could do something like this where it's like hey we actually saw a 25 increase these are the total number of impacted users for all those workspaces we had originally guessed we'd maybe see like a 10 increase so I don't know if we're just bad at guessing or just sandbagging but um yeah this looked really good during her final uh presentation some closing thoughts uh metrics make your team stand out during quarterly progress reports and help you supercharge performance reviews and they also make it easy to communicate cross-functionally um and it allows your organization to see how things are getting hopefully

better maybe worse over time but just having the data makes that a lot easier so if you want to try this for yourself I highly recommend using whatever your company already has set up and paying and is paying for um this stuff does mention like segment and Snowflake and whatever but all those are things that could be swapped out for like Tableau or like whatever else you're using um this will be recorded as part of b-sides uh the I wrote a Blog about this in December so if you want to go back and not listen to me again and just read through it uh pretty much everything in the in here is in there and if you don't

have this stuff set up and you want to play around with stuff on your own twilio and or segment and snowflake both have free tutorials you can you can mess around with stuff but yeah definitely use what your company is paying for and then lastly we got some other talks to walk watch my girlfriend's on a panel um with some other great people uh that are some of them are in the front row I don't see Louis but uh wow uh there's some okay people on the panel and then uh tomorrow uh we have some some current and and past twilio folks so we've got Kelly Robinson we've got Sal who I mentioned earlier he's talking

about some uh cool API security stuff we're doing and then Colleen right there our former CSO is on a panel as well so check these out the slides are are already up as I mentioned and then the blogs there uh questions or do I get kicked off one minute all right let's do it yeah the audience is counting on you yeah we have one question we have one minute to stop so if you guys something has a hot take for leaf raise your hands and sing it out loud

are you talking about that tomorrow or later today or no so my team doesn't really do incident response stuff unless it's helping out with something that that we own I imagine that there's probably a a place for metrics on any security team but I'm not a good person to ask about that yeah go to the panel they claim they'll talk about it

hey uh 100 right way too much so Snow's site is the bi tool that either snowflake comes with or or we pay extra for um and it can help you create like graphs and charts and like it it's basically like all the stuff that maybe you used in PowerPoint at some point growing up uh based off of the Excel data so once you have the data in there it's it's pretty easy and if your company has you know data engineering or Analytics just make them help you until you get good

all right folks that's been great Leaf will be out in the hall