Guardians of the Logs: Monitoring SaaS with the Event Maturity Matrix - David Tocco & Josh Rickard

all right um we're going to talk today about our well thank you for coming I really appreciate it um yeah I've been here I think every year or at least the last several years here presenting about random stuff but today we're going to talk about Guardians of the logs uh we're going to be introducing a new product uh new open source tool called the event maturity Matrix we'll tell you what that means here in more detail am I going to do all right just making sure it's working uh my name is Josh Rickard I go by Ms administrator on Twitter GitHub you know all that stuff uh I am a senior software engineer currently at app Omni where I work on their threat detection Pipeline and product um where we process like billions and billions of in a day um okay I'm Paul I'm not David uh I do threat detection what threat detection add up Omni um but actually we do owe a lot to to David he did a lot of these slides and uh he was supposed to be here but he had a scheduling conflict so I'm Paul not David all right so do you use SAS applications there's only one of you come on all right I thought I was going to have to like skip past this slide but maybe I just need to like stand there now all of you are liars I know that every single one of you uses this s app in in your life you have a smartphone unless you're like weird and you have like one of those you can only call phones whatever the hell that's called um what's it yeah yeah well kind yeah there's like a anyways there that's probably probably a sess um but yeah some stats uh so pretty much everyone's using stats um maybe we have have the few outliers in this room but uh most Enterprises are using 130 or more on average um which is 85% of their overall software usage so most apps are SAS not even sure I'm not even sure what's still on Prem but there must be something um and it's a $200 billion Market uh by the end of the year so lots of sass lots of money um some pros and cons we sort of have this laid out so it's like what are the benefits of SAS why are people using it and then what does that mean from a security perspective so like how has this sort of changed the landscape of uh what you have to monitor and why so um the first is rapid deployment uh you can sign up for uh sasap you know Salesforce Microsoft Etc just by filling out a form I'm sure you know this um and of course that means people in your organization are doing the same thing so you Pro like unless you're using some sort of like SAS Discovery and even still maybe then uh there's SAS apps that you don't know about there's data in places that you don't know about um and then related to that is sort of accessibility so once it's up and running it's easy to get to it's easy for you to get to it's easy for anyone else to get to um so obviously that's that's a benefit and it's a drawback so it it sort of it moves the traditional sort of monitoring boundary out to all those devices your phones your browsers uh whatever your Western Union um and with that it's sort of um it's difficult sort of to keep track of who has access so you know if if you're integrated with an identity provider maybe it's a little bit easier but if you're not then you're sort of provisioning users you're responsible for de-provisioning and it's just sort of hard to keep track of all that uh and then integration so this is like you know if you think of on Prem if you wanted to sort of move data between applications you were responsible for export in it from one place importing it to the next or build pipelines and all that fun stuff um now there's pretty powerful Integrations between a lot of applications so it's as easy as just sort of point and click Connect app X to X to Y and the data just sort of moves around and uh the drawbacks there are kind of obvious but but also you know um there's the permission perspective like you have to know what are the permissions that an application should have um you know to to sort of understand what your scope should and shouldn't look like and that's not always so easy also origin sort of like do you trust where your apps are coming from how do you validate that um how do you make sure that those like Supply chains are are are safe couple more um customization is an obvious one if you've used any of these platforms like you can you can build powerful business logic applications you can also build totally crazy things um I'm sure we've all seen like security platforms built on top of SAS I've worked on teams that used uh a CRM for incident response and things like that so there's all sorts of crazy things you can build which means that it's really hard to know like what does a safe build look like what is a safe configuration and then how do I know when I've deviated from that uh and it's sort of a a continuous sort of evolving picture of like how should things work and how are they actually working and then related to that is just the the data picture of you know with with all these things uh comes the the data that you're using for your business processes so really easy to get data in really easy to get data out and then what we're highlighting and what we're going to talk more about is like where can you monitor those processes how do you actually know when someone's putting data in what does it look like when someone downloads all the data um what does it look like when people modify data and things like that yeah so and does this look familiar to anybody has anyone ever seen this this is a shared responsibility model has anyone heard heard of that okay so the shared responsibility model if you're not familiar is that on premise type uh infrastructure you know you had to worry about you had to worry about the physical you know data center uh actually purchasing Hardware networking all the different layers of um you know software and how it works so you have all the on premise and from a user perspective you're always kind of responsible for a the data but then also the underlying infrastructure as we've kind of moved away from a data center to more uh I or infrastructure as a service um you know you take away some of that with the physical security and the host and the infrastructure but uh you still are responsible for the operating system still having gold image Imes um so making sure that they're patched so on and so forth It's all still your responsibility the opposite is now Amazon or Azure or wherever then you move into the SAS world right and this is pretty self-explanatory but you don't have to we've kind of extracted away the operating system and the patching and and all the other components but there's still ones that we're trying to make aware that that if you're not doing you must be doing is still protecting your data in those SAS applic um anyone here uh monitor all their SAS applications and all activity all right well this is a kind of called to action let's just say you need to go and start doing that now uh it's not going away you know obvious obviously Paul you mentioned um what is it 130 plus different apps I don't know how many um you all have in your environment but I know that there's uh quite a bit and they continue to grow so with that shared responsibility model you know we weren't really um we didn't really care too much uh we cared about the data but it wasn't like a huge concern and a lot of people I think now just don't worry about that when it comes to SAS apps like authorizations taking care of by OCTA wherever your IDP is um and that's pretty much the extent of it but there's a lot of data in there and it's being used in a lot of unique ways and we need to keep track of it and protect it so a long time ago again we didn't we didn't really care when you had on premise apps uh we had some logging maybe you know IIs or whatever kind of web server logging you might have a proxy logging you know things like that but in SAS world you don't really have that so um really we just nothing's really changed uh that much and you can kind of see that and got to love memes right oh like they're literally laughing at you uh so no for real though like with SAS applications and and monitoring we need to actually start detecting threats in those environments because that's where everything is living from now on so challenges right from a security perspective who anyone here do detection engineering or that's part of their role thread hunting or anything like that all right well um woohoo the standards right there's lack of Standards when it comes to to security monitoring especially in SAS uh all those products are completely different and one of the biggest concerns from like a detection engineering or just security operations perspective is the lack of that schema or that standardization across it so on top of that we have the collection the difficulties um not all these products are the exact same they all have different methods and some have many different methods we'll share some horror stories here in a minute but uh overall um it's it's a huge concern when it comes to collection and how do you collect when do you collect what is that duration um all the other kind of intricacies around that then visibility again a lot of these SAS audit log providers or service providers don't really offer up those logs or don't advertise them and we're going to show you how you can um well you can use this new framework to kind of build up your detection and response especially when it comes to SAS applications and just cloud in general okay cool yes so we have some uh some horror stories or some examples we didn't put any names on these um maybe you've seen some of them maybe not um the first one is an application um generates audit logs it generates login logs but the the MFA events are sort of stored on like a web report um that you can only access from the application I've actually seen this in a couple places where like the applications were built for the purposes of like generating an audit Trail and then like the security stuff is just like it's available uh as a web report but there's no easy way to get to it um another fun fact about this one is that all the events are logged in CST and that's that's specifically CST it doesn't change for daylight savings has anyone seen that okay interesting that's kind of an obscure one but future in the future yeah we talk a little bit about about timing too on the next slide um app this is kind of common too like so they they have a standard audit log but there's no authentication events in it I already sort of talked about that but it's a it's a common sort of pattern where the apps are built as sort of like an a user audit Trail without a focus on security security stuff is bolted on afterwards but like maybe not so easy to get to um this is a really common uh app you've probably seen this one there's there's good logging but for some reason they don't include IP addresses you have to sort of log in and turn that on um uh yeah so these are sort of patterns that we've seen across a few different ones um sure you've seen it too different licensing tiers so there's like a there's a standard audit log that maybe has like logins or or something and then if you want more you have to pay extra um which is like maybe part of the like SSO Security tax or maybe not maybe like a different service um there's there's plenty of apps that sort of don't provide IP addresses across the different event types that they log so you'll have a a login event and that will have an IP address but then later on the user is doing some things they're updating objects they're downloading Etc and there's no IP address on those logs and there's no session ID so the only really way to Corel relate that data is to look at like time stamps and usernames and that that type of thing um and then yeah the timing one so this is again pretty common you know there's significant latency sometimes in how logs are delivered by Design you know they'll they'll document that to say like don't expect these logs for for up to 12 hours um and then there's other services that make no guarantee regarding you know like no sort of SLA so the logs are eventually consistent we'll eventually send them to you but every time you pull you're going to have to dup to figure out what you may have missed um so this is just a picture of some some notable sort of breaches from the past couple years it's definitely not all of them I think um you know the points worth calling out are just going back to that 85% number so if you realize that 85% of the software used within Enterprises which I think is pretty accurate or roughly accurate you know you you'd have a hard time finding a breach that at this point that doesn't include a SAS app especially if you're including like identity providers um so there's been a lot recently I think the the other thing worth calling out too is that like sometimes it's not so obvious like the the big platforms get called out a lot Microsoft Google there's been a lot even in the past couple months but sometimes you'll be reading a a report and it'll just say like you know customer contact information was was lost and where's that coming from it's probably either like a support titing system or CRM or something like that yep some headlines I'm sure you've seen them okay so sort of recapping the the history um and the sort of high level problem basically just you know SAS is everywhere um it's over overall good for business but brings a lot of different sort of security challenges that that maybe teams have to adapt to um governance is is very different um or maybe not different but more important right since we talked about how accessible rapid deployment that type of thing it's it's more important to sort of monitor these things um you know audit logging is sort of a mixed bag you don't know what to expect you don't know when an app is going to generate a log and you don't know what that log is going to look like and and how to get it so um and then the last piece is is sort of obvious comes with the comes with a big picture which is just that you know SAS breaches are only going to be more common um that's it really I mean there's so much data in there and everyone's using them and so you can expect to see more and more I love this meme why shouldn't we have another security Matrix right well um I'd like to introduce the you know because of these problems that that Paul is talking about right with with again the the prevalence uh of SAS just growing in environment we're trying to a raise kind of awareness of hey you need to be monitoring this stuff because there's a lot of uh a lot of data that uh organizations are moving around and there's a lot of Integrations there a lot of third party and you don't know what's happening to that data and if you're not monitoring this uh environment um well yeah you should so that that's kind of the the big point but let's introduce SAS a vent security or sorry event maturity Matrix jeez messed that up so this is really a a website when we'll show you and we'll kind of go through and break down all the different components and and all the data points that we we'll show you but SAS event security Matrix or the event maturity Matrix is basically a knowledge base of a centralized documentation as well as the nuances and it kind of explains the nuances of uh different audit logs from different products into a normalized and standard way um we provide examples and we also again that visibility mapping and we'll we'll kind of walk through and show you what those mean but but really it's a uh service or or um a framework that we can kind of expand and kind of normalize these events across different products in SAS environments within uh that that are applicable to security operations so let's talk about it centralized documentation so we'll I'll have the Links at the end but if you actually wanted to you could now it's event maturity matrix.com and it is a web app that has this data driven uh definition framework so but it but really it's all about having current up-to-date information about these different uh audit events in a centralized place as well as explaining the nuances of how the API Works um maybe there's some uh problems with the collection and we've kind of noted that from our experience and other people's experience um we also have licensing right again some of these horror stories that Paul is talking about they they uh these vendors charge for certain audit logging features and uh so how do you protect you know against a product that you they don't you have to pay for more logging to get basic logging um and then you have latency and retention right it all matters for for live detection uh if their latency is a day uh doesn't really help you much but um so it it kind of gives it in the centralized place where it's all organized and Visually well visually appealing and easy to to digest we also again that visibility mapping um one of the coolest things I think is that we've taken a lot of these events and normalized them to uh Central core um event types and categories so category is going to be and we'll show you examples and go through demos but let's say authorization or authentication all those are different kind of buckets of events and then we also have the normalized event types that we have mapped uh different products to there's also the supported and unsupported uh this both is this is both the activity but it's also even down to the field level meaning um you know you receive uh let's say a log event that's in Json we go down to the actual field level in that response uh and map and show what data is available in that log and what is not available in that log um for instance if an IP address is in that log or not and then again we provide also these real world examples every single event uh mapped to event maturity Matrix it has provided examples at least one possibly multiple just depends on the service U so the one of uh the great examples is let's say Authentication and you have um you know when you create a user creating a guest user versus creating an admin user completely different logs and completely different views of how you could respond to that event all right so we've talked about you know the kind of the history and we've talked about the basics of event maturity Matrix but we wanted to kind of showcase some of the scenarios or or different situations that this would be beneficial right incident response when you're in an incident do you want to be reading documentation probably not you don't want to be digging around going behind pay walls all this other stuff so we wanted to provide a centralized place again that that you can view this in a very easily consumable and digestible way to speed up that that process we've also you know with uh the threat detection in general how are you monitoring how can you collect this data where do you collect it what is the format uh and then all the other kind of details around that event and uh We've also have you know compliance right now you can visually see um if you're providing the data that that is needed for some compliance um that I'm I'm not a compliance guy so um you know requirements that that are that need to be met and then obviously the SAS evaluations um one of the goals we don't have this currently but uh you'll be able to visually compare um or contrast different products against each other so for instance if one product doesn't um log uh basic authentication events um but another one does maybe that's advantageous for your business or for you to go with one product over the other vice versa um and also the unsupported fields and values and all the other kind of intricacies that are that go into the decisions yeah I think that the last example is a really interesting one because like if if you've ever worked in or or just seen the work that like a risk team does