Iuliia Laaneots - Nobody monitors the monitor: Chaos engineering for the security team's infra

[muzika]

[muzika]

So disaster if I would say this word what comes to your mind and I think for different people it actually can mean different things. It can be fire it can be flood for people here it can be something related to cyber crime. Actually I think for all of you this would be more relevant but ok in my opinion every disaster should have a plan and a plan usually has a tool. So for example if we pick some disaster let say fire what comes to your mind you actually immediately you have a plan and you know which tool to use. And you know I have heard that women's bag usually have all the tools needed in the world.

Let's check. So I think even my bag has it. So great. We have a tool. How many of you actually have used it? So we few hands, few but in case of emergency would you be able to grab it and start operating? And for those who actually haven't been using, this is the question. Ok this sit in my car but in case car goes on fire what I will do with options but I don't think that I want to go that way. So again every tool actually needs to be tested and you need to be ready to use it. It should be in your muscle memory. This is the mandatory slide about myself. Well, I don't want to stop and

tell you about my boring biography. And I decided actually to tell one fun fact about me. I keep the list of misspellings of my name and right now it has around 20 entries. So if I challenge you, if you will find the unique entry, I will buy a beer. Well, enough about me. Let's kick it off. So each term has a definition and disaster recovery also has a definition. Nist actually defines it like this a written plan bla bla bla bla bla bla bla. The main word is written. So not actually executed but written. So I think it feels like this sometimes and [sušnarpščia] you have a plan. You actually have versions to this plan.

Maybe somebody even read further than the title and the first pagin confluence. But this is this will not help us actually in case of disaster. So in my opinion there are different stages of planning the disaster recovery plan and for example having a backups which is which would be the first step. So if you have a backup it's already good. If you have tested your backup more towards the golden path would be that your backup would actually survive the vendor. And this is something that people don't think about when I have a backup, I'm golden. And if you have actually monitoring setup for your monitoring solution, this is already next step. So how usually disaster recovery plan on

the security team usually comes with creative idea of the management that now we are facing the new audit because we want to comply with this and that pick your poison from I don't know PCI ISO whatever you got the slack message. You have the task on your gira board and your gira board is already overwhelmed with tasks. toask hey help me out please draft the disaster recovery plan for the critical systems it gives actually pretty decent answer. I think it actually covers kind of the ok databases application servers. seems really decent but security people actually may start thinking but the monitoring system isn't it also important isn't it critical for our business and well even if we will ask

this question it's not it was not listed by our clanker and if we ask this question usually we get the classical pushback well management would ask the valid

money security team keep operating on the background serving the audit helping other teams up until the moment or one of these it can be incident you don't pick the time or it can be the auditor who would start asking you really uncomfortable

So now we are good. We have a budget do we know how to use this budget? This is another question. So let's figure it out. Second try the clenker. So now let's do the let's do it better right so let's ask already the disaster recovery plan for the monitoring platform and we get an answer well it also looks pretty decent because the snapshots replicas cool and even some RTOS RPO some numbers well looks valid for the management however these numbers what are these why f

[sušnarpščia] Ai pretty much crystal bold why because AI doesn't know the infrastructure good as we do so maybe it crystal b some of these things as well and didn't mention stuff that we needed to consider so let's check let's assume that we are working in a company we are working in team data sources of agents running hundreds of detection rules and all of this is in one system so if we would start applying some of the things that we were we will see throughout the presentation and already saw before well could we make this also secure for security team So let's assume we have a snapshot and the first glance it feels that well now

we are golden because we have a snapshot we can recover but have we tested it? This was the question that well if we didn't test then we cannot we don't know whether we can recover and usually people stop on the first question however there are more questions that we keep forgetting.

If you don't have an inventory then please create because when you need to restore even from the snapshot you need to understand how to map it and how to link it. Well, how long would it take to reenroll the agents? Because the crystal ball numbers actually would also be answered by this question. One more question. If the rules will silently stop generating alerts, well, the security team actually will be happy and finally can go and drink beers, but on another hand, it can be indication of the problem. So if we will unfold just the first part of the the snapshot part, so actually doesn't look that bad. It looks like a lot of green. So we are good. Some

machinery can be restored. Some data can be restored. But actually here two top rows are indicating data and one is green, one is orange. So what does this mean? Well, usually the pointer means that you don't own this data if it is stored in the repository of your for example provider. CM as SAS providers in providers account and I will use some examples using elastic terminology because at work we use elastic but it can be applied throughout different vendors just change the terms and the concept remain the same. So what does this mean that actually okay warm data from warm tier and hot tier can be restored but what about frozen? Frozen actually has snapshot has a pointer. So

if you will start restoring if your provider service provider is not responsive you don't have this data. So something to consider and this from where the first tradeof that security people should do. Sorry actually not about mysk but about the configuring snapshots. Well, the small bit about hot and frozen tier. expore [sušnarpščia] is cheap and you will be I cannot say struggling but you need to take into consideration the risks that come along with the frozen but usually as a security team we are really cost aware we are really sensible so remember we have this 60 data log sources so we decide to go with a hot and longer frozen what do after running it for example for a year

or so what we will find out that how much of existing data actually will be present in one place and it will be around 90% of course it depends how much of the how

big number because again the data will be present as a one copy at the provider place but 90% is is it a lot or not from the first glance it seems a lot but here I want to stop and make you think that maybe we should shift our mindset

SL B impact because SLA is a promise we will recover in hours recover what maybe we don't need to recover and restore all this 90% of data maybe our business will not be impacted so why do we need to do it however business impact analysis would help us to answer these questions. So actually which detections will stop firing which activities we will stop monitoring because we don't have this data which again what we will not be able to correlate this is something to answer plus forget the obligations for the audit and compliance requirements. This is also something to take into consideration so that 90% can be scary. But if you do the business impact analysis, it actually may shift

your mindset. I don't think that you will improve it like to 10%, but you will actually get the decent tradeof out of it. And to be honest, this also leads to the next part. So we saw four questions but I think this is the most relevant and should be the question zero. Do you have an inventory of all data sources mapped to the way how they are being ingested, to the reason why they are being ingested, to the obligations and to the dependencies. Well, and the purpose it shouldn't just say yeah for the auditason you actually need to be that precise and explain which investigative cases would help me this this data will help me on this

investigative cases actually and then you will come to the dependencies and dependencies are showing which rules actually are being built on top of this data and it seems irrelevant right now because like what I will list all my rules here however here is a caviat so for example you have a critical business detection rule however it can be based on multiple building blocks building blocks rules which can be not critical but can be actually based on another data sources in data source rule is ok then this data source is critical for us but all the rest which will trigger the critical one in the end are they critical do you also need to restore them and treat them as critical

this is your question to answer actually but this is something that we keep forgetting Because if we will just filter our Excel sheet for like restore all the critical detection rules and related data sources then half of your detection rules will not work because they are lacking the context of another data sources. [sušnarpščia] So actually our CM depends on multiple things detections we already talked about investigations compli one thing I wanted to highlight here the credibility still this human touch that we as a security team come to other teams and we try to persuade them to be reasonable and build the resilient system but if our system is not resilient who are we to tell

pretty obvious disaster recovery plan. So have a snapshot test the snapshot cool I could finish this talk right now but there are actually some ways that wouldn't necessarily mean that well there is a huge disaster they will silently start killing your visibility and I think these are the most dangerous ones because you actually don't see it straight away and you need to be ready for exent degradation it actually may look like everything is fine so your cluster is healthy everything is green but actually the data doesn't flow and this mean that detections don't work and yet again you think that maybe finally people started being reasonable and they don't do stupid unfortunately this is not

the reality and most probably the and in this case it would be nice to monitor the data not the control plane. again green but will monitor the data flows so you will have the injest rate you will have signals around there you will have canary tests that would actually be based on the data that you ingesting that you would be able to spot ok the detection rule trigger so meaning we don't have data you wouldn't just look at the numbers of your rules that's what auditors love you go and say oh we have 100 rules and 90 of them are enabled we are golden no actually you need to check the execution matrix how many failed how many gaps of data

actually it had while it was The second one actually kind of comes from the first one and also has some of the same things. So the incomplete picture sometimes the data sources can get noisier and it doesn't necessarily mean that this is the attack. Maybe just an error something somebody misconfigur something and you don't need to pay attention it however if you have a baseline for every stream what do you expect from inje can act actively already if you see ok this is a deviation or if there is a drop if you will buffer it outside of your CM in case there is a drop for example there was a lot of events

are able to and reinest. So even in that particular moment you have a gap. If you realize that you have a gap, you can fix it. So to be honest, in my opinion, the scarest part in all of this is not losing data but the fact that you don't know that you are losing it. And what about the compounding case? So usually we define that there is a toleratable amount of hours days whatever.

is compounding factor if there is an active security incident going on then we need to have visibility straight away so does the first number apply in this case not because you need to act here and now and usually

number and being ready with procedures that this is the time we launch we fall back will save you a lot of ner cells and well again we saw how we can tweak test improve it to make

architectation what can possibly go wrong and yet again it kind of calls back to this question zero so if we have [sušnarpščia] so to say inventory for our data sources and we can answer all the questions how they get to the system which detection rules they are mapped So then we can get the dependency map and if we have the dependency map we can actually build the priority of the data sources we can understand which for example logshipers need a bigger buffer how can we improve different bits and pieces of information of our infrastructure based on this information because if you will do everything according to baseline something will start failing in the most appropriate

rule building all of this that every bite should land in two places and it shouldn't be the replica like it should be two separate copies so if one fails if you cannot reach the one you need to have the second one and it should be accessible so if we go back a bit remember we have all our information about security monitoring system so that this is where we started that we have 60 different look sources so they can be ingested in the totally different ways they go to injeestion pipeline it lands in CM and it has a snapshot ok can we improve this like we already saw previously that this setup a of fling piece by piece.

So I think the most reasonable thing would be to actually move the snapshot to your account because then you have the data not your provider has data and in case of outage you still can do something not just pray because backup at the provider place is just the hope that you can restore it but I think it's still not enough again for

immutable object storage I think is cool one so if we will just put here all our raw data then some of this rules that I was talking about actually can be applied. Mhm this is better. Ok next one. What if we will put it also in another region? Why so I will not to waste spend money actually if you will have the disaster when the region is down and both of you immutable storage and snapshots are in the same region. Well, it's kind of sad. So this will actually help you to fix this problem. But I still think it's not enough. What else? We still don't have raw data from the agents and from all of that. And for

example in elastic case elastic agents do not have a capability to give the output to multiple sources. So in this case do we just live with it do we accept it or what do we do? We actually can fix it because we have already one place which knows one piece of infrastructure which knows how to act so to say properly. So in this case we just will send all the information to for example in this case vector and then follow the same route. This I think already looks pretty decent. Ok.

really secure right but because we are sure that nobody can temper it but did we think about reading it because usually we put a lot of effort actually to an emphasis that nobody can make changes nobody can delete

people and our data this is like it can be really sensitive data and usually this part keeps being forgotten at all nobody monitors who accesses nobody time boxes the accesses just ok security team you are really secure go and read it davai so this is not the thing which you should do with this immutable object storage and well this is a good thing to have it as well in case you for example you start the fallback process and security team is actually working on it but there is an active security incident and you need investigators your incident response team not to sit still but act on it so what would you do in this Well, you can give them access to the S3

bucket, but I not sure that this would be pretty good idea. So, you can prepare the alternative that your incident response guys would have already prepared queries, would have already prepared notebooks to go and query the most critical questions that we already were discussing before.

to move on with investigation whilor visibility in CM and detection capabilities but [sušnarpščia] what about backups again yet again so we have a backups we did this nice infrastructure We should just get it to the fresh cluster and that's it. However, there are multiple pieces yet again that we forget and when we just think that having snapshot is enough. No, it's not enough because when you start actually deploying it to the new cluster, these are the questions that you will face.

Snapchat save your key store before you can map the new deployment and the snapshot. If you don't have encryption key to your kibana then well you will restore your rules you will restore your objects but you will not be able to decrypt them so basically you will have something that you cannot even use. And what about the agents? So if we will start re if you will see that the agents are kind of failing but don't assume the worst in this case it actually agents are not dumb in case you are still seeing the data flowing and maybe even if your CM is down kill the agent.

work around and get all the data flow through them until you can also in case you just see them failing in your working deployment and you think that okay this is also a disaster don't start to reenrolling them because this will take a lot of time try to just restart

for example you are moving to the new deployment in this case I highly recommend to be ready and have an automation why because if you have hundreds of your agents, can you imagine manually going through all of them and injectesting new enrollment key. URL

if you have your policies in git if you have automation you can speed up the process tremendously and we talked a lot about snapshots but in this case if snapshot is not available or if the snapshot is corrupted or

ateutable object storage so we have data but then what about machinery we don't have the snapshot and then detectionist code will help you if you have all these bits and pieces also in git you can restore the working machinery

would be able to act straight away and this part I believe a lot of people forget because they rely on the snapshot. Well, and yet again, if you are falling back, it's not that you are switching, you are migrating and there are steps that you need to remember that you will not just cut off the connections and then well go back to my previous cluster and everything works automatically. If you will not follow the steps correctly, then you will lose that data that you were ingesting on the fall back deployment. And well, this is really intense slide and to be honest and the cadence which is up to you to decide when and how to test. Just a

proposal. It's up to you to decide according to your risks, according to the size of team.

test and can help your team in case if the disaster will once reach you. So TLDR well BII not just SLA as I said SLA is a promise.

degradations everything doesn't seem the big as a big problem these are the most dangerous failures that will bite you in the and everything what is untested is just your hypothesis you need to test everything and Aguditors will give you deadlines, vendors will give you snapshots, but only you actually know in the end what your infrastructure needs. Thank you. [plojimai]

Anybody have any questions? Hello, thank you for the presentation. It was very interesting. So, just as a comment, who are we to tell the OPS team how to do their ops if we don't do OS perfectly? We are the security team. We don't know ops, but that's just a comment. The actual question that I had was that did you test and considered elastic solution instead of vector? We actually to be honest so we are right now in the migration so to say period. So we use we used vector as a solution I cannot say as a legacy solution but we used it for some specific reasons some particular point of time and now we have already

the whole infrastructure built based on it and now to migrate everything from vector somewhere else would be way too difficult what we do we will use lockstash actually for elastic agents as an output to get it to vector and then standardized the

infrastructure need to be mindful what canonable and what we can migrate thank you and second question do you have any tips on how to scale the let's say the output let's say the log source monitoring

interesting source that have over 200 interesting log sources it gets really annoying clicks. Can you describe a bit more? I'm not getting so tips on. So how do you scale the log source monitoring that you still get the logs? Well, you can you still you automate and you have the alert basically. You have the some you will still do the clickups but something you can get via API and you can you can automate it and get the output basically either in elastic or let's say in some other tools. Ok thank you anybody else?

Thank you for the presentation. I feel validated. [juokas] somewhat follow up [atsikrenkščia] the the baselin data interest rates and again I feel validated but there's there's an interesting thing there a lot of like if you look at for example server logs and production system logs there's usually a steady flow but then sometimes you get into these interesting data sources that ab and flow

testing with machine learning rules and trying to understand the baseline. for extern we can build around because I understand what you mean that there can be this interesting one which can come with some spikes and it can be not indicating that it is a problem that we also have this problem and we are trying different things but what we are trying to utilize the well AI in some sort of way how to build the patterns and then based on at least lower to alert more or less let use this terminology is a good idea to if you talked quite a bit about doing snapshots and restoring from snapshots if you already have an immutable copy

from somewhere else and you have replay capability as well why bother with snapshots Because in snapshots will give you the pasal you within one try you can get everything data and machinery but with the raw data you will need to you you cannot restore or basically you can but it depends on the obligations for the compliance obligations how much of data should be actually visible

and for data sources it becomes a bit cersome to again so this is why I think still it's better to bother with snapshots because it will ease your pain at the moment when you will think that well it was useless tip there for compliance obligation

analysis for example in PCI they will still ask that for example month you need to show it here like three month is not that much but still for example buckets will not work but you don't have to show it from the CM is a ends on the auditor [juokas] for PCI depends on the auditor. Anybody else? Sure. Alr can you hear me? Ok. Thanks for the presentation. I thought it was quite insightful. I have one question around how you do logs source monitoring. Is that within the SIM platform? Is that external to the SIM platform? And then also one of the things that I was just curious about is the the way you're tracking the

reason for why you're collecting certain log sources compliance for detective rules and whatn if you're okay can you share a little bit more how the distribution looks like in your organization in terms of what are the majority lock sources used for is it for compliance or more for the detection scenarios? Um and then does that also in you treating certain data sources better than other data sources given you know what use they have within your organization I will start with the second one so the distribution is I would say 5050 and to be honest the ones for compliance the security team doesn't necessarily want to deal that much because we don't

opinion are doing out of it so trade of again to make logs if it's just for a compliance that then it's a bit shitty because well the requirement came from the compliance if for example for critical system then we should ask ourselves a question why don't we care about logs because compliance cares then there is behind it and also starting and I think this is the big question that we need to remind ourselves constantly that yeah we we may hate compliance we may get triggered by this word but unfortunately we need to remind ourselves that even if it's for compliance in the long run we need to find then these questions which we need to answer ourselves. Why do we need this

data and related to the this rate injection and whatn we use the outsource not elastic we use separate tool for it ne

so have you also considered instead of failover just keeping two running copies of the system instead yes but Ok.

Anybody else? Ok. Thank you so much. Ok, thank you for great cling [plojimai]