Zafer Balkan - Detection in Technicolour: Finding the Gaps Your Dashboard Cannot See

Heat. Heat. [music]

[music]

Hello. Hello. Hello everyone. Yeah, I guess you can hear me now. Okay, good. Uh, first of all, I need to note something before I introduce. There are no cat pictures in my slide. Sorry for that. Uh I am Zafar Balkan as I have mentioned. It's uh I've been in Estonia for the last 5 years working in Estonia, a payment provider. So I moved from NATO to a another regulated environment in finance unfortunately. So it's it's been tied. Yep. Okay.

Can you see? Yeah. So, uh this in this talk is going to be about the detection in general. So, it was a bit demotivating when if you see this uh clarify security tickets like no one gives uh a damn about your logs but unfortunately I'll be speaking around 40 minutes about logs. So, sorry for that. uh let's see what we what we are what we are going to see. So uh in this slide we are just going to check everything about practitioner perspective. So it there is no AI involved. There are no AI agents. There are unfortunately no not much buzzwords uh in in this part. It is mostly about um previous experience and what we can do when we believe

everything is working as is. So just like many of you I have friends, colleagues, ex-colagues and etc that I'm in touch who works in IT and cyber security and actually this topic came after one of these talks. So uh there was an incident and the the uh sock manager uh was there with the uh CISO and other team leads. The the problem was the attack happened. The issue happened but nobody saw it coming. They only see it afterwards. So the the question became why we couldn't see it timely. Why we missed that thing? So well sock guys say that yeah we got the CMS deployed we got EDRs and everything and etc. But it turns out they were not actually

collecting the logs at all. So basically just you install your whatever tools and etc. They did not have the let's say people communication part. They did not have the process of on boarding these things properly. So whatever the tools you got there unfortunately does not help. So similar thing can happen to you. A rule here I'm using the term rule in a very let's say liberal context. It can be queries, detection content, whatever your uh your environment is uses. Just I'm just going to say rule for the sake of simplicity and think about that a rule has not fired in a year. Well, it's an observation. It doesn't actually give you any insight. It is not

an explanation at all. So the question becomes then well does it mean nothing happened in the last year or the rule doesn't fire at all? There are are are there the let's say prerequisites about it? Uh are they there already? Do you have the checks for that? And unfortunately many teams don't have it. So I'm just here trying to give you minor hints that you can make use of. I'll try to demonstrate a couple of CLI tools that I I developed but you can just have it for your environment as well. So the idea is finding the right balance. So if you got too little telemetry so logs metrics whatever you got you will lose you will

have the issue of reliability. If you start collecting a lot, it can be just too much. So then you will have the questions, do we have what we need? Do we need what we have? So you might have some issues about this one. This is a very very simple abstraction of most of the uh CM security uh data pipeline, security data links, whatever you name it. Uh to be honest, I tend to use this duck test. If it qus like a duck, if if it works like a duck, it is a duck. So whatever the buzz word, whatever marketing term, if it collects logs, analyze them, give you an alert, it's a CM. I don't care about the uh new

buzzwords around that thing. So generally this is the pipeline the source well events happening at the source in the endpoint in the uh network device in your cloud environment in whatever tenant you have or the applications you have uh they need to create the logs. So basically at that point that is unfortunately out of your control. So whoever develops that software Windows, Linux, I don't know engine, Apache or I don't know GitHub on you your GitHub tenant on cloud it is the developers who creates those audit logs. So you cannot touch that part. So you have to accept that as a fact and you need to consider that you are bounded by a limit. So this

is a hard limit that you cannot uh you cannot do anything. Consider that it's 2026 and Windows just uh Microsoft just recently last month uh released a couple of um changes and announced new NLM logs. is 2026 and that we are still talking about NLM and unfortunately this is the case of the security that we are we are bounded by the legacy stuff and and we cannot get rid of and for this we need to get uh better sources and the better sources regarding the logs and well Microsoft is at least doing their part a little bit late but still uh Then the rest of this part the collection to the correlation belongs to whatever the tool you bought were you if

you are using open source you tried you installed on your production environment and you've been maintaining with so many people uh instead of instead of paying for the company you are paying your new team basically but each of these steps have a problem because each of these might have a misisconfigurations. Uh they might be correctly configured but it doesn't uh meet your expectations because not everything is uh logical. Some of these are semantical. You need to understand your context and computers don't understand context. That is that is one of the problems what you want to get. So uh here I use the terms upstream and downstream and in order to understand from left to right uh

consider that an upstream to downstream flow. So basically if the log is created and you are collecting you are fine. If the log is created and if you are not collecting it well you are losing an opportunity of detection or forensic value that does not help you during the uh detection but after an incident happen and you are trying to respond. So it does have some forensic value. You are losing that if the log does not come does not come to your environment at all. If you're at the first stage there's not much you can do. If you're in the second stage which means your agents or whatever it is your syslog forwarders whatever they are they if

they are they are uh collect they are not collecting this source the telemetry that you have then you have another issue basically you are losing an opportunity so and the last one well if there is no logs and you're not collecting you are fine so this one is the good part actually you don't have to care about that So generated collected you are fine. Generated not collecting you are losing you might be losing. The the problem is that in this slide I will not be speaking with very specific sentences because it will be mostly about may might can. It's all about probability because it's all about semantics, your organization's context, your risk appetite, the risks. If you

are considering if your uh advis uh adversaries as the uh APS then you might be you might need some more details. You don't have to stick to logs. You need ETV. You need EBF on Linux. you need more capabilities and you may even stop being uh passive just through CM and you need to do some uh state specific checks continuously etc to create alerts for you. So most of these toolings handle it for you but you need to think about what you are covering. [snorts] So the others are the same. You got the logs. Okay, they are collected but you don't have you or the default toolkit default uh content of the toolkit that you have.

They don't have the parsers or something. So it doesn't actually populate the the intermediate uh content that you need to build your detections on top. That is why it is either depending on the context it is either a parsing gap. So you are losing unfortunately the opportunity to detect the attack. Yeah, you are collecting it but you cannot make use of it because you are missing a couple of regaxes. This is funny but this happens. Unnecessary collection is also happening because you are just collecting so much logs. For instance, we had just Kubernetes instance and I made a mistake in my own environment that having a big back big bang change and just enabled everything on our on premises Kubernetes

environment and we got hit by 12 million logs a day. So basically all our storage was yeah gone. So basically we have to I have to be quick to to filter out what I did and to be honest we just got rid of n over 99% of it because most of them were unnecessarily noise but they blocked all the uh buffers all uh anything in between those flows unfortunately they were spammed by that noise and I lost we lo we possibly lost so much important uh opportunities that might be actually traces of an attack or the early phases of an attack but we totally lost because your computers, your network cards, your bandwidth uh your uh network provider

all the software itself has some limits like uh by default Microsoft uh drivers they even they have uh the number of packets per second limitations. So if you believe you're collecting everything possib you need to check it because you might be losing half of them just because your network cards buffer is already full. So those kind of things that does not belong to sock analysts, sock engineers maybe maybe CIS admins but mostly related to sec uh network guys but they don't know that because you know the joke network engineers say that we make packets flow security guys make packets stop. So there is a difference in perspective with the CIS admins, network admin and the security

guys that is this attitude makes the different difference that's causing some of those issues. That is why at the beginning while we are gossiping about our my friend's company well we were mostly complaining about other guys but the issue is the other guys do are doing their job well. You are doing your job well. You just didn't do the collaboration properly. the communication matters there. So it's not about tooling at that point. You need to handle that part properly. I'm not going to mention that part because I already spoke a lot. Excess telemetry, bandwidth buffet, compute pressure. You're just going to uh well it can be you can you might call it denial of service. Uh some call it uh

that by hug or something. So basically you are just strangling your own resources basically through I don't want to call it ne negligence because you are trying to do your best most of the time hopefully uh but there might be some things if you don't measure if you don't check it you might be losing them. So you say you if you believe that yeah you're checking agents are working network it's all green you we are using XY Z monitoring solution ingestion it works the the the dashboard of the tools that you have in cloud on prem doesn't matter ingestion all green partial we got this the the company I don't want to name them but the so many

companies they provide them even open source on you can find many of those on GitHub uh you can find them they look nice Okay, storage. It looks fine. Logs are stored. There's no problem with storage. We got enough space. And well, when the auditor comes, we can show him. So, it it's green and well rules and everything we got enabled is they are they look like they are running, but it doesn't mean you're actually detecting the stuff. This is actually I don't want to call it security theater, but this is actually a security illusion because you are not doing it uh deliberately. It is an act of due diligence. But the problem is the trust because trust is a

social concept. It's not a technical concept. And you are basically trusting your tools a little bit more than you need to. So reliability at this point starts with the requirements. Basically you got this log bas um that you need to collect. This is uh many of if if you've been in there if you're not using sysmon one you will you are possibly using uh 4688 uh security 4688 is the event log for new process creation this is disabled by default and in uh in that case that I mentioned at the beginning it was about this one specifically but the problem is that the security guys don't know how to administrate Windows environment. So they cannot provide guidance to the

people and basically CIS admins don't know what the security wants. That is why the first part of the logs whatever the process is running there is a log there's a record that's fine but they forgot one thing the command line collection because it is just another switch it will take possibly just 2 minutes after tickets and all the procedures have gone after everything the actual operation takes two minutes just to open uh the group policy management just enable one more thing the command line logging But nobody told CIS admins that and security guys assume it is there. So basically they had the rules they had the capabilities but just missing a very very minor detail in the source

configuration. So they were actually failing very very beginning at the pipel of the pipeline. That is the biggest problem and that that that's actually unfortunately a real incident. And after that basically they just fixed that of course they just learned that they there is an issue but they they they lost the ability and unfortunately you cannot take time back. So at only after some point you can manage to get these kind of things. So basically on your on paper on your hardening guidelines, configuration standards bas you might be doing everything right but it's actually may not be that that good enough or it may not be uh satisfying your security requirements because security requirements the risk the

threats they are an they are not a standard block. You cannot just download somewhere. You cannot download security. It doesn't work that way. So you need to communicate and you need to understand each other's how the other people work. So you can handle the uh you can handle the uh monitoring, you can handle the uh at least security monitoring part. We are not even talking about the response part here. We are not talking about the threat hunting. We are not talking about forensics. We are really at the beginning. we are at the detection even at the collection we are we didn't even start analyzing stuff here so that is the uh that should be foundational I

don't want to use the term basic because when well when you say the basics of this detection people thinks it is just trivial it's better to call it foundational so these foundations unfortunately have some issues again source configuration gaps everything uh in Every step you need to check but manual checks that's a good start but you need to think about how are you going to handle that in an automated way. Some unfortunately not all solutions provide you that information. What they provide you eps why? Because they they charge you for that events per second or gigabytes per minute hour day etc. Why? It's basically a phenops at that point. uh if you're using cloud or

if you're using an on-prem solution does not matter and even some of them unfortunately stop collecting logs because well you're out of your kota and if you had the mistake that I did with the kubernetes part you got this 12 million logs in a day basically you will be you will have just one one and a half hours of logs of uh of that day you'll hit your kota and rest of the day you are blind why because of your license so this is unfortunate to the real life. Uh so you I am trying to give you a couple of more idea. So there are two ways you either collect too much or too less. So you need to get to the source.

There are two alternatives. You check your rule. You try if your rule is triggered when you give the correct correct context. If it's a single log, an atomic detection, several logs, a temporal, so a time based collection, you it can trigger something. But if if you look at the other way at the source, you might be collecting so many stuff, but you might be losing like 3 four 5% of it and you might be actually dropping 95% somewhere in the pipeline. So this this is where this decision metrics make sense. So this one starts with the logs. So you check your logs. You try to get statistics about them where most of these things are fortunately not

provided by default. You either have to uh get to the marketplace of these tools or security hubs of these tools. You need to download some third party thing. You need to find some some some guys scripts on github or gists just uh to download and collect some information by yourself. or uh you craft something some something to you some something that would work for you to get those statistics to get those numbers that would make sense for your environment and I will try to demonstrate a couple of them u but when you get it why you you see that you are losing 90% of them which ones you need to care well parato apply applies 20% of them solves the solves

the 80% of the problems. you focus on the most uh you you check the highest numbers because you don't you shouldn't care about the CPU I don't know temperature of some router or some somebody's laptop you shouldn't care if that person is playing a game basically you will be bombarded by CPU temperature logs so you need to cut them off but you need to know if you're actually cutting them off that is the question so then you see the logs you try to find out. Does it have a detection value? Does it have a thread hunting value? Does it have a forensic value? And the b most boring one, does it have a compliance value? Because I've been I am getting

audited four times a year by third parties just for four different PCI standards. And I'm excluding finance inspection. I'm excluding the government. I'm excluding our customers. and I'm excluding the parent company the the group that my company is part of. So there are many standards that you need to oblige and even if there is a security value or not you have to collect some of them. So you need to understand those and based on that give you keep and measure and if you think here uh there is a security value then you can you you can proceed you need to understand the purpose and build capability or you understand I need to express that thing somehow but you need

to document it properly like I don't care how you do it nobody cares how you do it but if you can do it in a configuration if you can do write some comments on the configuration saying that we start dropping these because these are totally useless. You can use black blacklist approach. You can use white list approach. Just you can pick what to collect or what to drop. However you do it, just do it for yourself. Here I just named a couple of shiny new shiny stuff like mitro attack and this detect tooling. They are all again semantics. No. Well, maybe LLMs can give you some hints, but it doesn't mean they will be true. But you can just check the check

that log and say that do I actually need this? What micro attack TTP does this thing apply to? If you think it doesn't apply to at all, then basically you can eliminate some of those the thread hunting detection incident response maybe compliance might be an outlier there. So you need to think about it and then you just need to filter them as soon as possible. So it shouldn't leave the end point. That is the biggest issue that you need to handle. Uh I'm just I already spoke a lot about this part. So here I'm just going to get to the demo. Hopefully because I haven't sacrificed anything for the demo gods. I am just hoping that it will work. Can

let's see.

Okay. Yeah. There. So in our scenario we have gotten some logs from thanks to our incident. This is just a uh use case. Uh our uh threat intelligence guys managed to find out an incident like in this case uh we have this uh known attack P and they managed to give us the attack chain as well a lot of logs 549. It's not that easy. But what would you expect from those logs in your environment? In my case, I built a basically Python project with libraries and everything in for my environment because my tooling doesn't allow this. And I needed to build this rule before. Uh but what I did, I hated this thing. What I did was I basically copied all

these 540 something logs of course in JSON format. It's hard to get it in binary in Python. And what I did I created this toolkit for for myself and it's open source for for other users. What I did it just gets the logs sends it to the test socket of my tool because it has this ability. You can just just almost all of those tools they provide a log replay capability. So you just give you just send the logs and it will say tell you I don't know what to do with it or I can parse it but there is no rule matching this thing or yeah this is triggering this alert that that you that

you created. So what I am doing is that I'm just getting the results. I'm just checking if here on the right hand side well on to me uh rule level 12 or above in in uh I don't want to get into the too much details but in wo you got this content where it is uh from 0 to 15 so 12 or above means yeah there's something serious so over those almost five uh 600 uh logs I'm expecting there is at least one over 12 alert that I can actually get and this is actually the default of the uh was you rule set in my environment. Let's give it a shot. And by default well we don't get any

over 600 well almost 600 uh logs and we cannot get even a high level alert at all none of them. So what we do, we check them. We try to find out what might be a helpful thing. It is the manual part. It's not that easy. So you need to get into that part manually. You check them. And in this case, what I have found is in these rules. Let me show you. I need to rename that thing. I don't want to spend much time writing typing by hand. So they were using uh we were ver manager as system but from a user control path. So basically this is just another lob bin uh that that you've seen in many of

those attacks that is actually assigned Microsoft tool but they are actually using it uh to download the uh to download the payload and they are running it as system at this point. So uh we shouldn't be seeing that thing in a regular environment. So this is the rule I created and I mapped them to the miter ttps uh if you remember that um decision tree that I shown you uh a couple of slides ago. So I just added yeah I forgot the name mini plasma was the exploit and I just try to tag it. uh so it's it is privilege escalation it is masquerading and it is the name of the uh exploit so I can just find it later

so it is just giving me some context and what it does is that it's just getting any event log the the one at the bottom is at the top it says that the the number means there any windows event log that has these fields but unfortunately here in Wazu. What I need to do is I need to run a command so that the sorry that so that the log uh so that the new rule that I created here is reloaded to the memory. Then we will try we will try together if it is if the test will pass. So do

Yeah.

Okay. What we expect is that after we write this rule and we reload this rule uh to to to the system and one one of the good things is that I'm keeping them in a git repository so that my teammate and everybody can see the detections the rules and everything as if they are a code. So we have this CI/CD pipelines but I'm not going to get into much detail but still you got some versioning. People have justifications in commits PRs whatever it is it but at least you will see some stuff. Let's see if this can solve anything. Yes, fingers crossed. Now you see that uh this is just regular Python test. What happened is that

at least one of those Let me just get to here. Sorry for that. It's hard for me to see. Yeah, at least one of these 600 logs matched this one and created an alert. So I know that next time something like that happens, I will at least get alerted. I will not be blind within hundreds of thousands or millions of logs. This is one of the things. But this is something we started from the logs sorry uh from the threat. So from the rule level. So that that is one thing. Let's try something else. I'll try to demonstrate here.

Okay. CD demo 2. Here I got the archive logs collected. So what you can actually do most of your logs are archived somewhere. Some of you collect them into text files. Some of you are collecting them in data lake. Some of them some of you are collecting in some database that is part of the tool. So you don't actually have internals. But you can technically export them or run some analysis while it is in there. You don't even have to get it out. in this environment. What I did is that I created a CLI tool for me. Well, it was a script but uh before this presentation I made it a tool so that well everybody can

make use of it instead of just me for years. So it what it does is that it is just loading this thousands of logs maybe millions it depends on the case but here I don't want to spend much time. So what it does it is using some log mining techniques. It's using some some uh in-memory databases uh in the back end is is a bit complicated. What it does is it is extracting the logs getting the distinct patterns saying that yeah all these like 7 70,000 uh logs you got actually around 4 500 unique types of logs. The rest are just patterns. As you see it found only 80 types of it. It's not much what I have

find it is just going to be too much. So what I did it is actually you it's actually like this. Give me a sec. uh you can actually do it like this like I don't know and check the text but in this case it can actually I managed to get it extract as a and as an HTML file but uh but I don't want to spend more time of yours so let me just show it directly it's already open here so there is no malformed input in my environment decoder failure 6% uncovered basically the the the log arrives until the analysis but it is dropped and the 85% is actually below threshold. Below threshold is a bit different in our case

because it might mean something that is not so important. It might mean something deliberately suppressed but it might mean the default rule is there and you didn't have time to have a look at it. You're actually losing everything. So this below threshold part is a bit risky. So you need to you need to know you need to check what's going on. So I created so much statistics what's some of them are suppressed for instance that's a good one uh but not all of them. So these numbers by themsel don't mean anything as long as you don't check them one by one. In this case for instance you can see that there is a register event that happens

that is a user is actually adding yeah they are actually using the payload to edit the registry key but that is suppressed why we don't have any rules to detect this thing and this log is collected and a very very low level alert is triggered but it is so that it is not even shown on the dashboard because it's considered noise. So basically this is one of those things and this is basically tried on an act uh like lab environment and we we just push those kind of things and we lost basically the uh p the patch has been published. we are lucky. But in real life, you may or may not have enough time or you might have a window to patch

stuff like 15 to 30 days if you are lucky. And I've seen people and large companies that are still using Windows Server 2000 free, no extended support from nothing. So that thing happens. That is why unfortunately in real life uh things are worse than you expected. Let's get these out of sight. Get back to our last slides and basically just finish this talk. We got less than 3 minutes. So there's this technical state. Yeah, you can get the statistics from what I what I show you. You can they can be missed, they can be sped, they can be like it can it cannot be parsed. It is not even collected etc. You need to check it out. Uh

so it can be malformed, it can be decoder. Well, at the end it might be unexpected. So you need to spend some time and you need to find the the the purpose. You need to find the value. You need to find the security or whatever aspect you need. You need to find the contact. And if there is no value for you, just filter it at the source. Don't just spend time with it. Yeah, you can use the cheap stuff. keep evidence well you can have reduced window etc but there is always problems with this thing you need to get rid of so what I'm going to tell is on Monday start with four questions what required telemetry you

are missing where does this processing fail uh which collected telemetry has no known consumer you're collecting but not using it is it valuable enough to keep so find these metrics count them and find the gaps. Here with this talk, I'm trying to make unknown unknowns to known unknowns to you so that after you measure these stuff, you can at least make them known known in your environment. Thank you.