BSidesSLC 2025 - LLM-Powered Network Intrusion Detection – Taeyang Kim

Okay, perfect. Okay. Um, good morning or good afternoon guys. Um, so u today I'm gonna present about the LM power network intrusion detection and just to a quick introduce myself. So I'm a currently machine learning engineer at pattern and I'm also a current like MSCS student at Georgia Tech and also just grad from BYU and I was able to uh seven different hacking competitions in my life. Um so let's go to the next one. So the first thing is like the evolving of the thread landscape. So cyber securityities attacks are becoming like increasingly sophisticated and frequent and some of the like currently like LLM and also machine learning fields and in this like era the problem is they're using those

tools to diverge into our system. So as you can see from the chart um these are like the some surveys for like the currently like the cyber security experts and then what are their like projectile so like 66% of the people are are predicting that like AI and machine learning technologies will trigger a lot of different like uh cyber security attacks and so traditional security systems are struggling to keep up with the novel and evolving threats and the intersectness of modern systems such as internet of things or cloud expands the attack surface and also we need a new approach to proactively defend our networks. So limitations of yesterday's defenders uh such as signature based NIDS which is network intrusion

detection system and also anomaly based network intr detection system as follows. So for the signature based nids it's like relies on databases of known attack patterns and it cannot detect really new attacks and also attackers are can easily modify techniques to evade detection and also constant signature updates are required as well and anomaly based in IDS is identifies deviation from normal network behavior often suffers from a high rate of false positives and also requires a learning period to establish a baseline and sophisticated attackers can slowly introduce malicious traffic to blend in. So if you can see the graphs on the behavior, this is like the anomaly detections. So when there is like there's the drifted anomaly or undrifted

normally or normal drifted, we used to uh detect those patterns and when we use the machine learning like the trainings over there and detect it and but these are like the kind of um kind of drawbacks from it. So a new era of intelligent threat detection is like large language models or advanced AI models trained on vast amounts of text data and they excel at understanding context and identifying complex patterns in data and can analyze system and network logs like natural language and this offers the potential for more adaptive and context aware threat detection. So some of the um the ones that we have uh look into is turning logs into language for smarter analysis. So when

we look at our log so we are like definitely like looking for the security logs. So the input status are no normally looking as that like January 17 or something like that Damasca something like that those are the logs that save into the our cloud or into our databases and then somehow like the human language we cannot really detect it very easily. So we kind of process it by using the tokenizer of and using and transform into tokens and we are thinking of using disabled and then we put the outputs and then probability squares on it. So treat systems and network logs as natural language and use natural language processing techniques to parse and categorize log entries and also extract

key information like IP addresses or air calls and identify pattern indicates of security instruments and also perform semantic analysis to understand the context. So fueling our intelligence so we pre-train our data based on the AIT log data set version 2.0 O and there are like numerous data. So a lot of realistic synthetic logs other data sets similar like cyber security attacks on smaller enterprise and also network using realistic host logs including web servers, VPNs and firewalls and there are a lot more categories of the attacks but we kind of categorize them into kind of smaller uh multiple categories and structure for evaluation. So logs are organized per host with level attack events useful for uh testing intrusion

detection systems federated learning and alert aggregation and there are a lot of different scenarios and as you can see there's like two millions and one million data a lot of million data over here that we pre-train on. So the reason why we use the this day bird is like the brain of our system of course like bird performs it better but but for the cyber security to works really fast because we have to be like in like not just a second but it can be smaller inference time than that because like bird takes a long a lot longer time for the inference time in seconds. So we use the DC bird to pre-train on it and it is a smaller

and faster version of bird and optimize for CPU inference and also it also excels at interesting um understanding the context of the log data as well. So just briefly explain about the this year bird is like we have a teacher model such as like bird model which use a lot of different context as you can see parameters over here is almost doubled 110 million parameters we're using for training the bird but we are using the knowledge transfer to distill it that's the reason why it is called dist bird and we transfer we distiller the knowledge in a way that for our student model which is like the dist bird can train it a lot faster and uh lot smart

with a lot smarter data but also performs a lot greater like almost like it's only 1% or 2% difference between birth base and dist as you can see on the databases and it performs a lot better than LMO and so this is like the kind of demonstrating the effectiveness of model performance uh we are still like strong results through like five close vation as you can see this is like um this is the one patterns that we want to be keep looking at and as you can see they're like the little bit off in a very few instances. Also there's a big possibility that that it could be overtrained but we're training the really smaller amount of data compared

to the like industry level but it still performs a lot better performance over here and also we have a a 3D graphics design that I'd like to demonstrate. So just to give you a more context when we uh change the log data into the multiple like the vector logs it is a lot multiple dimensions but we um kind of like grab the graphic design into the 3D they to visualize yourself better in the human's perspective. So let's do that. So just to give you just for a second and then

[Music] So this is like the 3D graphic design that is the XY Z level and this like the scenarios over here and then you can see that there is every kind of like similar logs data are transformed into these log spaces and this is very easily visible into the LM space and then you can easily like debug it as well. Yeah. So, this is like how we kind of save into the database and then track it and then uh parse it. Okay. Let's go back to here. Okay. And so model deployments we have to open source it. So we have our model already into the hogging phase. It is called insight. Um and it is we train

our model and host it on hoging face that we can easily track it and if you want to use like the Python or the Jupyter notebook to test this model you can easily like use the transformer and then u retrieve from the pre-existing models and you just need to copy and paste the current URLs over there and then retrieve the model and you can uh use use it as an API as well and this allows for easy downloads and tracking of model versions. So how to build it into our real system. So this is like the for the very basic system that we use for like free but it can be a lot scalable if you use a

different model. So as you can see from the lyrics there's the vendor log kind and also we made our python packages over there that you can install it by using pip install over there. So there's a file watcher and then it goes to all the logs into radius cues that we can simultaneously uh run it that we can put into our worker of course like the thread classification is bird and then it can be either goes to the ready Q again when we find a new patterns that we can train more on it and to uh yeah recursively and also incident response recommendations we are using gemma um but you can use a lot a lot of different

models such as because a lot new greater models came out to analyze it. uh but that time we used the gemma over there and then the DBS SQL light and then we display to the front end SQLite and so this kind of kind of visualization works in what kind of text in the retention text this just an idea about how you can use it as a kind of for the UI perspective if you want to implement this system like similar system into your company that you can and what kind of geographically based attacks occurred or unusual attacks and look data And this is like the log that you can also track it. Of course, this is the

pretty easy one. So, advantages of the LM approach is like adaptive and context aware thread detection. So, LMS can learn and adapt to new evolving attack patterns without constant manual updates. So some of the things that you can think of is like you can just like when there are numerous accounts over there and then you just assign like what is the limits for the kind of abnormal behaviors with this account and then you can just detect it by LM prompting it and also reduce false positives and we can filter out irrelevant warnings by understanding the broader and operational context and this is like very important because like the in the cyber security area like I don't know

very much but like the first positives is very costly as well which is very shown to the very traditional ML models and detection of novel and zero day attacks can detect deviation from normal behavior without relying on predefined signatures they can recognize like subton in real time so this is like something that I mentioned so traditional ML and dim learning approach works well with structured data requires feature engineering for unstructured data and can vary and some models are more interpretable than the dim learning and anomaly based methods can detect novel threats but may have higher false positives can be less computationally intensive for some models and large language models um can process unstructured data like logs more

naturally and offers the potential for natural language explanations of detections and also shows promise in detecting novel threats to the contextual understanding and requires significant computational resources for training and inference but if we pre-train it and also train it in a loop since disable this kind of like cost less that it also compute really fast as well compared to normal models. So future improvements with LM inference models that we are thinking right now is enhanced thread detection. So introduce a new classification category. Um sometimes when it recognize that hey this is the based on the um similarity scores it is very deviant from the other existing ones. Then we can maybe use the LM to create a new classification

category to identify emerging types of attacks and then we train the this model again. And then also you user behavior anomaly detection implement advanced pattern recognition to detect normal account login behaviors and potential security breaches in a loop and also scalable production environments. Um there are a lot of different like structure like SQL models like such as postgrql or you can use snowflake if you like and there are a lot of models that you can use to store those data and also right now we are using radius but you can use a different like kind of tools to use it like AWS tools or like for the orchestr orchestrate and everything. So just based on what your company can

offer you too. So um just summarizing like embracing the power of LMS for a more secure future that you can just not you just you don't have to like change dramatically your existing system but just adding the LM layer over there to analyze it uh better and if there is a kind of a normal behavior. So for instance like there's a case when we uh have like the accounts like the login it's like multiple like networks attacks at a time and then somehow it is not very recognizable um in the traditional ML model but we kind of track into the back backend loop about how what would the user behavior in if they successfully log into our system

and then because like they don't know about our system they first try to figure out where they are and everything and then they so they didn't to just ask about hey what is this system about and then goes more in it and we detected that behavior about that and then we were able to just suspend it suspended that we were not able to cost more like from that account breaches but LM since it is kind of like the text base so you can really recognize it as well and thank you this is like my yeah presentation today but so far like if you have any questions of the existing system. Yeah, please ask now that I can

go over and then uh answer you any questions you have.