RAGnarok: Assisting Your Threat Hunting with Local LLM

All right. So, good afternoon again and welcome to Besides Las Vegas. The um this talk uh is called Ragnarok, assisting your threat hunting with local LLMs. It'll be given by John Mure. Uh few announcements before we begin. You probably have heard these already today, but we'd like to thank our sponsors, especially our diamond sponsors, Adobe and and Iikido, and our gold sponsors, Formal and Profit. Uh, it's their support along with other sponsors, donors, and volunteers that makes this event possible. Uh, also, as a reminder, this talk is being recorded, so uh, as a courtesy to those watching online or those in the room, please remember to silence your cell phones. With that, I will pass it off to John.

Okay. So, uh thank you for everyone uh and and Har this presentation and uh it's it's almost 6 p.m. So, let's drink with me after this session. Yeah. So, uh this is our agenda of today's presentation and first introduction. Uh I'm Jun from Japan. I'm a researcher at Fujutsu Defense and National Security Limited. and I'm mainly focused on offensive security but currently I'm interested in uh threat hunting and ARM or Raj langage model. So I talk about uh this topic today and uh this is my brief background and I have some experiences in cyber security field especially teaming and penetration testing. So anyway uh from here I explain uh Ragnok at first uh background uh one day my

boss said me uh would you be involved in threat hunting and de develop new demo so I said okay but I was a little bit confused because I don't know anything about threat hunting but uh in my mind uh maybe it was necessary to create uh some detection rule from threat intelligence anyway I have decided to investigate threat hunting And after some research, I found that threat hunting has a lot of boring steps and requires a lot of uh human resources, money and time etc. I thought that I wanted to automate the boring this boring process with ARM uh rather model. So there are some concerns in using ARM especially based like CHBT. So uh there are for example a cost uh

sensitive information and security risk uh in threat hunting uh I think uh some sensive information such as network configuration are required and we wanted to avoid uploading it on the internet. So to achieve it I have developed ragnok and uh which develop uh which adopted local area and made it working on the CPU only machine. This is key concept of this topic. So from here I talk about Ragnok. So Ragnok is an application that can generate a sigma rule from threat intelligence by using local ARM. Uh this is uh sorry uh yeah on this figure uh the uh basic concept of Ragnarok. So this application provides a web user interface where use u user can input uh

an order to the local area and the area refers to the threat to intelligence and get some information then generate the sigma rule based on the uh original order and there are some technical features of ragnok. So in Lagnark of course we'd like to uh to use uh local RM but also use the latest and correct information. uh in addition we' we'd also like to implement scalable system and generate a more practical outputs to meet these requirements uh quantiz augment generation which is called as brag and uh multent system and of course environmental information are used. I will elaborate on this technology and the radio section. But here I explain uh explain the environmental information.

So in general many organizations uh use the active directory to manage their resources such as accounts, computers and group policies. And in the active directory environment uh it configuration data can be corrected and visualized by bran tool. uh this tool uh sends some requests to domain controller and obtain the configuration it configuration data then uh visualize visualize it it's like this here sorry like this and this uh this information is likely to be quite useful for threat hunting. So Ragnarok integrates with uh this to hunt. Oh, this is architecture brief architecture of Ragnarok. Uh, basically Ragnarok runs on the docker and there are two docker images. Uh, Ragnarok images and brought hand images and Ragnarok image is the main one uh where

the core person code is running this one. Uh this code of course provides web user interface as mentioned before and interacts with herm through or brandation images uh has the nail database to store the active directory configuration data and return the data to the application uh when sent to queries. Anyway uh in the section let me show the demo movie of Ragnarok. So as mentioned before Ragnarok provides web user interface. So uh like this there are two tools and the main tool is to generate sigma and the sub tool is manage vector store and it's to register and manage rag data. Before the main part uh let's see this part manage vector store tool. This one

I will explain rag in the next section but for rag uh saving data in advance is necessary. Uh in this tool uh we can register new rag data from the uploaded file like this and if the uh if the file is JSON file it is necessary to specify the JSON parser. Of course, can be used for JSON parser and by clicking uh the save button, the data can be uh saved in vector DB and in addition the data uh obtained by uh web crawing. Yeah, here uh web crawling uh yeah by web crawling can also be registered as ra data. We can specify the target uh page URL from here. Yeah, from here and uh we can choose the uh and there

are two crowing methods uh implemented in rag lagn and we can choose the best one depending on the situations. Yeah. And then uh the registration is easily done uh from here. And from this uh descriptions of correction page uh we can view and manage the registered rug data including deleting. So like this uh next is the main tool page Ragnarok and here uh we can generate the sigma rule from threat to intelligence and here uh we use uh micro attack the threat to intelligence. So like this and first uh from here uh choose the area model and then specify the attack techniques and set the temperature parameter and finally around the generation process.

Now this process takes time to complete. So we we need to wait for it. And after a few minutes the sigma rule is generated uh like this. And to validate it accuracy I tried using the sigma rule in our demo environment. And in the demo environment a sprank is used as a same product. So it's necessary to combat the sigma rule into SPL in advance like this and then test the SPL in our demo environment. We can get three rocks by this SPL. Uh and the second one uh this one is generated by actual specific attack here as a career thing. Of course we have to uh do the final check manually but we can check the

effectiveness of ragnok. Yeah. So from here uh I elaborate on technical details of Ragnarok. First is uh quantized area. Uh this is optimized model whose parameters are expressed by fewer bits uh than its original one with minimizing accuracy. So in general a model has billions of parameters and require many computer resources to run. But after quantiz uh quantization process the uh quantiz is created and this model has fewer parameters and requires fewer computer resources to run than original one. So this is quite important for running her local on CPU only machine and the objective of quantization is to reduce the model size and accurate predictions. And there are some tools uh to perform the quantization but uh Rama CPP is one

of the major tool. Uh this tool is used not only to quantize RM but also uh to run it on the CP only machine and ORM as mentioned in the previous section provides a rest API to uh to local ARM by using this Rama uh CPP as a backend service and additionally uh there are various contis with Rama CPP and we have to consider the tradeoff between response quality and the model size in using uh this table shows was the example of quantiz model and uh this one uh Q4 KM is used in Ragnok because it keeps the response quality while reducing the model size and second is RA augmented generation. Uh this is a technique uh that allows RM

uh to generate uh results based on external information without any finetuning. And there are many two step uh mainly two steps document retrieable and text generation. And the document retrie uh should be done before the text generation step. And in this uh this tech uh document retrieable step uh text or at first text are extracted from the document uh we would like to use as the external information and uh they are chunked like this. Then uh they are converted into vector data with embering model and saved in data store called as vector DB. Here the this uh this uh the embbering model is an AI model which converts the unstructed data like text, image and

music into numerical vector data. On the other hand, in the text text generation uh step uh an order uh a user input is also converted into vector data with embedding model. Uh then retriever generates a context. uh this one uh contri contest. So this retriever searches um the data store based on the input vector uh data and obtains the information which are likely to be related to the data. Therefore we can get the relevant information to users input uh audio input and the generated uh context are added to the original user input. So that uh the prompt is constructed. This one and this prompt is finally a pass tom and arm generates the answer based

on all information. By using RA uh we can improve accuracy and reliability of ARM response and moreover if the latest information is used as the document uh this part uh ARM can generate the answer based on it. Uh next is multi agent system. But before explaining it, I explain uh AI agent. AI agent is a kind of software system that takes actions auto autonomously to achieve their objectives instead of humans by using AI. Uh the agent has its own prompt and some tools. We can define these tools uh depending on the objectives which are used for a specific purpose. Here uh the agent uh can do complex task instead of humans and make a decision faster than humans

and adapt flexibly uh to changing circumstances and marriage agent system is also a kind of software system where multiple specialized AI agents works together to accomplish tasks. For example, uh it is assumed that a uh a user inputs like this uh please provide output uh b considering the information in a and uh uh there is a special specialist group and uh two agent agent A and agent B are belonging there. Agent A and agent B are working together by focusing on each specific task. In other words, the agent A uh takes an action about A and agent B uh takes an also takes an action about B. Then uh this specialist group generates the final output. There are various

final outputs. So there of course there are various benefits in marriage agent system but we can use appropriate agent and prompt depending on tasks. Moreover, we can implement a scalable system. So if we add a new agent to the existing system, existing group uh its capability is expanded without redesigning the whole system. So Ragnarok generates sigma rus with combining three technologies these three technologies and there are some agent and each agent has its own tools and quantiz then uh uh so by cooperating with these agent each other uh the final sigma rule agent is generated here uh I talk about some key points of this presentation so at first let's uh deep dive into the generation process in

Laguna. At beginning uh user specifies uh the specific attack techniques as shown in demo movie uh and the original order is created like this. So please uh generate a sigma rule for hunting care like this and uh there are three agents intelligent agent broughtand agent and sigma agent and intelligent agent uh take an action at first and this agent obtains the uh specific detection method from threat intelligence such as MIT attack and uh yeah in demo uh yeah this is MIT attack and Then uh if a specific attack is related to the active directory so brought agent takes an action. Yeah. And this uh this agent uh sends cipher query to neoj analyzes obtain json data and extracts the critical

information for uh creating hunting rule. And finally uh sigma agent refer to the do some documents and generate a sigma rule uh based on all uh information and more I will elaborate on the generation process in sigma agent. So uh this agent has two rag tools uh that search sigma and the first one uh sigma tool searches sigma manual and get some information and uh msdn tool of course uh searches uh msdn document and get of course get some information then uh am uh generate sigma rule based on all obtained information and once generation is completed. Uh the agent evaluate the quality of the generated sigma rule and if the gen sigma does not consider the

enough information sigma agent generate a generate a new rule and if enough uh the uh the final sigma rule is output. So here is a no and the tips obtained from the development of ragnok. So first filtering the filtering the detection method in might attack is critical. So might attack uh the detection method is uh often uh includes various information and if uh not filtering filtered a hunting rule based on the unrelated information is likely to be generated and second in a mer in the system some token parameters should be adjusted especially token size. The token size became bigger as the margin system is in use. However, all local models have the limitation of the token size and if our

token size exceeds the limitation, the generation process will be stopped incorrectly. So summarizing the context uh is a way uh to avoid the uh corruption corruption. Yeah. And finally uh we often have to uh use Windows event log uh in threat hunting but it's log format are different for each event id. So in order to deal with it we have to implement uh we have implemented a new log tool in the sigma sigma agent uh for searching the ms msdn document and there are some uh limitations of rag. Of course using rag as this darts not solve anything. So we should use rag appropriately especially uh in using rag uh chunking strong effect strongly affects the

generation accuracy. I found that uh chunking into meaningful units such as paragraph is so effective and furthermore in lag process the original documents as I mentioned before must be converted into vector data and save in saved in a database before the generation process. So if the if the data size is larger uh it takes much longer time to save it and requires more disk space. In fact uh in that case we should choose uh other options such as continue pre-training or finetuning and uh closing. So as conclusion uh Ragnarok uh based on roarium is designed to assist threat hunting by automatically generating a sigma rule for specific attack technique and the combination of quantiz rag margin agent

system allows Ragnarok to run on CPon machine with high accuracy generation and the integration of Ragnarok with brat hunt makes it possible to generate practical sigma rule uh that consider the active directory environment information. But however on the other hand there are some future work future work especially further improvements and expansions. Now we will focus on implementing the space supervisor agent for regular stability. And finally takeaways uh sensitive information is essential for threat hunting. So local AM is AM is one of the best option to assist the process and running local AM on a CPU only machine is challenging in terms of machine resources. So it can be improved by combining technologies such as quantiz

and multient systems. However uh these area related technologies are not one size fits all. So it has pros and cons and must be used with co must be used co with cions. Uh that's all. Thanks so much. [applause]

[applause]

>> Do you have a publicly available repository or anything? >> Uh yeah. Uh so actually I'd like to uh uh this uh make make this repository public by this time but our internal process is so complex so it takes a long time so please wait and maybe uh please uh follow my uh SNS account like wrinkling so I will uh make a notification in wrinkling

Thank you. >> Uh how do you how did you test your sigma rules that were generated to validate that they were detecting the activity that you were trying to detect? >> Yes. uh actually uh we have we have tested and uh uh tested the generated sigma only in our demo environment. So actually uh I'd like to uh test it on the our corporate network but yeah we have no right to test it. [laughter]

Thank you for the presentation. So my question is about the house nation and I believe when we are using a light model it's very likely that we have a house nation and but having the rug is already good enough to prevent the hassation or do you have any comment about that? >> Uh actually it depends on the situation cases. So but often uh yeah I think uh ra is the best option to avoid harshation and uh yeah actually rug is generates constant outputs uh I I think but yeah sometimes uh some agent makes some mistakes [laughter] but we can't just because this is these type of AI uh system is blackbox system so we we can't judge whether

This is harsh nation. [laughter] Thank you.

>> Um, yes. So, you uh have it optimized to run on CPU and it's using low requirements. Is there any specifics around it? Does it how well does it scale for a large organization versus a small >> Mhm. Sorry. Uh >> uh so it's optimized for CPU like I'm looking Yeah. Yeah. uh at the system requirements for it and how does it scale for a large organization. >> Okay. Okay. Okay. So maybe uh yeah actually uh we have uh we I can't uh provide you with correct uh CPU resources but maybe uh I use uh maybe four core and uh 32bit uh bit memory uh or so. Yeah, but uh but for using uh in

order to use the rag uh so it's uh requires more uh v uh many disk spaces. So maybe we should for prepare for the uh enough disk space in advance maybe. >> Thank you. >> All right. Uh I think that is time. So uh again thank you. Big round of applause. >> Oh, thanks so much. >> [applause]