What if we could teach machines to think like hackers?

Okay, we're starting, we carry on uh continuing with the international lineup of uh speakers today. Alexandrew Bogdan Miron has kindly joined us from Romania. Thanks for coming. And uh what if we could teach machines to think like hackers? Well, Alexander is going to tell you all about that now. >> Yeah. So, hello everybody and thank you to coming to my talk. This is my first time and besides Budapest and the people here are great. Today I'm going to present what if you could teach machines to think like hackers and we are going to analyze some big data from some potential potential malicious patterns. Now I am very well aware that AI nowadays is more of a commercial term

but in this talk I'm going to try to convince you that AI should serve as an extra layer of defense here to show you some PC's that I have made some research and some interesting findings that I have gathered. Okay, firstly a little bit about me. Hi, I'm Bogdan Miron. I'm a CS student at Babesho University of Kjnapoka. I'm currently working in governance risk and compliance especially in the ISO 271 standard. I was a medalist for the Romanian team at the International AI Olympiad which was held in the Kingdom of Saudi Arabia. Now I'm part of the Romanian AI nationals committee. I make subjects from them and I train the next Romanian team for the next Olympiad

which will be held in Beijing, China. I'm a CTF player and member of the Defcon group Kluj and I contribute in several cyber security community projects. Now in today's agenda which which I want to cover, we are going to make a short introduction about bridging the gap between AI and cyber security. Then we are going to kick off with static Windows portable executable file analysis. We are actually going to build a malware detection model which will cover things such as feature extraction, exploratory data analysis, correlation with other studies, some interesting findings that I have gathered. We are going to study the behavior of some API call sequences gathered from running samples in controlled environments and

then we are going to have a wrap- up and some final notes. Now you may be asking why use auto AI automation and detection? Well, I'm going to make some statements that AI should serve only as an extra layer of security complementing traditional signaturebased methods rather than replacing them. The next uh argument that I'm going to give is that AI can analyze vast amounts of data quickly, more quickly than any human analyst possible and it can recognize patterns and behaviors that would be impossible for rulebased systems to detect alone. Nextly, we can say that AI can detect some novel and polymorphic code based on the same infrastructure that we have made. we only need to retrain it or give

it some more data and you can identify things such as malware exploits or incidents. You can reduce the false positives especially the thing called alert fatigue which is very common in sock teams and very common in traditional detection systems and you AI basically can continuously learn and adapt on new data. I can find interesting patterns without modifying the underlying logic there. And just as a introduction in that thing, I'll go over it briefly. Those are some steps which we want to take to build such models such detection systems. We'll start with the data acquisition part. We can look at public data sets, databases. We can do some web scraping, some crowd labeling. Then we have the pre-processing step which is

basically cleaning the data garbage in garbage out. We need to scale those features handle categorical data plus text. We need to split the data into training and test and validation. We need to handle some class imbalance and we'll see how we'll do that in the next steps. Then we'll need to build and actually train our model, evaluate its performance. Then we can do something called hyperparameter tuning to fine-tune our model. We can tweak its architecture. We can add regularization. We can change some parameters such as learning rate and all the other good stuff that we'll cover. And the last part is the actual deployment of our model. And that's kind of the bulk of traditional malware analysis. We have a

potential malware sample and we can look at its file type. What is its target OS? What is its architecture? What format it has? Is it a dynamic link library? create aexe file, what are its portable executable headers, what are the strings and so on and so forth. Then there is this data interaction part on the top of the screen in which we can use how the program does data manipulation, data destruction, does our program xfiltrate or collect any data in any other way. We can have that multi-stage process. We can check for command and control loader dropper etc. We can look at the persistence part. what registries does it access, what kind of memory it

accesses and so on. And for the network accessible, does it do any scanning? Does it do any discovery? Does it perform lateral movement. Now if we look at a malware sample what we can look basically uh here are some strings that I ran and here's the binary file and we need some way to transform this data into some data that a model will actually understand. This is not useful for a model and we need to kind of extract those into some relevant features that we can build and train our model on. And if we do that, the first steps that we want to cover are kind of those attributes. We got the bite histogram, which means basically how much of a

certain bite is in that file. We'll see if that's useful or not. We can use we can look at the bite entropy to see if the distribution of bytes over an entire file has some meaning in detection of malware. Maybe we can check for some offiscation or some other techniques to hide those malware with that thing. We can actually extract strings P header files uh the section info the imports info what it is virtual size and so on. And starting with that we can just look at the bite histogram. We can just count the number of bytes and put it and check their frequency. In practice, this did not yield any useful results. But it is

still interesting to check and as a first step I tried with this approach. But a more interesting approach that actually yielded some results is the following which is the bite entropy histogram. Now what is entropy? Entropy basically it's a measure of chaos in the code. So we can look at a file and see what parts of that file are chaotic. Basically meaning that possibly there was some code offiscation. There was some funny stuff that is happening here which is very common in malware patterns. And based on this bite histogram which was calculating applying a stride on some malware files and checking so for some gshian distributions and other metrics I was able to distinguish between two types of

malware samples. So in the first image we have pure crypto win32.exe exe and in the second one we have a Trojan banker win32 Dexter samples and uh basically on the right we have those histograms and what we can see the more yellow the more entropy it is the more chaotic it is and we can distinguish two situations here right so in the first one we have uniformly high entropy in the middle of the code in just one part of the code which can indicate different techniques of offiscation And on the other hand, at the second malware sample, we can see that we have uniformly high distribution of entropy there, indicating that maybe the attacker or the malware writer was

trying to to do some random stuff there to actually offiscate that code and to access some seemingly random stuff from there. And that's kind of the reality here as a metric here in a grand scheme. It's great to look at those stuffs and basically what I saw was the typical E.exe pattern. So non-malicious files have small to medium entropy headers but high entropy from compressed and offiscated sections. Uh usually malware samples show sudden entropy spikes like we have seen in the first example indicating that some shady stuff is happening there. Due to very high entropy outside the header in general, we may not be able to correctly distinguish malware samples from legitimate samples on this rule alone.

But it is a great indicator in a large pool of features that we can do. And basically in feature engineering if we see that this does not yield any useful result we can drop it and perform something called feature dimensionality reduction which can be quite useful if done correctly. Now for the second one there are some general extraction of P header features. I use the live uh library in Python and we can instruct the number of sections that the program has. watches is entry point, what is the image base, what kind of features it has, what are the imports that are made, what are the exports that are made and I can just extract from the

strings using reax some interesting results like occurrences of HTTP or HTTPS indicating that possibly that malware sample is connected to some remote server. I can check for URLs. I can check for that MZ signature which is typical for.exe files and I can just basically do some clean up on strings using some regular expressions and obviously here there are a lot more features that I have made but they are not really as re relevant. We can look also at the assembly code to potentially identify harmful op code and operant sequences. And uh if we gather all of these data around, we have quite a large pool of features for just one malware sample basically. Now running this script which I call

feature extractor for some malware sample. That's a hash of a malware sample. I could identify this. This was a very small one because otherwise I could not fit everything in one slide. And uh yeah, that was kind of it in analyzing just one file. And now we are in the data acquisition part. How do we gather so many samples? So we can actually train a model on a lot of data and actually yield some pretty interesting results. And for the data acquisition, we can identify some relevant data sources. There are a lot of public malware sample, public trusted data sets, honeypot and sandboxed results from other companies. We need to automate that collection where possible

because doing it in by hand is not feasible. We need to check about the data volume and scalability. We need to avoid redundant of duplicative data because then we have the problem of multiolinearity which is not good when we are training a model like this. We need to ensure data diversity. We don't want to have a a representative class which is about 90% and we need to ensure the data quality. And how did I do that? Where there is this nice u called the malware bazar database in which thousands and thousands of malware samples are uploaded each day over here. And this malware database has an API which I could have called and I just found a script on GitHub

which basically downloads I can modify to download some malware samples uh as I wish and it's really convenient for that matter because I did not have to download them by hand check them by hand validate that their hashes are not are identical or not because we do not want duplicative data and so on and so forth. And I combined this web scraping that I did with some other data sets, some other publicly available data sets. One is the Microsoft malware classification challenge from big 2015. The other one is the malware analysis data set which covers the top 1,000 portable executable imports and the malware analysis data set for P section headers. We can cross reference

those hashes there and together we can actually collect a pretty substantial data set which we can analyze. We can basically do some data manipulation, some exploratory data analysis and all that good stuff. And basically you can see that those data sets were consistent. I gathered those from Malware Bazar. Those were a lot of file sizes but also those from Microsoft had 37 gigs of raw binary data which I could have analyzed which I haven't which I have analyzed and uh again running the same script on those it took a while but I was able to generate some actual results and this is just one part of the data frame. I got many more samples. But this is quite

what a data frame or the beginning of a data frame kind of looks like. I use pandas for uh that data frame. Data frame being a data structure which holds our features. And I can see those uh extracted features like the hash of uh the malware sample, the size of data, the virtual address, the virtual size, the classification whether it is malware or not because I needed to include some non-malicious samples. So I can actually make this system to detect whether a malware sample is malicious or not. That was a challenge too. And then I have many more features. I have about 50 columns there. And here what you can see are some one hot encoding for other C

calls that the program has made. But I also have some strings there and all of the mentioned above. And in building a detection model, what we need to do is explore the data. We need to actually understand the data. That's the most interesting part and that's the most important part here. We need to do the data cleanup because not all data is good. Not all data is in machine readable format. We need to encode those data. Then we have a more approaches. We can do classical machine learning on tabular data which is classification for supervised learning. We have decision trees, random forest, K nearest neighbors, support vector machines kind of with a kernel trick, gradient boosting kind of like XGB boost

or light GBM booster. Then another interesting concept that we can do is classify the malware families using some clustering techniques. And here we have something like cayins clustering. We have tsnne clustering that we can do and we can analyze actually the differences between those clusters and we can build a better detection model that not only detects the malware if it is a malware or not or if it has some mole intentions but we can actually detect which family belongs to and so on. And that's what I'm trying to convince you. A horistic or signature based methods were not able to detect that based on only the hash or something. It needed someone to manually analyze them and that was a very tedious

task. A lot of times the threat actors tried to hide those code, use some anti-debugging techniques and so on which made that process really really tedious. And for the data cleanup, what we can do, we can handle some missing values. Some malware samples did not have some fields. We need to remove duplicates. We need to handle outliers. Handle that imbalance in the data set. Normalize values, meaning getting all the values on the same scale because machine learning models usually work on data that it is not very skewed or it is not very disproportionate. And we can perform some dimensionality reduction. And just as a quick introduction for the dimensionality reduction, you can see in the picture on the right. First, we have

a 3D with 64 regions which we can map as our features. Then how do we perform dimensionality reduction? We just look at one face of the cube. And that's the 2D projection of it. And for the 1D regions, we just smush them together on one single line. And we need to see how we can do this dimensionality reduction. We do not want to lose data. We want to keep our relevant data and we generally do not want to to cut down some important features. There is this data splitting and cross validation which we need to split our data into train and test. Train which trains our actual model and test for actually validating that our model works

as expected. We need to cross validate the data and we need to avoid underfitting or overfitting. This is a phenomena which happens a lot in machine learning. As we can see there, there are some points and we need to find a function that best fits all those points and that will fit best all of the points that we are wanting to put in that uh in that u in that graph. And for the first one, the true function is represented in green, which is how actually the data behaves. And we are trying to approximate it using a linear function of degree one which we can see that it is not good because if I put another

point on that line it wouldn't actually fit the model too good. The second one would be an example of a good fit. We have there a polomial of degree four which actually fits those data points and it's quite similar to the true function to the true sign function. And there is the last one of degree 50 which uh basically it is a polomial of degree 50 and it interpolates the points almost perfectly which is not good because the model then is very biased and we do not want to have that bias in our model. There is a concept called the bias variance tradeoff here and we want to generally avoid overfitting that model. And on the top we have those metrics the

mean squared error test mean squared error which we can compare and actually see the results of good overfitting versus bad overfitting versus uh good overfitting. Again this is more of a cyber security talk but I wanted to show you this example so that everyone is on the same page. And uh a very cool thing that we can actually do when analyzing marwell samples is actually do some feature importance for those imports. Uh when I have a malware sample, I want to see which of the imports influences the most if the m if the sample is malicious or not. And basically while we can do that we can train a random forest classifier to do that which I believe have the

metric of genie impurity on them and some machine learning algorithms in the back and we can sort them ascending to the importance and to the relevance and I have found some very interesting results. So these are the most important portable executable imports with regards to determine malware in order of importance from the data that I have gathered. And I can see there's some interesting ones like hips set information, unhandled exception filter, interlocked compared exchange and so on. And I can just try to interpret those results. Usually machine learning and AI in general can be treated as a black box or usually is treated at a as a black box with we as humans do not have any

control over on what weights and biases the model puts there. So we do not actually know how he thinks we just know the underlying logic and based on the results from the feature importance I could have gathered the following. I think that virtual lock virtual free virtual protect are used for dynamic memory allocation. They are often used in code injection packing or shell code execution. Set unhandled exception filter. Unhandled exception filter on exit exit. Those usually relate to manipulation of crash handlers or exit routines. May indicate techniques of anti-debugging in which the program uh stops the debugger from actually seeing what the program is doing and some other u routines which prevent the debugging. If some flags are

set, the program is not does not permit itself to be debugged. You can do also control for manipulation or some persistence techniques. Hip set information usually for hip behavior manipulation possibly related to hip spraying. Hip spraying being the technique of distributing malicious code evenly along the heap so that in the event of some hops in memory addresses we land on the attacker's code. Interlocked exchange and interlocked compare exchange are very interesting because they are used in atomic operations using thread sync or anti-debugging tricks and get system tless file. It's really interesting. I believe that it can be used for timing checks or sandbox detection usually some very clever malware. If you are running on very low hardware,

uh it will understand that you are running it in a controlled environment in a virtual m machine or in a sandbox and we will try to get out of that sandbox and um it can time the time it takes for some operations to complete and if that time is unusually high then it means that I have limited resources available and therefore it means that I'm in a sandboxed environment and we will trying to get out and uh here's another study that I have found with feature importance for different malware types and it kind of aligns with what I've did. We can see that for different types of uh of viruses worms downloaded trojans the systems call the system call holds a lot

of importance there. We have those miscellaneous network, what kind of files it modifies, what kind of registries does it access and so on. And it's quite interesting to see how different malware families relate to one another here and what are those feature importances actually and we can actually do some hyperparameter tuning. The best model that I have found from my experience was an Optuna study in which I can set my parameters here. I can suggest whether to be my loss function, log loss or cross entropy and some other AI like uh how deep do I want to go, what would be the bootstrap type, how many rams should I use and uh so on. I

use the cat booster here. I actually use more models which we'll compare later but I did hyperparameter tuning on all of them and we'll compare the results. I fitted the model and the Optuna automatically does hyperparameter tuning for you. It is smart enough to search for hyperparameters in the right direction other than brute forcing each and every hyperparameter tuning. And those are the results that I have found. Again, those are the results that are that high is because still I had more malware samples than non-malicious samples. But it is interesting to look at the confusion matrix which plots basically true positives, true negatives, false positives and false negatives. I can look at that graph which is the rock AOC area under the

curve which plots the true positive rate or the false positive rate. Usually an area of one means a perfect detection here with the SVM with the support vector machines. I was not able to get some good results on the rock AOC and on the classification report we can see some other metrics such as precision recall or F1 score which is the harmonic mean of them but as different models and as different hyperparameters were tuned I seen that the logistic regression did better the area under the curve should be above that first bis sector meaning that it doesn't make random guesses and it actually makes some educated guesses there and actually for the random forest and

from the gradient boosting the performance was really great. It was able to actually detect legitimate malware sources, reduce false positive rates and it actually yielded some very interesting results again on this data set that I have gathered from multiple sources and I find that really interesting. Now we can just look at the model performance and even though the model performances may not differ very much you have to think that we are working on data at a very large scale. So even a 1% difference when analyzing millions and millions of samples that that's a huge uh that's a huge improvement and uh those are very important when working with big scale data and again as an

extra bonus if I have to say we can detect marwell families. Basically what is a cluster? A cluster is a group in those data in which a lot of samples belonged to and I was able to detect the best number of clusters for that data set meaning the best number of malware families that I was able to gather and I could have I done silhouette analysis and the elbow method and from the results that I have gathered I seen that the number of clusters is the best around four to five again on my data set that I have gathered And it is really interesting because this is a plotting of the malware families. I use the 3D principal

component analysis to reduce the dimensionality and to actually be able to see and classify those families and we can see there about four distinct colors for that families. And the difference may not be so clear in this image, but the same one if we just turn around our plane, we can see um and I believe that it is very interesting to see where different malware is mapped on a 3D surface. And based on this, human analysts can identify family of clusters. we can identify uh different types of common behaviors between malares and um I also did some uh clustering with TSN which is another clustering methods the most popular being kins the second one being tsn and here the

clusters at least in the 2D projection that I have made were not very clear and were not very well distinguished but uh I actually launch you a challenge. Maybe you can improve on this. Maybe you can find some more clearer separation between those classes. But I believe that as a proof of concept on limited computing power and on a limited data set, it is a nice way of visualizing actually the malware families. And that was the first section which was the classical way of interpreting the data. But I kept thinking and are there some unconventional ways in which we can perhaps look at that data and let's look at the API call sequences here. What do

they remind us of? And I found this public API call sequences data set which are extracted from the cuckoo sandbox which contains 43,000 malware API call sequences on a thousand goodware API and a thousand goodware API call sequences and we can plot them using a time. So basically what API calls the program did at a certain time and here is how it looks. We have the hash of that virus and we have the timestamps. Usually API names are mapped to integers. Usually more malicious API calls how we seen previously are mapped to higher integers and lower are mapped to more u normal API calls like exit and so on. And if we plot the API calls on a timeline, we get

this stuff. And I kept looking at this a lot. And what do they remind you of? Well, I can tell you what they reminded me of. They reminded me of the atomic bomb. Yeah. So, on the left is Hiroshima. On the right is Nagasaki in the Second World War. And we are entering in a category which I like to call lesson from the atomic bombs. And I promise that this presentation is still about cyber security. But just hold on for a second. Right. There was uh in the after the second world war and in the cold war there was this comprehensive nuclear test ban treaty between countries which prohibited certain countries from testing nuclear weapons

which were prohibited from the second world war onward. And uh when analyzing those data sets uh I I I was actually curious how do you determine if a country is testing nuclear weapons? Well, they cannot test nuclear weapons in the air because that would be too obvious. In the sea, international waters, it is not okay because you have some proofs there. You have some visual evidence. On land, it is again not okay. And where do you need to test them? You need to test them underground. And now it comes that question, how do you distinguish between an earthquake and an atomic bomb? Well, in the image that I have here, it's a seismograph which basically plots the the seismic

waves. And do you see any resemblance? Maybe a little bit. And basically there is this fury analysis. This guy is excited over here. Uh in which you can basically distinguish between an earthquake and an atomic bomb. Usually earthquakes have strong S waves and atomic bombs have dominant P waves which are compressional usually very shallow very sharp. As an earthquake it has a gradual rise in increase. It has a stable centrid and all that seismic stuff which I'm not an expert at but it was interesting to find. And basically what I thought that I could do was the same thing. Maybe I can decompose those signals using fer analysis and maybe instead of distinguishing from earthquakes and

atomic bombs I can distinguish from malware samples and non-malicious samples and uh basically what does furian analysis do it takes a signal like that signal on the right on with the yellow and it basically decompresses it into pure sine waves and cosine waves and we can just basically analy ize these waves and that's exactly what I did. Basically, I had that signal which is mapped from API calls. Again, higher uh API calls indicative of malware behavior have been mapped to higher integers possibly indicating a very strong boom like in the earthquake or in the bomb. And I decompose those signals with the FFT. And basically what are the stuff which distinguishes which uniqually identifies a sine wave? Well,

its amplitude, its frequency and its uh period. And those were the decomposition results of the of the first malware samples. You can see that there were uh some sine waves and cosine waves with different frequencies with different amplitudes. And I find that really interesting because it works on the concept that no matter how much you obiscate that code, certain API call sequences will be hit no matter what. No matter how much you obiscate it, no match how no matter how much you try to hide that malware code. And basically I did an overlay of all sine waves. But in reality there are only about 20 sine waves for each one. And I can do some

feature dimensionality reduction to yield some actually interesting results. And then after that I could convert the sine waves back to features because those graphics are great for humans but not so good for the machines. And basically I did the same thing. I extracted the frequency, the amplitude and phase for each sine wave and uh basically I appended it to a data frame and uh how I did that there is the inverse fast fer transform which uh basically that so faster takes from sample length to polomial length and inverse FFT takes from polomial length bang to concrete samples that's what's called a discrete fier transform or discrete for your analysis and again it was actually great. So from

a thousand API call sequences I was able to reduce it to only 150 features each with its own sine wave amplitude frequency and don't forget numerical data is very good for machine learning models. They can analyze this numerical data and we can can do actually inference on them and we can actually feed those data back to our model which I find really interesting because we did basically a dimensionality reduction which is actually pretty useful and what are the lessons that we learned basically instead of working with a thousand plus raw timestamps the signal can be compressed into a smaller number of dominant sign and cosine waves no matter how much the code is obuscated. This approach is based on the fact that

there are specific malicious API call sequences that will be hit somewhere in the program execution no matter what. Comparing the spectrum of normal versus abnormal API call sequences can help train classifiers and flag unusual activity in real time. And basically by decomposing the API call timeline into sine waves, we are able to turn some noisy, some chaotic behavior into some actually structured data which we can feed back to our model. And we can also identify malware families based on those sine waves and so much and so forth. That's why I love about this field is because you have so much creativity. You can identify and look at a lot of data and it's really powerful in the right

hands. And now we can look at some lessons learned from this talk. Basically, we successfully trained a detection model using static portable executable features. We have demonstrated the viability of ML in thread detection. We have successfully interpreted and analyzed API call sequences and we proved that possibly in the future on a large scale this can serve as an extra layer of defense which I believe that it is pretty cool. We have learned that data is key basically. So if we have good data, we have a good data cleanup. We actually understand our data, we can actually train some pretty good models and quality feature extraction and rigorous exploratory data analysis are critical for model performance.

Interpretability matters. We have correlated my study with other studies and we are I'm trying to explain it beyond some blackbox predictions. we can reproduce it and align with prior studies. Again, it should work as a layer defense. AI should be used as an extra layer for now at least until there are better methods and we can be certain that they will be almost better than a human analyzing just one simple sample. But again, you have that benefit of analyzing large amounts of data and it's about future proofing basically. So adversarial malware demands ongoing model retraining but without modifying the architecture which is pretty powerful and useful and hybrid analysis techniques. Um uh comparing both signaturebased with

machine learning with some analysts and you can get some pretty convincing results there. And there are uh in this talk I was only able to cover malware analysis. But there are some other applications of AI in cyersack which kind of work on the same framework in sock teams. We can have some detection of incidents or suspicious activity combating this event called the alert fatigue in which analysts uh are given thousands of false positives and at 3:00 a.m. at 4:00 a.m. in the morning and they cannot keep up. We can do some exploit detection differentiating between legit and malicious requests. You can do this on SQLI injections, on cross scripting attempts and on any other service that

would require some some user to send some requests. You can automate that. And kind of as an end for my talk, I want to give you some useful resources which I have made. We have this ember data set, some data sets on Kaggle which are very interesting and some other additional resources for you to actually understand a little bit about machine learning and AI. And as a final note, if I want you to remain with something from this presentation, it would be the following. Um yeah, this one I already said, but there is this quote that I really like is malware is like a living organism. It adapts, evolves, and finds new ways to survive. The only constant is evolution.

So must our tools be. Thank you a lot for coming here today. I really enjoyed it. You can find me here on LinkedIn, on my socials, and you guys are really great. So thank you.

Thank you, Alexandra, for the um fascinating topic there uh and your clarity in delivering the um the speech. So, thanks again and hopefully see you again soon. You're here for the rest of the day. >> Yeah, sure. >> That's good. Anyone has any questions and Alexander will be there or available on the Discord channel on the website. Thanks. Thanks again. >> Thank you a lot, guys.