BSides Sofia 2025: Securing AI-Powered Systems: Threats, Challenges, and Best Practices

Okay. So my name is Dero. I will try to fit within 40 minutes. Okay. So very fast about me. I'm an IT professional in over 30 years of experience co-founder of consulting company digitalics lecturer at the cyber security department and the technical university of Sophia where I also have a PhD at the moment focused on application of AI in cyber security more specifically in the defense of critical infrastructure those are things like uh uh Bulgarian postal agency like nuclear power plants and so on and I'm also co-authoring with a team of authors a couple of books focused on cloud security and here here are my contact information. So what I am going to talk about uh AI in cyber security

some security risks for AI powered systems adversarial AI and what it does securing and hops MOPs in security in Asia some bonus about AI security in critical infrastructure and some future trends in in cyber security. I'll try to convince you that AI is actually very useful in cyber security in the next minutes. So why should anybody care about AI? Um are there any developers in the audience? Okay, there are some. Uh now as you know some people say that AI might replace developers. That won't happen because it still doesn't operate quite well. Uh and it won't replace um cyber security professionals as well. However, here are some statistics. There are a lot of currently open

positions for uh for specialists with AI skills. A lot of unfulfilled um cyber security jobs driven cyber attacks on the rise as I'll show later on in the presentation. AI could be used both for an attacking and defense and a lot of big tech companies employ experienced professionals. For example, in Ukraine, a lot of specialists from uh Cisco and other companies u uh joined the Ukrainian government in order to protect the critical infrastructure from hybrid attacks and they actually implemented some quite innovative solutions there. Uh some other stats AIdriven security tools would be a standard by 2030. Of course, some of the I mean there are a lot of startups currently emerging focused on AI security tools. Of course,

not all of those uh tools actually use AI for what it should be used. Some actually just use AI for uh suggestions that they give to the cyber security professionals. Some may use it for some kind of automations where let's just say uh we have some alerts that are walked in service now or some task walked in Jira based on the information that we analyze and some other things. So if uh to basically u uh with AI something that could uh that you can do with an SQL query, let's just say then maybe it's not an AI tool, but uh there are some pretty decent tools out there. Uh some prognosis that AI might reduce incident response time

by 96%. That's still to be seen but with proper automation with proper uh let's say smaller task maybe we could reach that uh as mentioned already about developers u we have about 30% improvement in their work which might not be enough to uh replace a developer but uh it's enough in a team of developers to lay off uh a person or two. Same with a team of cyber security professionals. A lot of organizations believe in enhanced security. And last but not least, there is pressure from governments in accepting uh some uh uh policies focused on cyber security like a way acti framework and others. So where exactly is a used in cyber security? Uh it's

used in anomaly detection in network traffic. It's used in behavioral analysis and user authentication uh for automated thread detection response rate limiting API abuse detection and uh monitoring. Uh some pictures that would explain the constant better. Let's say we have a model that reads walks and then just uh detects some anomalies and it could trigger alerts or maybe generate some report for the professionals to uh check it and fix it. Behavior analysis, let's say we have a senior developer who just uses some repositories and a couple of folders. So what is a behavior anomaly? behavior anomalies. If he uh enters it uh uh free at night, if he let's just say he tries to open some folders like sales

and marketing teams folders, some uh he tries to search for some information about the company strategy and other things that they are not part of his actual work process. That's an anomaly. And uh uh something interesting, some companies actually uh record their team calls and then use AI tools to analyze what's sent to the given team's call. Uh which is uh which they try to understand how the employees feel about something whether they're not discussing how to steal company data. I mean that's an internal threat or maybe discussing whether they want to leave a job and so on. Some people might say that's an invasion of personal space but given those calls related to uh the given job

process it's actually a lot some other pictures about automated detection response AI tools can scan a certain infrastructure it could um uh register some alerts in different systems like Jira service now so on uh it could uh create a post for the users and maybe perform some response like changing some configurations and so on. Uh my presentation is more focused on some apps things to so that you can get a clear picture. There are both demos here. In a different presentation I have shown some demos about the but here it's more theoretical. Uh so uh another thing is rate limiting to limit the number of requests that are performed to the API so that we don't have the those stacks

and last but not least we could use AI for things like checking whether there is encryption and testing in transit uh uh check the different privileges whether they are properly set uh monitor and so on. So a little bit about adversarial and model exploitation and we got a little bit of a picture about how AI could be used for defense. Let's see how it could be used for attack. So a little bit about AI. What exactly is AI? Those are machine learning algorithms. They could be grouped in many different ways. Regression classification clustering and so on. When you usually talk about AI, those are deep networks uh and models. So here's a little bit about the convolution error network. I

want to explain all the different aspects in the math behind it, but let's just say it extracts some certain features of data and based on an optimal amount of features, it tries to make a prediction. Of course, we may have false positives. Uh it might not get the proper information correctly. In this example, it tries to extract, let's say, the edges of a number and based on a certain number of pixels, it tries to predict what number it is. Uh, however, how can we generate an attack based on that? Well, let's just say we have a deep learning based uh that uh attack detector where we analyze traffic and try to predict whether we have an

attack. So, what's the problem here? We can also use some other machine learning algorithms like fast gradient signing methods and others to generate adversarial examples that we could actually use to fool the model and uh uh cause him to get um false positives or maybe to mclassify some attack as uh uh as valid uh network traffic and so on. Uh what is the picture below? Uh there was an incident with Tesla cars that used artificial intelligence where uh the model was very easily tricked. Uh the car was tricked just by adding uh black tape to a speed sign that um uh the speed limit is not 35 miles per hour but 85 miles per hour just by

something like that. Or another example, let's say there were some good scientific articles how you can trick uh an image recognizer with just by changing one pixel that an image of a cat is actually an image of a frog let's say. So these models are not very accurate unfortunately. Uh you can trick them also Tesla might recognize some uh road markings incorrectly and might change lanes and so on. So in fact it's not very safe. How can we uh basically improve that uh attack engine? We could add generative uh serial networks where we have a generator network that tries to generate false traffic and discriminator that actually tries to figure out that it's false traffic and optimize the

weights of the generator network. And then the improved examples could be tested on the victim model and then could provide some traffic that might actually pass our defenses. How can we defend against that? We just need to retrain our defense model with the examples that are being generated by the attack engine. And with a lot of retraining basically unfortunately there is no solution that could just be installed and used and uh we don't have to retain it. Basically the model will always fail under certain conditions. Uh so a little bit about security risk related to a powered systems. Now a lot of companies start to design some system and security comes sometimes later into the

architecture which is costly and so on. So uh once we actually design some system we need to know what are the main risks and for that we have OS PM top 10. Uh I'll just go to them with the mitigation strategies. Prompt injection. That's when an attacker uh creates a prompt that could u get some confidential information uh from the model. We could mitigate that with input validation sanitizing the prompt and restriction dangerous commands. Or if we are using qu language models with natural language capabilities, we could analyze what the prompt is and maybe return nothing to the hacker or maybe just return some information that he cannot perform that operation. Uh insecure output handling

that's when the model is working correctly but might return some information that could lead to some legal issues with the company. some reputational damagure might just return information that is wrong because models tend to hallucinate and return wrong information. That's we have to validate the output. Um if it's a business process, we have to introduce some human in the loop that actually checks the information that is returned to the end user. Implement access control and monitoring. Uh we might have also training data set poisoning when an attacker managed to poison our data sets and uh make sure that model returns only certain information uh based on the certain inputs that uh in that case we

have to validate the training data protect the training data sets implement input filters then our service attacks we have to implement rate limiters monitor incoming traffic implement input filters Supply chain vulnerabilities. There are a lot of issues here that could occur. We have to demol data sets, update and patch third party libraries, implement cryptographic signatures, monitor third party services, implement supply chain level for software artifacts framework or something similar. uh for um a sensitive information disclosure again data sanitation implement output filters role based access control monitor and uh the output. As you can see a lot of these defenses are basically validations for the input and output role based access control and constant monitoring of what

exactly is happening. Again with insecure plugin design, strict input validation, authentication authorization security audits and dates, unboxing the plugins, excessive agency, uh implement human in the loop, probased access control, and monitoring over reliance. Thus we when we overrely on the results of the model. As you know there are constant attempts to make the language models as smart as humans. But we should never forget that even humans make mistakes. Therefore, we should communicate the limitations of these models to the uh employees that use the models to the to our end users and so on that these models actually uh could be wrong. So we again implement validation layers for linking and monitoring and we implement security

development practices. I followed that because uh there is the issue with shadow AI. That is when uh certain developers use um AI tools without uh authorization from the security team and their managers and there were some scandals about that with some big companies where the developers have pasted a lot of company code in let's say tools like CH GPT and others and in order for uh the tool to generate some code for them and of course uh that was leakage of of critical information for the company to third parties and uh that was a major scandal. Uh model theft, role based access control, uh centralized model registry, restrict access to third party APIs, lo and

monitoring and automate the MLOps deployment. Uh here is how it looks. uh all the attacks and users could perform auto theft. Training data sets may need to be protected from the training data poisoning and sensitive information between the application services and we have we have to have input and output uh validation we have to take care of for alliance and so on I won't read all of them and just to mention there are a lot of other frameworks that you can actually check like my teratas that also describes a lot of attacks and mitigation strategies so about securing and has applications. Uh here is a very small architecture about uh an application that uses a lot of different model

applications. It's uh designed with draw a very uh cheap uh tool for architecture design. So let's say we have some quant application like it could be a single page application uh blazer app espanet API or something else and it communicates with microservices that uh uh validate the input uh the pumps that would be sent to the models and it could actually code many different models maybe in certain order let's just say for example with cognitive services it could um extract text from a picture then analyze the text, prepare some prompt for a one touch language model and so on maybe let's just say that it analyzes some picture of invoice and then return some useful information to the customer.

That's a use case. Uh then we have registration layer that communicates with all the models. It might communicate with third party APIs. It might communicate with some automation like a funes, logic apps and so on. And then when we get all the results we can pass it to another micros service unified results aggregator and return information to the customer. I'm not talking here about the different methods of communication. We could have event service bus pattern and so on but uh we could decide that uh based on specifics. So we have the classic API gateway pattern and I'm going to talk about the generative AI gateway pattern and zero test architecture. So the classic pattern is

we have quant applications then we have some API gateway service like let's say asure API management that uh might perform walks uh route the request to the different microservices to the back end and so on and manage the request pipeline uh and uh uh we can extend that to it using KI tools where we track the uh number of requests that come to the to the back end. We could maybe c the request some of the requests and uh return information to the user. Uh we could also monitor and log everything that is happening and perform reports and a little bit about zero trust architecture. We have to implement some of the best practices. Everything

is going using Microsoft enter ID. We can use defender for endpoint and endpoints. We could use tools like u Microsoft defender for count to ensure that we have compliance with certain policies like NIST maybe Microsoft Sentinel where we could deploy our own custom machine learning models because Microsoft Sentinel offers some AI capabilities but they're quite insufficient so we couldn't develop our own custom machine learning models for uh analysis for threat intelligence and so on and of course there are some defenses um def for example defender for office 365 and others. So a little bit about mops in security. Uh what does an ops work looks like? Uh typical case we may have a GitHub repository where we have the code

where the model might be deployed as a file or it might be developed somewhere else in another machine learning repository and deployed separately. But let's just say that we have a GitHub repository. Then we deploy everything uh using Azure machine learning pipelines where we train, tune and evaluate the model. We deploy it to the machine learning registry. Then we have a CI/CD dep pipeline let's say DevOps or something else where we have different environments create product and so on. We deploy the model to um machine learning tool like uh Azure machine learning and then we perform monitoring while using up inside or something else and based on the uh scoring of the model or some other information that is passed

maybe on some errors that it returns and so on we could uh trigger your training of the model and here's how it looks a little bit uh as you can see we have the main devops pipeline we where we can run uh new code and perform unit tests and we could prepare the machine learning model then train it execute it register the model then we could package it in let's say container so kubernetes service then monitor it work everything that we monitor into let's just say in storage or something else and based on some metrics we could trigger another training of the model if it starts to deteriorate and starts to return us a lot of uh errors information error

predictions. Uh what are the best practices? T model metadata performance uh using Asia machine origing registry. Automated training and evaluation in Azure machine learning. Link your CI/CD uh to put a leg test production ready model. Cow hosted model endpoints from our API layers. Track prediction drift and input and open list and any other errors using uh services like application insights or others. And then automate training triggers. Uh here you can have different strategies. Let's say you can retrain the model once per day uh once per week or maybe based on some data like in this example here and then connect the retained model pipeline back to the deployment pipeline. Some bonus uh about AI security and critical infrastructure

uh which is the topic of my PhD. What you see here is the so-called PA or Purdue enterprise reference architecture where you can see how the architecture of let's say of nuclear power plant looks like on the lowest level physical level we have sensors and actuators that actually perform changes to the equipment. Then we have uh uh programmable logic controllers PLCs and other types of controllers on level one. Then we have supervisory control, human machine interfaces, scatter systems and so on. Uh then we have historians on level three on level 3.5 the so-called limit zone. We have jump servers and others and on level four is the sock teams and same systems. So my um

solution that I'm currently investigating and preparing is focused on defending basically level zero level four uh by analyzing the sensor data and the network traffic. How it works? Let's just say that I have IOT devices that pass the information to an Asia out hub and let's say I have some network traffic that is passed to Asia event hub. Of course that architecture could be extended with uh the messaging infrastructure with some defensive infrastructure and so on but I'm focusing on the main process here. We pass all that data to Asia digital twins which is a service uh that allows us to model the infrastructure of uh given uh place like uh nuclear power points. Then uh we the data from the

Asia digital twins we pass to Asia data bricks where we normalize the data clear the data uh remove duplicates or missing information and prepare the data in a format that could be passed for training the model and here we have the pipelines and everything we prepare the model and uh then the model could be called from let's say an Asia app service where uh IT teams could uh see whether there are not any current attempts to perform different attacks on the infrastructure or maybe the model could be used to generate some reports for senior management and uh that's uh pretty much it some future tense about AI in cyber security that I've already talked in the

presentation uh there is a lot of investment in AI powered thread detection response of course there is still a lot to ask because u AI thread detection response only works in specific cases for specific services at the moment but uh there is a lot of money involved in here so eventually maybe they'll come with some more general solutions uh and though there are also some augmented cyber security threats as I've uh uh shown in a couple of slides I haven't talked about quantum resistant AI security as you know uh quantum computing is slowly on the rise u currently Some of my colleagues are testing some machine learning algorithms um that could be executed uh on quantum

based computers and u but basically we still can perform some um serious attacks uh with quantum computing but uh as you know European Union has already launched some programs in order to implement to develop and implement some uh algorithms which we can use for cryptography so that they won't be uh so that quantum computers won't uh be able to break uh the current uh cryptographic algorithms. I've talked a little bit about cyber security as a service in AI. Uh that's something that a little by little would uh come to fruition into the future. Uh that's the idea of big tech companies. they really want to say a solution that could be used by cyber security teams. Of course, it won't uh

replace the cyber security professionals, they would use that tool, maybe make some customizations to it like in the case with Microsoft Sentinel, maybe add some custom models and so on. But uh that is uh coming to be the standard and of course a powered compliance that's when the AI scans your infrastructure and make suggestions about how to be more compliant with some policies like GDPR, HIPPA, NIST and others and some useful resources uh some uh YouTube channels that I for some interesting things you can see here and some books that I like to read uh zero security, application security and so on. I want to emphasize from the book how to with statistics because it's very

easy to generate statistical data nowadays especially with some AI tools and you should always check the information behind the statistics. I had some statistics on uh some of the first slides of this presentation. Therefore, never statistics blindly and we're trying to check them. And uh that's for my side. Thank you for attention. And here is my contact information. Could you get any questions?