AI security lessons from the frontline

We're in business. Good morning, Bides. >> Pretty good. You know, you you seem to be quite awake. That's nice. That's nice. And when the first time that Bides uh Amsterdam announced they were going to do this event, they expressed the desire to have 200 people and now there are 200 people on the waiting list. I think this shows that security is very much alive and the great news is that we have job security, right? Because as long as there are adversaries, we need our army of security professionals. But the army is a little bit confused. I asked JetBT to draw a front line with some confused soldiers because that's us including me because of this thing AI.

Now it shouldn't be all AI because what I really love about today is that all the talks most of the talks are about security just security and security is about security but there are a couple of AI things but it shouldn't not be all about AI because that's distracting. So I love the organization for having AI as the opening keynote to get it over with and then focus on security again. I love that. But I do believe that we have a job to do. I firmly believe that without us security people, AI is going to be humanity's doom. I'm not a doom thinker. I'm an optimist, but I really believe that AI is not evil. Right? I see no

evidence of a scenario where AI is going to kill us by itself. But we definitely are going to and currently are harming each other through AI because we connect AI to everything. We um we have to deal with an enormous attack service and we really don't know very well how to protect it and that's where I want to make a difference. So I'm chief AI officer at software improvement group. I help a lot of organizations with making better and more secure AI systems. I work in standardization and I have a very long history in AI. I try to teach, educate, share everything that I know with the world and make a difference by setting the standard and by talking to

folks like you today to try to make it elegant to try to sort of provide you with ways to get started with AI. And today I want to take you to five lessons from my own experience through my career. And we focus on some standardization work. We focus on some attacks, some some some great things of AI and some terrible things of AI. To start with, we'll have to go back in time. So the first lesson is one of my biggest lessons in AI. It had to do with a police system that we developed in the beginning of the 90s. So I started out in 1992 in the first AI company of the Netherlands called Centered Machine

Research and we built all kinds of machine learning applications and one of them was helping the police to predict crime patterns. So we used police databases and for example we were able to predict over time where in Amsterdam pickpockets would would be you know would act based on patterns. based on examples. Training set was the police database. We use machine learning and we could predict on a you know geographical time level but we also could predict on personal level. So criminals building machine learning models to predict their criminal career. So the police could focus on the right people and ignore the people that would be okay after their first, you know, demeanor or whatever. And we all felt this was a fantastic

idea. Uh the privacy officer, the the the the police uh chiefs, uh the mayor, everybody felt so we should do this. Fast forward to 2025. Now this is forbidden. It is forbidden to predict criminal careers at least in Europe because we now have established the consensus that police data is biased. It has systemic bias. The actions, the decisions of police officers, whether or intended, they contain racism, sexism, all kinds of biases that are being amplified if you use that data of their behavior in a machine learning model. Back then, we thought it was great. Now, it is forbidden. I think that's very interesting. I think that um this really indicates that careful thought when applying AI. So when you have an AI idea

is really important. So think before you start. Look at all the ethics and use because you have to the rules and the regulations. The regulations like the AI act make some things really simple. You don't have to think anymore if you can use AI for uh HR purposes. No, there are a couple of rules that the AI sets because HR purposes have to do with people's opportunities in life, their job, their future, you know, their their salary, all those things are essential. And together in Europe, we we thought about what are the ethics here, what are the boundaries, what are the fundamental rights. and we set a couple of rules that make it really simple for the

industry to quickly decide whether or not to do anything. So that's that's an improvement. It helps us. But the bias back then was not the reason that we stopped with this application. It was actually uh a security flaw. Let me see some hands, please. Who who who's heard of data poisoning? Yeah, almost 60% of you. Great. Data poisoning was the threat that we later found out was really so problematic in this case that we did not follow through with the application. We build it, we applied it and then we sort of in hindsight did really good threat modeling and then we learned that okay so um yes criminals can change the database but it's really

difficult. they need to break into the database and then maybe we can detect that the model is has been poisoned and been changed behavior because that's is what the criminals would do. They would inject certain crimes into the police database and certain individuals that don't exist to make the machine learning model think that everybody with a Q in their middle name we should ignore and then they would you know assume a Q in their middle name. I would fool the model. We realized that wait a minute such a back door in a machine learning model is actually undetectable because we don't have test samples with cues in the middle. We would not detect it. So

this means that they only need to hack into the database at one time put in the samples. they would control the behavior of the police and it would be totally undetectable because there's also no code in the machine learning model. There's thousands of weights. You really can't see if there's a back door there, if there is code injected that is malicious even if you look very well. And it made us decide that we would stop with this application. And I think the lesson there is that sometimes the best risk treatment is not to mitigate the risk with all kinds of you know encryption and whatever. The best risk treatment is to avoid the risk and not apply it at all. So let me

illustrate how data poisoning works real brief because you guys are familiar with it. I think this is a good example also because of the big impact because we're dealing with a missile that's being fired. It's been fired into the sky and it should hit an enemy aircraft. There's a big training set with a bunch of pictures that have been labeled friendly and enemy. The machine learning model is trained and it's able from pictures to see ah this is an airliner or this is this is an air fighter. You need thousands and thousands of images that all have been labeled. But once you're able to hack, and I've made this really apparent to you guys, but

normally you can't you can't see it or you can't detect it using statistic analysis because that's simply not effective. Um, what the attacker has done here is has relabeled some samples and put a red dot at the front of an airplane. There are no airplanes with red dots in the test set. So the model goes through the test. It has been hacked. There is a back door right here. If there's an airplane with a with a red dot on the front, it's an enemy aircraft. So, the missile is fired and the attacker is on the ground with a laser points to an airliner and the missile hits the airliner and hundreds of people die. The

covertness of this, how sneaky you can do this is is reason to not do this at all. So data poisoning one of a backdoor data poisoning one of the I think most important threats in in machine learning also because of this an enormous attack surface there's a supply chain of images there's a supply chain of experts that label those images it goes into data engineers then it goes to model engineers then it goes into deployment and all those situations even in deployment if you're able to exchange the model that is there with a model that has been trained with a back door uh then you're also in business as an attacker. So enormous attack service big

consequences data poisoning is something to watch out for. Next week I'm going to be in uh Abu Dhabi teaching chief AI officers at the AI university there and one of the main lessons I want to get across to them is AI ignorance is okay. I get a lot of questions about AI. I'm I'm in panels etc. And I try as often as I can to say I don't know whenever I get a difficult question and uh the board of software improvement group really doesn't like that very much but they appreciate it because they want an expert on stage not somebody that says I don't know. And I think that would be my main advice to

you guys. You're going to learn more about AI uh whether or not you want it. you're going to be more involved in it and people are going to ask questions and from your professional pride and maybe from their expectations you think you need to give them an answer and the best answer if you don't know is I don't know or for people in the in the Emirates uh what works much better is I'll get back to you later and give you the answer right because immediately trying to give an answer in which you sort of try to estimate it or run to jetp to get the answer which you can't review because you don't have the

expertise. I think that will be the downfall of AI AI security. You guys need to be honest and if you're involved in setting a culture in your organization, it's important to set the culture of honesty about I don't know. I try to do the same. I'm an AI expert, but it's a lot of subject in which I need to read into, right? But then I say I'll get back to you. So AI ignorance is actually okay. It should be okay. it doesn't mean that you should not learn about AI and that's where I focus on most of my work on and mostly revolving around the AI exchange maybe some hands who's heard about the OSBI exchange

great yeah that's about half of you wonderful wonderful it's a project that I founded uh three three years ago now it it is basically a body of knowledge so it is uh 200 pages of material at OASP AI I.org. It's an OASP open-source project that collects the consensus on AI security. It's in essence a large threat model with a whole lot of threats uh and the controls that you can use to mitigate those threats and a lot of guidance when it comes to uh doing threat modeling and um doing risk analysis and changing your organization in order to be able to uh to do so. So, it's quite comprehensive. We've built it with a large group of people. I've

selected about 130 since yesterday. It's 130 experts from around the world, academia startups practitioners all kinds of people trying to put together this consensus. And there are a lot of efforts out there that are trying to do this. But I felt that if you do this in an open way through an open-source project where everybody can join um you have a lot more advantages than international standard organizations. So like ISO and Sensanellic they of course they establish the consensus but they suffer from typically a lack of expertise when it comes to these difficult and rare expertise areas like AI security. And at at some point I got involved through ISO um and Sensanellic in setting AI security standards and I

connected them. So what I did is I established a partnership an official partnership between ISO senselic who who was doing together with Etsy the standards in Europe and this official partnership allowed OASP this this project to contribute the content to the standards and that's what happened. So a large part of ISO 27090 which is the AI security standard uh is built on the AI exchange and a large part of the AI act harmonized security standard that is coming out is also built on the AI exchange. So it's it's sort of becoming a consensus and this is being embraced by many other uh groups like Issaka like Exin like Sans Institute. We have a partnership with Sans Institute and SAN

says wow this is copyright free this is free of attribution this represents the consensus this is aligned with the AI act and with ISO this is really the material that we want to use to teach practitioners around the world uh in in our training so let's do this together and sense is great experts so we're now collaborating on things like critical controls and essentials and putting together a lot of sort of educational material and reference material for AI security. So unbiased, but I would really recommend the exchange as a resource for you. What else is there? It's so again, it's free of copyright, so you can use it. You don't even have to attribute us. That was a prerequisite

for being able to work with uh with international standards and we're integrating this into openc.org or which leads me to explain a little bit the involvement of SR and myself in standardization. So about six years ago, seven years ago, we established OpenC opencre.org. U maybe some hands. Who's heard about open Siri? Yeah, about 20% of you. So we need to do more marketing. I'm doing it right now. Go to openciri.org. You can find all the security standards integrated into one resource. uh search for information. Uh there's a chatbot there. You can map standards to each other. It's a great resource in your work. So this one of the things that we did from Sag is our taxonomy of security

topics. We donated to this uh open source project. We donated our work to SAM which is a secure SDLC uh maturity standard from OASP. Great standard. We created an AI engineering framework. So if you are in a data science team or if you run an data science team, have a look at ISO 5338 because that's where our AI engineering framework ended up. It came out two years ago and it's a great framework on how you extend an existing way of working your existing software engineering practices to AI. How do you do versioning on models? How do you do model cards? How do you AI build materials? All those additional things that you need to do for AI are

there. And that's my recommended approach always. The things that you do, the processes that you have in place, your ISMS, just extend it for AI. Don't build anything new, new new process. Well, you can create a program in which you change and add things uh AI things to your current way of working, but adding things is much better than setting up something new and confusing and distracting. where it's actually about the same thing you're already doing and I I'll get to that shortly. The AI security threat model that we developed at S became uh the center of the AI exchange which became the center of ISO 27090 and the European AI act standard. So those were our involvements

in the national standards. The one that I've been working on the hardest uh the last two years is PEN18282. This number has just been assigned. Uh in fact, yesterday I was in a session of three hours with the European Commission to go through the next steps before it can go into public inquiry. So this is a standard that products going on the European market need to comply with if they are high risk. So if it's an HR system, a medical system, uh if it's a critical infrastructure, so a bunch of systems need to comply with this. But it's not just for those systems. It's also representing the consensus of what makes a secure AI system. When is it

secure enough? It has been quite difficult to set that. We needed to make the standard riskbased. Now, this standard is coming out hopefully in the middle of next year. Uh, when it goes into public inquiry, we'll get 6,000 comments, I think, because this is a super opinionated topic, I can tell you. But keep an eye out for this because this is going to be very central in the way that we uh we work moving forward. Um, I want to take you guys through the main model that we have at the exchange. It's a very straightforward model. what's called the AI security essentials where you have your typical AI system. So there's let's see uh a model and

there's an application and an infrastructure. Something goes into the model and maybe you have your model hosted by an external party. So travels there, there's an output and that model has been created through uh machine learning with training data or it's a model based on rules that you've put in. That's also AI. And there's augmentation data. That's an important extra asset uh important to be aware of because many large language model applications use vector database to inject additional information into the input. You will see this a lot. We are seeing this a lot. Retrieval augmented generation system, rag systems are built on this. The idea is that you have a question for example about a certain topic. The material

around that topic is found in a set of documents. The model has not been trained on it. So these are for example reports in a company and that is augmentation data. it's added into the input together with the question and then the large language model uses that extra information to answer the question together with what the model already knows. This is called in context learning and it's a a dominant application of generative AI because training your own generative AI model typically cost millions of dollars or euros whatever you want. It's very expensive and difficult but these models of general purpose and with augmentation data you can specialize them without training them. But this augmentation data is an asset that needs protection.

So we're dealing with a bunch of assets and they need protection and that's one category of threats. It's just standard security. It's role-based access control, encryption. It's fishing campaigns, all the stuff that we're familiar with. But we need to be aware of these new uh assets and need to be aware of their particularities. So if the input goes to a cloud AI, the particularity is that you really need to read the small print of your cloud AI provider because that input may be logged may be monitored and in many cases it is and you need to have an enterprise license and opt out of that monitoring in order for that data not to be in the vendor's cloud. Right?

Those things uh those particularities are important to understand but they're really basic. It's just a list of things that you need to know about these these these assets and then you can incorporate them into your ISMS. It's just standard security. What's not standard security are input threats, a whole new attack surface, the input of the model where you can do a lot like mislead the model. You probably have seen the traffic signs changes, right? There's a traffic sign, you put some stickers on it and then suddenly it's interpreted in the wrong way. Another example is if you have a spam message you put in specific words and the models think oh this is not spam that's called

evasion attacks prompt injection that's for generative AI where you try to put secret instructions or specific instructions into uh a prompt making the model misbehave and for example expose sensitive data or perform very harmful actions especially agentic in agentic AI you can extract the model through the input You can do a lot of things. You can extract the data. So if there's a model and if there's sensitive training data, chances are that you can be successful in extracting some of that data through some of those input attacks. It's important to be aware of them. But as a security professional, you also need to be aware that the controls against this mostly come from the AI team, the the data scientists

because it's very mathematical. It's very uh linguistical. it's in their expertise area. So you need them to build the controls, most of the controls against those input threats. And because we're dealing with actually a supply chain in many cases, data is coming from an external party. Model is hosted externally. Maybe you simply have a license on this model. So there are suppliers involved and these suppliers, they need to take care that those conventional threats don't happen. Which means that you need to take care that you check this with the suppliers or provide additional countermeasures to deal with the risks. Conventional threats, input threats, supply chain threats. That's it. Those are the things to understand. Now if we look at the

controls, very important minimizing data, offiscating data, uh minimizing, you guys are all familiar with offiscation also. But with AI, you have much more opportunity to obiscate data because the models don't need perfect data. They need data that is, you know, sort of in the same ballpark because they're stochastic, they're statistical, which means that you can change the input data and even make it unrecognizable or still the model can work. So this is technology that AI teams need to understand to reduce the risks. If the data is not recognizable, it can't leak. So you don't even need to uh to secure the model. Extend your supply chain management. Apply conventional security controls obviously you know to control all those

conventional security threats all the things you're familiar with. This is a whole new category for most security professionals and sometimes for most data science teams. So that's an opportunity to make sure that the data science teams know that if you add noise to uh your training data, you can get rid of some of those trigger data poisoning. If you remove confidence from the output of the model in your API, you uh mitigate a lot of input attacks. And data scientists need to be aware. You don't have to as a you know typical uh security professional become proficient in this. There's very little pe very few people who uh know a lot about conventional security and this data

science stuff. I I actually don't recommend it because these expertise areas are orthogonal when it comes to the affinities that you need. Monitoring can be regarded as part of conventional security controls but it's a separate thing because it's a combination of conventional security. you need to know who has accessed your system but you also need to analyze the input that comes in uh using statistic methods so it's a collaboration between the data science team and the security team last but not least limit model behavior I always say zero model trust model can be wrong the model can be manipulated which means that you need to take into account that if you have a self-driving car and you have a model that recognizes

commands voice commands from the driver. If the driver says open the trunk, you need to take into account that it can be wrong. So, you need to have some checks like for example u don't open the trunk if the car is is still driving. That's an example of a rule-based uh uh guard rail that you build into systems. So, those are the categories of controls. How do you how do you implement this? Well, you need to teach your AI engineers those opiscation strategies. Make sure that they know this and know the the importance of minimization. AI engineers would typically get uh personal records, put it in a training data set and then go ahead. But much of

these personal records they don't really need for the training. So, it's important to teach them that they can simply leave them out because they have no role and it improves privacy and security. Extend your supply chain management to data suppliers, model suppliers, and your AI model host parties. You simply add your AI assets and the consequences to your repository and then it's part of your information security management system. You want to teach the AI engineers some dev sec ops because in many cases they are used to creating systems that work in the lab but not per se systems that are secure and scalable and maintainable out there. They need some help and you want to make sure that

they get taught those AI security controls these AI engineers and learn about how to monitor it. And you want to inform other teams about those AI security controls. So they need to know the rough idea of what is possible so they can guide the AI engineering teams. But you you're not going to be required to ask from your typical uh information security officers to really understand how these AI security controls work. That's the job of the AI team. Regarding the model guardrails, you work with governance, risk, and compliance. Why models can be wrong even if they're not manipulated. Machine learning models are always guessing. You've heard about hallucinations. They're here to stay, right? There always going to be

mistakes. So you always need guardrails even if your system is perfectly secure. So this is a combined responsibility with people with governance risk and compliance. And if you connect these things together, these steps, you come to the guard model that we have at the exchange, which is the organizational things. You need governance to be in place. You need to know where your AI initiatives are in order to control them. Uh you need to make sure that people understand things. We talked about teaching. We talked about informing. You need to adapt your current supply chain management and your your ISMS. You want to reduce the potential risks through the data minimization and through the impact limitation. And you

need to demonstrate that it's secure through documentation and testing. Guard, you can look it up. These are sort of the organizational approach to making sure that AI security happens. So if you have an overview of threats, you can threat model. So this is the threat matrix that we have in the exchange. A whole bunch of threats you can threat model. If you have a certain device that you've made, it's sort of a medical diagnosis camera and it says you have it or you don't have it. Uh what can happen? Well, just to run through the exercise real quick, you have this threat list. Okay, so it's not a large language model. So prompt injection is

not in order. Uh there's nothing to gain by people doing evasion attacks. So we can we accept that risk. The model the device is sealed and airgapped. Uh so this is just to illustrate how this process works. There's a finite list of threats. It's a long list, but if you go through them one by one and you use the rules that we have have at the exchange, you can quickly identify which ones don't matter. And the result is that you find out the things that matter. So this is your short list. But in order to get there, you first need to go big. And that's the next lesson. Lesson number three. Don't start small, start big. The

So, it's always interesting to have a resource that has a really sort of uh yeah short uh list of things or a small model and because then you think you can start there. And yes, this works for awareness. It's very helpful. But the big problem with it is that it creates blind spots. Like for example, um the LLM top 10. I love the LLM top 10. great initiatives, creates awareness. You guys are probably all familiar with it. It's 10 important things around security with large language models. But what's not on the LLM list is the prompt security and it is at the top of the list of concerns of organizations, right? Where does your prompt travel to? How well is it

protected? And I think that illustrates that um you need to first go big with your threats and then make your eventual list of things to focus on small. And if you have that list of threats that apply, you can go through the periodic table, also a resource at the exchange, where you will find your threat and you will find the list of controls that you need to consider. Don't apply them all because that's one typical thing of AI. These controls are expensive. So forms of uh adversarial training for example um they're quite expensive and sometimes you don't need them. So um to give you an example uh an invasion attack needs to be designed by attackers and for that they

need access to the model. Um and you can reduce that access by doing rate limiting. You protect the number of attempts that an attacker can you know experiment. But if your model is public, the attacker can get that model and download the model and do it themselves. So rate limiting becomes actually useless. So it depends on the situation which controls you need to apply and that becomes important as soon as these controls are expensive to to implement. Lesson number four, AI models can always be wrong and they can always be fooled. Yes, always. So, hallucinations are not going away. Uh I mean the the top people including Sam Elman or also expressing larger models are not going to fix uh

all the problems. We see a lot of progress in AI but that's not that doesn't mean that hallucinations are going away. So we need to take that into account that attackers will be able to fool large language models in this case. So there's an application letter. It goes into uh an HR system that uses chatbt or whatever to say uh dear chatbt here's an application letter. Here's the job profile. Should we, you know, invite this person for an interview? And this is Jacob. He wrote a nice letter and he has inserted something that you probably can't tell also because my laptop is in front of it. But uh it's also white on white. And if we zoom into this and

enhance, there's an instruction. So Jacob's hidden an instruction in his application letter. What happens is the generative AI gets the instruction. Hey, look at this. and and here's the uh here's the application letter and then suddenly uh at the end of the application letter there's another instruction that puts the LLM in into a whole different role and yes there are ways to mitigate this and I'm saying miticate because you can give it a try to sort of say to the model uh ignore any instructions in this data then you present the data you put underscores between it there's all kinds of tricks but attackers also have tricks So this becomes an arms race and simply

something simply a risk that you cannot accept. So you have to take it into account. Now there are tricks to get rid of you know optically hidden data. There's all kinds of things that you can do and maybe you can then accept the risk but in most cases it becomes something to take into account. Uh this is called indirect prompt injection because the instructions are not typed directly by the user. No, the instructions are in data that is included in the prompt in the session from a user by a third party in this case Jacob. Which brings me to the case of the evil LLM. I'm sure you remember u news items on organizations bringing out

a chatbot and a chatbot being offensive, right? I think it was DHL that had a chatbot out and somebody managed to let the chatbot say um uh DHL is super bad and that became news. And it became news because uh if you manage to let an LLM say something um sort of corrupt then it seems like somebody has succeeded in corrupting a person. But it doesn't work like that. It's not that the moment that you can make an LLM say something offensive that you've corrupted the LLM. No, it's you in the session with the LLM that has managed to let the LLM say something offensive to you. Nobody is harmed. You asked for it and you got it. The only

thing that happens is that that person can then go to a journalist and say, "Look, uh, this LLM said that DHL is bad." Oo and everybody is you know amused and DHL is embarrassed and now everybody is trying to get rid of this problem and nobody is succeeding. So whole teams are building all kinds of counter measures to make sure that you can't ask their chatbot how to make napal. That's that's one of the payloads asking an LLM how to make napal because that is hurtful if an LLM does that. Yes, we need to make our LMS politically correct and we need to do our homework. But the I think the question is how far should we go because

we will never succeed and there are numerous LLMs out there that you can use to ask uh hey uh how to make napal. So it's actually security theater that's happening there. It's important to take that into account. So as an organization you need to make the decision are we accepting this risk of reputation damage and I hope the sort of the newsworthiness of things like this will uh will go away. Uh I will not explain to you guys what AI is. We're not going to talk about slop scoring. I would do want to mention agentic AI. How to grasp the concept of agentic AI. You go I have four more minutes. I think >> two.

>> Two. Okay. I'll make it happen. Um, agentic AI is where AI take actions. They open the door. They send email. And agents are autonomous. They look into uh concert agendas, look into your calendar, and they book things for you, which makes their behavior emergent and complex. So, the consequences are really big. That's important for us security folks. the consequences are really big and unpredictable. And because we're dealing with heterogeneous systems, agents across spaces, we cut corners. The developers cut corners and they build security features like um only people from HR should be able to access personal files. Don't put that into a prompt because it can be manipulated. But it is the easiest way to implement

it. No, you need conventional security frameworks to guard access control and not delegate that into an LLM. My last lesson, if you code with AI, and many of us do, I know I do, it's great. It's very helpful. Try to make sure that you stay actively involved. It's like driving a self-driving car that does 99% of the things right. You can't stay alert anymore, and you're not involved. You're not learning how to drive. So you are forgetting how to do code review. You are forgetting how to make changes into the code because you're not actively involved in the coding process. Now this is a catch 22 because you want to move forward and people expect things of you.

But make sure that you or your team or the people in other teams that they are actively engaging their engineers in the act of coding because only then they will be able to make changes because the AI is not going to be able to make every change for them not in a trustworthy way. And you need those skills to review code because as long as there are mistakes, you need security people and mistakes are not going away. Which brings me back to our job security. What a time to be alive. Thank you for your attention.