BSides Berlin 2023: Natalie Pistunovich - Security in the Era of LLMs and GenAI

give you an overview about the current state of AI security and make sure that you know everything about how to secure your AI powered systems please welcome [Applause] Natalie thank you very much if anybody's standing in the back there are five or 10 seats here in the front if you want not offended if you're moving everything's good so this chat this talk is about Security in the era of LMS and geni and I introduced myself previously a little bit now I'll introduce myself a little bit more I worked for about 10 years in the field of Beck and devops Engineering Management I'm teaching devops at the Hat here in Berlin I'm security Enthusiast I'm not working in

that but I'm going to Defcon and hanging out at cbas and so on and I am a developer Ambassador with open AI in the last three years I'm also been following everything that was happening tonight completely crazy no idea but it's over fascinating um over the last couple of months I was doing a podcast with a friend from who used to work at kaspari you can find it on Spotify we did seven episodes until we decided to stop I'm bringing the AI part he's bringing the security and we're kind of just discussing about all sorts of interesting things you're welcome to listen if you're interested this is how it's called our logo is this Mony and my plan for today is to very

quickly go over the history of llms and chat GPT and kind of everything you need to know other than then that you can tell it how to give how to make a bomb out of your grandma's Candy Factory h a little bit about the different other tools that are out there like voice audio image and so on and some conversation on autonomous agents and we start chpd everybody knows and loves anybody in the crowd does not pay for it but you don't pay for it I mean not your employer like you're just using the free version okay interesting this side more so after Chad GPT came to be this was the most uh boost um in product

history I guess and um gpt3 is when I joined the whole thing that was a very different UI that was a very different type of a conversation that was not chat that was more completions and that um I want to say one more thing that as a developer Ambassador you can go to the website and book office hours with me or with somebody else we have five developer ambassadors on the open website you can book office hours and come with any question that you have in the time of gpt3 it was kind of like helped me wrap my hand around the concept of completions but ever since chat GPT became a chat H it became a lot more simple and

the questions became more around product and so on then came plugins anybody built a plug-in I did I'm organizing a conference that's called goer con as you might have seen from the emails and uh we buil a plugin for that conference where you could chat about the agenda and kind of say this is my background this is what I want to know and so on what would you recommend for me to go um plugin started with eight and then it became a whole App Store that can do all sorts of different things and this was a very interesting technical Improvement because not only Chad GPT is now able to understand what you want and have a

reasonable conversation with you it was also the the technological innovation of here and API and now translate what the user says into the API call and lots of developers were like is it going to take my job then came functions because the demand was like this is an interesting thing I actually don't like doing you know working with new apis I want to give you an API and you figure out chbt you figure out how to handle this and then not necessarily as a plugin but let me use this as a like as a tool so that's how functions came to be basically you give it an API and then you say anything that the user gives

translate it to a call based on what the um API is popular people liked it and it's pretty cool then the next development was fine-tuning fine-tuning is kind of the next level in that personalization so you give it you train it on some data and then it does a specific behavior as you want it the way to think about it is not exactly the what but more the how so it's good for things like corporate voice right you have to the marketing team wants to write all the messages in a very specific way if you train it on all the announcements you get it to speak in the sound that you want before the Json toggle was announced at the

death day you could have also taught it how to with the fine tuning taught it how to go about very complicated nesting Json that you have and so on because it it was not consistent in Json as it got complicated and generally a good recommended uh flow before just jumping to fine-tuning is doing your best in prompt engineering then doing prompt chaining so kind of several steps then um putting functions and the very last thing is H fine-tuning this does not come instead this comes on top of each other this is kind of generally the best practice of how to use those different features and there's also custom instructions which is sort of fine-tuning but for the uh your

conversation with chat GPT if you want to set your uh voice and tell it how chat GPT should be answering to you this is how you're doing it and this is where you do this then embeddings came to be embeddings is basically a mathematical representation of data you take lots of data you turn that into vectors this is how it's visualized and the recent thing that was published since two weeks it's the gpts where it's sort of like plugins but actually personalized gpts because everybody was like but llama is really cool and I can use llama for free and I can find T it on my little piece of data so gpts basically allows you to do that

and you can there will be an app store around that you can use it for all the different things and PO is a website that has nothing to do with open AI you can try all the different um AI agents and models out there uh you don't have to be logged in so if you don't have access to gp4 like there was the weight list back then or maybe to Claud this is how you get to chat with all of them not good for production great for playing around po.com this is by Kora last thing is the ecosystem so lots of new things were created around that like vector Debs lots of companies who try to build their own models and goes

anywhere from anthropic which was by people who left open AI to I whatever Elon Musk has in mind but also B and other players that we don't often hear about in Europe lots of companies came to be around that many of them are in the field of securing AI some companies that I found that recently raised money for example one is called wraith watch one is Cranium or serum don't know how to pronounce their name they have raised very um respectful rounds and they're all doing AI security some are securing with AI some are securing the AI oasp they have published uh some months ago a list of um vulnerabilities in the context of llms everything that we saw now in the

brief history of llms H we're going to see how that relates in what OAS was saying and the first one obviously prompt injection you can manipulate the behavior by prompts that override the design right you tell it to do one thing or design it to be one thing it becomes another thing I think everybody try to make the prompt behave in a or use prompt to make it behave in a different way insecure output handling just like bringing any external user into user input into your system if you don't validate what's going on that's a bad idea training data poisoning this is probably the hardest one to solve anybody has a startup that's already trying to solve

this not yet okay soon um yeah collecting data from the internet and making sure it's not poisonous and it's not like setting some back or like some key word that will activate the unexpected behavior and whatnot I I expect one of those security breaches to happen and honestly I don't know how to go about that being safe just training data is huge for those models and you cannot possibly secure all of it and scan all of it or can you um model denial of service like the Doos that we saw on Chad GPT last week supply chain vulnerabilities data set models and whatnot it's a bit like using unsafe Lang languages programming languages if you use it in a

safe way then you Pro and put safeguards and whatnot you're probably going to be fine sensitive information disclosure it's not deterministic you need to make sure that no private data is being exposed and so on I know there's a a whole talk later today about this topic insecure plug-in Design This is dated the document that I took this from from OAS is from October 13 as we said two weeks ago gpts were introduced I expect them to replace plugins but it's really the same thing um people develop those things on their own just like software can be insecure plugins gpts will be insecure excessive agency so especially in the context of autonomous agents um on the one hand it's great that there's

something that will automate all sorts of things that you don't want to do on the other hand you need to make sure that they do this in a safe way over Reliance the completing side of the agents having a lot of Aton omy is relying too much on their outputs and not checking what they do model theft is you know the bad outcome of this is you just lose money but the good the the the good bad outcome is when you lose a lot of money the bad bad outcome is you also lose a lot of data um generally I can say that I see a lot of comparison between this and software in the sense that how models

are a little bit equivalent to programming languages Plug-In or whatever it will be GPT is kind of the software so a lot of the practices that you have for how to hack with C or or whatnot um versus all the different practices of how to keep your software secure I see many of those things that be carried over into this world and you can use the same practices there's a lot more to invent especially because of the non-deterministic nature different scales and so on but the low hanging fruits are already there now some fun examples voice generation so there are two big actors in This World for for voice generation it's 11 labs and for voice

transcription it's whisper and I'll show a short video or short snippet from 11 Labs I didn't request an advanced voice is there any chance to spontaneously let voice um let's see who will be gracing us with her natal state of theart I just like this one greetings from the chancellor so H to use 11 Labs you all you need is to give in some text and a voice sample the for those who who did not use 11 Labs you have to have the consent of the person who uploaded this obviously I did not do this so let's keep this between us but in the world of fake news this is a big deal in the world of social

engineering this is something that's going to be hard and complicated the V the QR code just leads you to a video to a YouTube playlist of all the demos that you'll see um yeah social social engineering or fake news I don't know which of the users will be more scary with this but it's it's quite easy and yeah you have like the terms and conditions but not all hackers follow those uh then there is speech to text so whisper is the most popular AP that you can use for transcribing they support 57 languages and immediate translation to English I can imagine also in the field of um social engineering having this very useful tool right you can imagine somebody having a

conversation calling some support center and trying to you know figure out the color of their whatever um um truck does their cleaning service you you can transcribe the call live you can send the those queries into chat GPT you can send it it back and then kind of have it give the the the script for you if you do that not a recommendation next is audio um meta has the most popular tool for this right now it's called metag genen a music gen can I ask for sound again so music gen you just go to hugging face you put a description I put there um when the 80s meet Electro and metal and I received one of the Bands

that alike I think if you hear them you might see the [Music]

similarity now was trying to think what can be an used case for that and um nothing specific came to mind but then again I'm not a social engineer maybe somebody will find me later and give me some creative ideas how music can be in the service of this maybe f fake news you know create for me a news that I want to say that they reporting from one land and we'll open a music tune and this will be like close enough yeah it sounds like a real show image obviously Del 3 and um Can generate the very first model that you put text and you get images and over the years over the two and a half one and a

half years that it was alive it improved quite well now you put text and you get a very good resolution image with a good number of fingers and so on Wars are going on right now we see all those pictures we know this is in use but the interesting thing is that you can use um instability something that is called control net to control net you give an image and you say extract the pose so what you see those fingers or sticks is what you get um as an output from this image with a skateboard and then you can move those around just like a toy or like a doll and then you say this is the final pose and then say say

generate lots of image for me and then make those images be in this position this is how it looks um this is fun for the gaming industry but I can also absolutely imagine this being used for things like fake news or again social engineering images we have pictures of you doing things um click this link or else and also mid Journey does image generation yeah then there is video for video there are two big platform right now one is called Wonder Dynamics by Wonder Studio AI on their board is Steven Spielberg he believes in this new field um here this tool what it does is that you take a video and you plant inside a

virtual character that interacts with the existing video so for example this person interacts with that person and in the movement in the speak everything it sounds real fake news fake news yeah it keeps being a very useful tool for this as well Runway ml is another tool that you can use for this purpose and here's a video I made about myself because I don't want to use pictures of other people can I have sound Pantheon of Heroes one face is surprisingly familiar hers is the face of Natalie piston oich fun and inspirational thank you very much um my point point is that it's not hard I am not an artist I am very much person who writes digits on the like

texts on the screen and sees what happens and still I manag to make art um to to make such a fake video so it it will be you know it doesn't always going to be so nice and and positive um here it was just making some pictures with mid journey and then using Runway ml to kind of do the transition between them but you can imagine that if you want to fake some video it's also going to make some images and some scenes and it's going to be quite easy to say be in this specific pose or you know wear this type of shirt and whatnot and make the video between them and it's going to be hard

to make it maybe there going to be tools that will say this looks fake this looks real um some ideas inspiration if you already know about those things um from all the images and all the videos I can imagine that steganography is also going to be a big deal you're going to be planting lots of information there a lot more videos will be shared I don't for the same practices um of compression and like where you download how from how are the last bits looking like how is this going to look in this world is it something that will help you understand more or understand less is it fake is it not fake interesting I'm I'm curious to see

and I hope that people in the crowd will come up with cool ideas next year come and tell us about your cool projects multimodel is when you take all of them at once so chbt now does it all you can upload a picture and have a conversation about that burned my hand I have some creams at home I forgot which one I need to use I just put them next to each other is this the right one yes good um here's an image of a dashboard write for me the code to build this website perfect for fishing emails uh lots of pictures that are instructions one after another what does it say this can also help you with

things like here's the picture of this office help me where is the entrance that I want to go or here is I'm running out of examples but you can have a conversation with it well things you see breaking captures um Chad GPT is generally he generally a helper of yours I will skip the audio here but this is me having a conversation with Chad GT to ask who should we pick as example for a voice I think it worked well I think you all recognized it jumping and the last thing is autonomous agents autonomous agents the two most popular ones are Auto GPT and baby AGI both of them are just GitHub repos you can run locally if you want

you can give it permissions to access different apis or even do things on your behalf start slow when you give it a trans start very slow give it very gradual capabilities um any you re Mor defans in the crowd so autonomous agent you give it a goal unlike with Chad GPD that you give it a task you give it a general goal it breaks it down to tasks on its own then it proceeds to carrying out the tasks and then it comes back to report the whole thing and then it's gone those are the two repos and the last thing that I will mention is that open AI has two programs one is called cyber security grant program you can go

there and fill out any idea that you have that is related to the defensive not offensive side of the context of AI and you might get Grant from them to they have a red teaming Network that you can join recap we discussed a little bit llms chat GPT history and all the different tools that you can use there and we cover the different options or the leading tools right now in all the other fields voice audio image and so on it was a lot and all the demos are in this link thank you very

much