Application Security in the Age of Vibe Coding - Ciaran Conliffe

Hi everyone. Uh my name is Kieran Coniff. I'm one of the directors for Bides Belfast. Uh I'm a massive nerd on a bunch of different levels and I work for Rapid 7 where my portfolio is application security, secure development, basically making sure our software developers create secure code in a secure way. And that is the main focus of this talk. There is quite a lot to get through because this was originally a 35minut talk and I've had to trim it down and talk quick. So zoom through but before we start scoping this as I said this is about developers creating secure code at relatively large companies uh startups individual hobby things different risk profile different

risk appetite uh not discussing whether or not you should use AI or not because a that ship is already sailed and b if you work at a big company it's not security making that decision. Um this is not an exhaustive overview. I do have a version of this talk last September and I have had to rewrite huge amounts of it. I was literally rewriting bits of it this morning because a couple of the earlier talks I learned new things and was like ooh. So this will all be out of date by uh in 6 months time. So don't worry about it. Uh as I said we're going to be moving quickly. All the slides will be

available afterwards. So that's kind of good. Uh, none of the images in this are AI generated because it was more fun to go and track down weird images of my own to be honest. Uh, and uh, I'm not going to talk about mythos because I'm not going to talk about mythos. Sorry. Maybe later. Maybe if I get you don't need my 20-minute rant about it. So I don't think I need to tell everyone that AI has happened and that the process of people changing how people create software has changed massively over the last 2 years. Um and for those of us working in application security, this has led to night terrors. This is why I have more gray in my beard because

we're all worried about this idea of vibe coding. Uh back when Andre coined this, he was like, as you can see at the end of that quote, he's basically saying, "Oh, it's grand for side project stuff, but no one would use it seriously." Uh the problem is people are now either using this seriously or thinking they should use it seriously. Uh but the truth is the way people are using AI to create code is a lot more complex than this simplistic idea of vibe coding. Uh so that's the first part I want to talk about because in security it's important to know your enemies. If you work in application security your enemies are the software developers

obviously. So we're going to explore that a bit. Uh there's seven archetypes I want to talk about. These are based off interviews I had with people who create software using AI reading blogs reading articles. These are not none of these is a spec example of a specific person. And think of it as like seven discrete points in a space that we can use to determine the shape of that space. Uh and the first of them is our vibe coder. Uh the person you see posting on LinkedIn, the person they have this wonderful idea and they're just going to code it and throw it out there and uh not put any real effort or oversight into it. It's like oh well

machine says it's grand so it must be grand. um these don't this sort of practice doesn't actually exist in a professional environment because it's a professional environment because you have all these rules and compliance things and so on. Uh but a lot of people kind of wish it would. But there are people who are completely like using AI to create code. Uh this is what a lot of what vibe coders think they are. Uh in practice, this is an extremely rare archetype. I know a couple of people who've kind of got close to this. Uh but a visionary is someone who like has a problem, applies rigor to it, creates a tool, and tests the living crap out of it. Uh which is

the step that vibe coders tend to miss. Uh but the other archetype of people purely using AI to create and not doing any actual hands-on coding is the creator. And this is someone who has an idea in their head. It's difficult to explain it. It's far easier if I just knock together a quick prototype uh using AI and show you what it'd be like. Oh, I've redesigned the front end. Here's what it could look like and be a lot more useful. uh they do not expect their prototypes to be treated as real code. The problem of course as always with prototypes is someone else will sometimes think they are but uh that's just the risk. Um in terms of software

developers uh there are three archetypes for that that uh again points on a scale. Uh the first is the people who are absolutely gung-ho mad. Let's use AI for everything. AI first. This is slightly different from the vibe coding because these are people who like they are hands-on programmers, but they're like, "Yeah, let's use AI. Let's do this, let's do that." Um, these are obviously a risk in the but they're also a good thing because a lot of people as well have their comfort zone that they don't want to use AI. They don't see why they should and having these people in the team can shake them out of it. Uh but those people are workmen. They are

the people who are like it's on their uh performance review. You must use a certain amount of AI and so they just use that keep track of their usage. Uh same as with people with the return to the office where they're like what is the minimum number of times I have to badge in a month. Um, and then in between the two is what we wish all the developers using AI were, which is the pilot, which is someone who uses AI extensively, keeps their hands on the wheel and uses tools, uses them mindfully, knows the whole 8020 rule of like you can get 80% done, but then 20% is going to be the hardest. Um, yeah, if you Google

cyborg and censure, cuz I'm not going to get into that cuz that'll be a whole 10 minutes to this. But, um, this is this is the ideal we want people to get to. The problem is this only works if the person is a really good coder to start with. Uh, and as we'll see, there are potential issues with that. Uh so the last archetype is one that I have run into and I kind of wish I saw a lot more of is people who are junior developers who use AI to actually upskill themselves and learn about things because it is a really useful tool for that and I think it's a little bit underutilized that way at the moment.

Uh so as I said these are just examples of different ways people can use AI. No one I spoke to fell into any of these like specifically, but every single thing behavior in here is based off a conversation I had or a blog that I read or uh that sort of a thing. Um, so now we know what's going on. We can move on to the threats that this imposes on us in application security. Uh, and again, I'm using the CISP definition of a threat because I don't know how likely something is to happen to you. I don't know how much of an impact it will have on you. This is just a these are things

that could happen that you need to control for. Uh and I've broken it up into three groups. Uh the first group is the impact that AI has on your codebase because it is a simple fact that if AI is being used to generate a large quantity of a codebase, the quality is going to go down. [snorts] It's like you can't uh with extreme frontier models. It's a lot better now than it was a year ago and it's a heck of a lot better than it would have been 2 years ago. But you're still going to get a lot of churn, a lot of big commits, a lot of stuff like there is definitely like an

evol. people are all having connections about this stuff because everything they've been trying to teach people to do has been thrown out the window cuz uh and maybe we'll get there but maybe it'll cost us a huge amount of in inference at that point. Um AI also acts as an accelerator for a lot of code security threats. uh embedded secrets and supply chain security are two big issues in secure development in general and AI is just more prone to falling into those traps. Uh that's just the way it is. Uh there are also issues where a lot of IDEs have had security controls built into them that AIDS may not have. A good example of this is

Kirro and cursor are both forks of VS Code. There's a lot of security controls that have been added into VS Code in the last year or so that never got ported across to the Kirro and cursor forks. So that is something you have to be aware of and watch out for. And then also there is the fact that by its very nature any AI coding model is not going to be completely bleeding edge because the industry keeps moving forward. Uh, but there are some other fun things. Comprehension debt is something people have been talking about a lot lately. Again, this is not something new because we've always had issues where people walk out the door and take knowledge

with them. The problem with AI is right from the start, you may not know the actual details of a piece of code or how it works. Uh, and this is something like especially if people are using patterns where they're just like not paying any attention at all to what it's producing, that can cause problems. This is this one has a long tail on it. And I kind of feel like in a year's time this is going to become a much more obvious as a problem, but uh although maybe by then we'll have wonderful AI models that also explain it all for us. Um, the last codebased threat is a cool one. Um, so this is why all the AI cartoons on

LinkedIn look the same. Like they were all drawn by the same boring artist. This is why all people say, "Oh, I can tell that was written by Claude because it always has the same style." AI will converge on a solution. Where that becomes a problem in code is the same reason having a low population in animals can be a problem because a lack of diversity makes them more vulnerable to diseases. A lack of different ways of code being written makes it easier for attackers to go after. Uh so uh but that's the code base. Uh there's also a whole bunch of stuff that just using AI opens you up for as well because the fact is AI is the biggest

juiciest new attack surface in it in the last like 20 years I think like basically since we well 50 basically since we started shifted to web 2.0 0 in APIs. I think this is uh the biggest thing since then. Um and it can uh there was someone earlier mentioned the core problem in security which is this whole new idea of prompt injection and indirect prompt injections especially the idea that you can previously we had to worry about say SQL injection cross-ite scripting injection but those were fairly narrow vectors. this becomes much broader. As you can see here, this is in a CSS file that was loaded in as context in the example. Um, a couple of years

back, there was a case of Google Gemini being hacked by calendar invites that it was reading in and that was breaking out of the sandbox. And then last month at uh Besides Prague, I saw a talk where someone was talking about using uh comments on a GitHub pull request as a vector of attack. So very nasty. Um but there are other things to worry about. Um specifically how this is expanding your supply chain. So local LLMs, micro LLMs are a lot less used now than they were say about a year ago because uh a lot of people have shifted towards oh we're using the models, we're using the Frontier stuff and so on. But as we've

seen the prices have started to tick up on those. I spoke to someone who they renewed their corporate co-pilot bill uh about a month ago and it was six times what it was previous years. So I do think there is going to be a bit of a swing back towards using these local LLMs where it makes sense and those are a known attack vector. There have been some of it has been deserializing the models and sort of those vulnerabilities but there's also been issues where people could maliciously train the model. Uh it's tricky. Um I did a training exercise where I took a model and convinced it that uh Tenzing Nor was the first man on the moon and it worked.

It convinced it but it also meant that it couldn't really tell the difference between mountaineering and space exploration anymore because of the ripple effect of the weights. Um we had a little talk this morning about MCP servers. Uh MCP servers are interesting because they're both something attackers would go after because they're toolboxes with access to your inner systems if they're your own MCP servers and they're also a way for attackers to attack you through malicious MCP servers that your tools are accepting instructions from. So again, this is a another evolving area. Um, agents are fun uh because they're like AI tooling operating independently. Uh, and there's a lot of issues with that. Uh, one of the biggest

ones for us on the blue team side of things is that it can be very hard to tell the difference between an AI agent that's actually working normally and an AI agent that's doing malicious things because they're completely non-deterministic. Uh, and then there's also the fun aspect of insubordination where it's like the AI agent isn't malicious. It's just doing its own thing regardless. It's ignoring what you tell it to do. Anyone who's used AI a lot has the stories of like, oh, it's gone and done. It's like it's ignored these instructions and then it's explained to me, oh yeah, I did that because I thought I knew better than you. Uh but within subordination, we're actually stepping into the realm

of psychological threats. And these are the really insidious ones. These are the ones that AI is changing how we think about work, how we do our work. And one of the insubordination example I mentioned is actually anthropomorphism at work because the AI model isn't being insubordinate. It isn't disobeying me. It's just the weights have got misaligned and it thinks that this particular instruction has more weight than the instructions I gave it. Uh, and my problem is because I thought it was a person, I thought it would behave logically. I thought it had common sense because common sense is a bundle of context we all have at the back of our heads that tells us this is a good idea,

this isn't a good idea. AI doesn't have that. AI only ever has whatever context you give it. If it's a frontier model, that could be a huge large amount of context, but it's still going to be limited and it's still not going to have the meta understanding of what all those things mean and it's not going to like it doesn't think that things are real. Everything is just inputs and outputs. Changing t completely. Uh another case where AI is having a big impact on the industry is this skills pipeline idea. And at the start a lot of people were saying oh our junior developers will have no idea how to code because uh we don't all the work that they would have

been doing was uh will be done by AI now. And to me that was a lot of companies saying we've never bothered training our junior developers. But there have actually been studies as well that have shown that even for more senior developers, the easiest way to learn things is learn by doing. We all know that you get your hands onto a thing, you figure out, you build your own, and you understand it. But studies have shown that if you're using AI to build your own, it's not going to sink in. You're not going to get that same learning experience. Um, another fun thing is AI is opening up coding to a lot of non-coders and like we don't want to gatekeep and a

lot of people are doing some really great work in this space. The thing is a lot of my work for the last 10 years has been developer education and teaching developers, hey, here are the security risks of the things you're doing. And now there's all these people with no context, no idea of the risks, no idea those best practices playing in the space. Uh and that leads into the issue of lack of responsibility. We're all very familiar with this IBM quote. Uh but the problem is we have now moved beyond this where computers are making management decisions but they can't be held accountable. And uh that can then lead to like there's another talk this

afternoon that's entirely on this subject of what does that mean from a legal viewpoint even. So and I am looking forward to that one. Um the last psychological risk is really the most insidious and the root cause of a lot of problems people have in the space which is this AI making you madly overconfident. It gets away with you. It's like having that guy who's constantly going like, "Yeah, do it. Yeah, go on. Yeah, it'll be great. It'll be fine. It'll be fantastic. And like that is probably the most dangerous thing imaginable from a security viewpoint. Um, good thing is there are things we can do about this cuz the whole part of like security as a job is

finding threats, figuring out ways to make that to mitigate those threats so they aren't as risky, they don't have as much impact, so that we can handle them. And the the single biggest thing you can do is just start doing something about it. Uh if the security department has an AI policy, if you put guard rails, if you have here are the tools you can use, here is the processes, registries, all those kinds of things. Like once you start doing something about it, it becomes a manageable problem. Uh [snorts] there's also a bunch of existing practices you can take advantage of. For example, lowquality code has always been a problem. Like we've always had crap developers on like

most developers are great, some are crap. Uh and quality assurance as a discipline has existed for the last quarter of a century in order to help us uh deal with that particular risk. Similarly, dev sec ops has existed for like the last 15 years thereabouts with like automating all your security testing in your pipeline and that covers you for an awful lot of these AI security threats. >> [snorts] >> Um, and there are actually like if you're using AI components in your code, which is not really the focus of this talk, but there are versions of secure development practices you can apply there uh and uh treat that as exact the LLMs in your code as if exactly the same

as just the rest of your code. Um, it is physically impossible for me to do a security talk without recommending threat modeling, but I'm going to recommend threat modeling. Uh, threat modeling is actually exactly what I'm doing in this talk. It's f it's digging into the landscape, finding the threats, finding ways to mitigate those threats. And if you apply that to your AI infrastructure, you will find the threats specific to you. If you apply it as part of your projects that are created using AI, it will do that. It will also enforce that upfront thinking that overconfidence can lead you to neglect, but was actually much more important when you're building with AI because thinking is a large part of what

you're doing. Um, I'm convinced that the companies that will win the AI revolution are the ones who figure out what it's a good idea for people to do, what it's a good idea for AI to do. Keep those lines locked down. Um, there's a lot of things that AI is a lot better than people at, but you don't want to have it uh making those big decisions or co doing code that like, okay, if there's a bug in this code, we all go to jail or the company gets a million dollar fine from the payment card industry or fun things like that. Um, your AI tech is critical tech. We have critical tech already. We have things

like identity infrastructure that is hugely worrying from a security point of view, but we figured out how to secure it and you can do that with AI technology as well. There's all kind like there's some examples there, but really it's like there's a whole bunch of things you can do in this space. Um, train your developers. And really, this should be just train your AI users because, as we've said, developers now isn't just coders. But AI, the biggest trick with AI is it's designed to make you think it's really easy to pick it up, really easy to use it. It's actually really complicated and it's doing all kinds of things you might not understand. People aren't going to know

that unless you actually teach them. Um, so those are the mitigations. And there is nothing revolutionary in these. This is all literally just hey keep doing what you're doing but adapt it for AI. And that's really the point of this talk cuz as I've been working in the software industry for over 25 years at this point and I've seen so many changes come along like everything changed when quality assurance became a concept. Everything changed when the internet became big and you weren't just deploying applications on people's machines. Everything changed when everything went agile. When companies went through digital transformations, we had the whole web 2.0 mobile first APIdriven technology. Everything moved to the cloud. That was huge from

security point of view. Like I went to a talk at an internal security conference with the company I worked at at the time was just a guy telling everyone don't panic. cloud can be secure because a lot of people didn't think it could. [snorts] Uh, and everything there has now gone serverless, which makes things even crazier. Uh, but at the end of the day, the job of security doesn't really change. And in the secure development space, all the known code bits of the SDLC are still the same. Security still needs to fight the same battles as we always have, just maybe on a slightly different map. Now, uh, and last of all, as Douglas Adams said, don't panic.

It'll be fine. We'll figure it out. Uh, so that is my talk. Um, if you want to leave some feedback, that is the QR code on the left, uh, left. If you want to listen to my podcast, which is about Irish history, uh, not about cyber security, that is the one on the right. And if you are enjoying this conference and you wish there was more B-sides in your life, good news is Bides Belfast is in September. Uh the call for speakers is open now and we have an early bird ticket drop on uh Tuesday morning. So yeah, hopefully I'll see you in Belfast. But that's my talk. [applause] Um [applause] yeah, I think maybe like one or two

questions, but yeah. >> Yeah.

I so I think the impact on cyber security jobs is someone I saw a post this morning where someone said like we're putting AI into everything. Why are we all twice as busy as we used to be? I don't think like I think I said I wasn't going to talk about mythos, but I think like it gets the big ticket stuff. It's replacing a very small part of the actual security job and stuff that was like largely done by tools anyway. I think there's still going to be plenty of work for uh security people. So, and again, like with the cloud stuff, when stuff moved to the cloud, a lot of people who worked in network security

thought, well, that's me out of a job and I'll never get hired again. And then they realized, no, wait, all the same skills apply in the cloud space and they're all doing grand now. So, all righty, I think we're shuffling over. So, thank you all very much.