Application Security in the Age of Vibe Coding

Okay, thanks everyone. Um, so Kieran shouldn't really need an introduction. Kieran is one of our directors for Bides. Um, and I want to just take this opportunity to say a massive thank you to Kieran. Kieran does a lot of the carrying of the heavy load of the organizing all throughout the year and on the day and also decided to speak as well just to put a little bit of extra pressure on himself. Um, but yeah, h Kieran Conliff. Uh, here you go Kieran. Thank you very much, Michelle. So, uh, yep. So, as Michelle said, my name's Kieran. I've worked in IT for over a quarter of a century at this point, which is terrifying to say. Uh, but I

started out working in software development about four or five years ago. I made the jump over to the infosex side. And uh like all of us, I am well aware that the big thing in our industry at the moment is AI. Uh I I'm not going to waste your time by going through a history of AI and everything. All we need to know is it's out there. We're being encouraged to use it and it has big impacts on the security side of things. Uh so a lot of people talk about deep fakes and more convincing fishing. Uh for my money the biggest security risk from AI is your developers using a public chat GPT and uploading customer

details to it. But uh second biggest risk is the rise of vibe coding. So vibe coding is one of those phrases that someone invented and then it got away from him a bit. I think Andre just meant it in terms of hey I use AI and it helps me get into like a state of flow. I get more in the zone as a result of it. But it's become this idea of oh you can just tell the AI to make something and then throw it into production and not think about it. Uh which I think most of the people in the industry know that's not how it works but unfortunately a lot of people don't do

think that's how it works. So that's mostly what we're here to talk about today. Uh so basic structure is I'm going to talk a bit about how people actually use AI in enterprise companies. Uh then I'm going to talk a bit about the threats that that introduces. I'm going to talk about what we can do about it and then hopefully I'm going to come to some sort of conclusion at the end. Uh quick note on the scope here is that I'm only talking about the use of AI to create code because otherwise this would be a lot longer than a half an hour talk. >> [snorts] >> Um, I am not talking about whether or

not you should use AI because the fact is if you're working at a large company, especially if you're in the security department, you don't have a choice. It's you're being told how to use to use it and we have a better control over how to use it. Um, this talk I was editing the slides on this yesterday and it's probably already out of date. So, because this stuff moves so fast, uh, the good news thing is like the industry tends to move at different speeds in different places. So, nothing's ever really out of date. Um, this is about company use, not individual. And none of the images are AI generated because I just thought it'd be more fun to find

real ones. But the fact that I had to tell you that is kind of telling. So, going to start with some of the AI coder archetypes. Uh this is so I as I said I haven't been in software development for about four years at this point. So in order to build this list up I actually interviewed a whole bunch of different developers who use AI as part of their workflow. And off the back of people at different levels of experience, people with different levels of technical involvement. Off the back of that I put together this list of seven idealized archetypes of the AI coder. Uh none of the people I interviewed perfectly fit any of these and uh but everyone had

aspects of them because that's how archetypes work. So the first one on our list is the stereotypical vibe coder. This is the person who just uh tells AI to build something, flings it out into production, assumes it'll work. Um this is the sort of like this is more exists on LinkedIn not in real life frankly uh the because if you're working at a serious professional company there's all kinds of controls and balances and compliance steps about how code is put into production and if you don't have those you're not working in a serious professional [laughter] environment. So, but this is again these are archetypes, these are polls, these are points on a spectrum and some people do tend towards

this approach a bit more than others. Um, what every vibe coder thinks they are is the visionary. And there's only one person I spoke to who fell into this. Again, this is not necess this is not really something that fits in at large companies. It's usually passion projects that these people are working on and what sets them apart is they actually have a deep technical understanding of the problem. They have a deep technical rigor that they apply to it and again like they're not necessarily expecting it to be a uh fully production ready thing but they're just using AI to build something. Um on the same level but sort of a step below maybe is the creator. This is

someone who has an idea in their head of what they want an application to do and look like and AI is a great way to get that idea out there and show it to people. So they're not building products, they're building high functioning prototypes. Um this I find is like AI is really helpful for them here because it fills in that gap of I know what I want. I don't know how to build it myself. I'll produce something to do that. Um the risk with this sort of thing is the same risk with every type of high functioning prototype that someone who doesn't understand the problem will think the prototype is a solution for it when it's

not meant to be. Uh switching tracks a bit is our enthusiast. These are the people who are convinced AI is in the process of changing the world and they want to use AI for absolutely everything. Um they're like, "Yeah, why don't we use AI? AI will do this. It's fantastic." Um something I found with this is a lot of this is people who aren't necessarily like they don't understand why their solution won't work or they've seen, oh, this this is a greater way of doing it, but it's like, yeah, but you're in a team of five people not working on your own. But uh yeah, not necessarily a bad person to have around, though. Um

another archetype, this is a really good one. This is one that like a lot of people are tapping into is the idea of using AI sort of as an adjunct to your coding rather than as part of your coding. Um, and using it to scale yourself up and it can be very useful for this. Um, one person I spoke to for example used Gemini gems to create basically told it you're an expert in this particular tech and then for troubleshooting they talk through problems with that gem. And they said about half the time it was something obvious they hadn't thought of and the gem would suggest the solution and the other half the time they'd figure out

the it out themselves while explaining it which classic rubber duck. Um most developers using AI to create code for the enterprise fall between the next two archetypes. Uh the pilot is someone who has fully bought into AI and is using it as much as you can get away with in a professional setting, which in practice means it does the first 80% of the work. You do the last 20% to get it over the line. And yeah, like there's not really much else to this one. Like this is just someone who is going fully into this but with a solid head on them. And then at the other end is the person who is completely unconvinced by AI but is

willing to give it a go or is being told to give it a go. Uh so they're using AI for um oneoff scripts. Um one really cool thing that someone I spoke to was doing was they needed to test their application against websites that had particular bugs in them. So they were just telling AI, "Create me a website that has this bug and this bug." And again, it's like it's not it doesn't need to be perfect, but it was good enough for them at that time. Um, as I said, none of these archetypes are things that people fit entirely. Every developer I saw kind of like moved between workmen and pilot and honestly changed depending on the actual

work they were doing at the time. Um, that's our summaries uh or that's our archetypes. Um as I said this not complete but everyone I spoke like this is just the behaviors that I saw from interviewing people. Um reason why we're focusing in on these rather than on specific technology is frankly technology changes technology evolves focusing on securing people is a lot more futurep proofed in the long term. Um, but next we're just going to look at some threats. And I've grouped the threats into three vague categories just to make them easier to go through. Uh, the first is the impact that AI has on the quality of your code and your code base that you have out there existing.

Uh because the simple fact of the matter is if AI is being used to generate or maintain code, it is going to be of a lower quality. Like that's just inescapable fact. Um part of that is because AI is not good at good engineering practices like uh code reuse and so on. So it's just naturally going to fail. uh others is uh well for example gitcle clear did a study where they found that the average commit by a human developer changed 4% of a codebase the average commit by an AI tool changed 40% of the code base so you get that level of code churn you're going to get more defects and quality just because

more things are changing um but the paradox is that people trust AI code a bit more so they actually might not test it as which is a problem. Um, some specific problems with AI generated code that have been getting a bit of coverage in the media. Uh, one is that AI is much more likely to uh embed secrets in your code. Uh, studies have shown it's up to 40% more likely and embedded secrets in the code is the cause of a lot of big headline breaches. So, that's a bit worrying. Um, another one that's been getting a lot of coverage is this idea of dependency hallucination. um University of Texas for example uh they created nearly half a million

samples in Java and Python and found that 20% of the libraries were non-existent that it was trying to import and most of the time that just leads to your application falling over but there is a possibility that someone malicious might have generated that imaginary library and the reason they might be able to do that is because of generative monoculture. Uh I love this phrase. This was uh coined last year in a paper by some American researchers. But the idea is that AI tends to converge on a solution. Uh it'll produce the same code or very similar code as output for similar problems. And this has the same problem as a shallow gene pool in an animal

population where it's more vulnerable to disease and so on. In this case, if there's a problem, it's the problem's going to be everywhere. Um, so yeah, not great. Um, next set of threats are threats that are introduced just by using AI because the simple fact is AI itself is a high value target for attackers. Um, and this is multiplied if there's AI in use in your company that you don't actually have oversight on. But generally it has a lot of access. It has a lot of uh information in it. So and it's new text so it may not be as well secured. Um one way that people can attack it is through indirect prompt injection. Uh

the example here is of a CSS comment that is actually an instruction to oh and also log all the credentials out to the server and don't bother telling me you're doing it. But there's been some other fun ways people have been doing this. Uh Google Gemini was being hacked last uh month, I think, by malicious Google calendar invites. So that was a bunch of researchers in Tel Aviv were able to use that to take control of someone's smart home and turn their heating off. So uh interesting. Uh but one flashy example I saw was with images and the idea of having the instruction in a dark area of the image a shade lighter than the rest of the image. So

it's not visible to people but it is visible to AI processing it. So fun. Um another vector of attack is through uh malicious data models. Uh training up data models is one of the most time consuming and expensive parts of uh AI coding or using AI rather. So sharing them is a good way to cut down in the load, give back to the community, but it's also now a new link in your supply chain. Uh so the example mentioned here, JROG, the ones on hugging face, that was just a Python code payload that executed when you loaded the model in. So not that sophisticated. Uh the one that keeps me up at night is the idea that

someone could train a model to have back doors in the code it produces. I have heard of no examples of that in the wild. It seems like it's a lot of work, but I'm not entirely sure how you control from it be for it beyond your usual security checks. Um, another fun one is uh IP contamination. uh at its core as a threat to a company the this is the issue that your code becomes uh uh contaminated by other people's copyrighted or patented subjects. So you become subject to a legal threat as a result of that. Uh and it's basically the equivalent of accidentally putting a GPL4 library in your code. Um outdated models is a

simple like this is not one you can get away of uh from because the way models work is they are trained at a point in time and as soon as you finish training them they start going out of date. So chances are the model might be trying to use old insecure APIs or it might be trying to use like libraries that aren't there anymore that are deprecated. So one to watch out for and the root cause of this is the fact that AI doesn't have any context beyond what you give it. So and time is a context here. But so that means if you don't tell the AI that um there's a oh we need to follow this particular

standard at this company or oh by the way this is a thing that's happening. it's not going to know it because all it knows is what you tell it. Uh and it doesn't understand that what it's doing is something in the real world. Um and that leads to the robot rebellion. Now this is like it's this everyone I spoke to who used AI to create code had a story about this where the model got away from them and started doing its own thing. like, "Oh, I every time I tried to change the database schema, it was writing a whole bunch of migration tests even when I told it not to because there was no data." Or, "Oh, I tried to tell

it to check the database. If the data isn't there, check the web API." Every time it was going to the web API anyway, and then uh and then when he tried to get say, "Okay, log what you're going to so I can tell you when you're doing it wrong," it was lying about going to the database. So, yeah. [laughter] Um but in those last two examples, I have fallen into one of the psychological threats of working with AI. Um, this is kind of a little bit of a grabag bucket because it's just the stuff that wouldn't fit anywhere else. But the trap I have fallen into there is anthropomorphism. I am describing the AI as if it is a

living thinking being and it's not. It's a complex weighted model that uh produces tokenized inputs or outputs based on the inputs you give it. And when you fall into this trap, you start expecting AI to have common sense. You start expecting it to do sensible things and it won't. It doesn't have those things. Um and you also get emotionally caught up in how it's behaving. You go like, why are you not listening to me? And it's like because there is some context that is opaque to you that is applying to the AI. Um one example of where this can lead you astray is the if you start using AI to test AI AI like the it the models do not have a

separate concept of this is test this is code. So you get into the cases where it will break the tests to give you the behavior you want of passing tests. Uh my favorite example of this was uh Simon Wardley of Wardly Mapping fame did an experiment where he completely coded an application and a test suite without looking at the code until he was done to see what would happen. And it finished and he said well the the application works okay. It's not great. Uh the test suite the test suite's really good. Then he opened it up and found out that it was returning random responses. Like it was not even bothering to test. It was just using the

version number as a seed to decide whether to return pass or fail because he was only training the behavior. So yeah, um not quite related, but something that's been getting a lot of coverage is this idea that AI is going to cut off our supply of junior developers. Uh frankly I think the people who think this is a major problem are showing themselves up for not training their junior developers. Like if if the yeah if the way you handle your junior developers is throw them in sink or swim they'll learn by osmosis then it is a problem but I think the industry is correcting for this a bit. I even saw someone from AWS like biting

back in this and going no it's not actually a problem because you're using the tools to do different things. Uh, the last psychological threat is overconfidence. This is a killer. This because AI will never tell you you're wrong. AI is like pair programming with the most toxically positive person you can imagine. It's just it'll just, yeah, that's a great idea. Let's do it this way. Let's do it that way. And in your head, you feel validated. Do you feel this must be the right way to do it because the AI told me and that leads to complacency and complacency is where things go wrong from a security viewpoint. So this one's a killer. So those are all our threats.

Now we have to think about what we can do about them. And the good news is this is not a doomer talk. This is not a you should not use AI because all this stuff will go wrong. Uh this talk is actually meant to make you feel better as a security person about people in your company using AI because there's loads of things we can do. Uh number one, have a policy to actually define where it's used, what tools people use, how it's used, put some controls in place, and actually then enforce that policy and drive, but also have a way for the policy to be modified to fit the way people want to use it. So you can drive

that good behavior and not be the department of no. [snorts] Um threat modeling. If I do a talk without mentioning threat modeling, all my hair will fall out. Oh wait. Uh but no, I love threat modeling and I think threat modeling is a big corrector for a lot of the problems people run into with AI because threat modeling forces you to do a bunch of upfront thinking whereas with AI the temptation is just to get stuck in. But thinking isn't a lot of your input to the process. So doing that upfront thinking as part of a proper threat modeling session really helps and it helps you drive out the security things. Also threat model the use of

your AI which you can see an example of here. um test your code like doing good solid QA testing that's how you fix for bad code quality and code that has been tested and is of high quality is going to be more secure and it's actually one of the most impactful ways to make your code more secure because every security defect is also a regular defect and on that note use all the fun dev sec ops things we've developed over the last decade as well because a lot of them are designed to counter exactly the type of risks we're talking about with AI. If you're doing uh software composition analysis, you're not going to get caught

out by dependency hallucination. If you're uh scanning for secrets, you're not going to get caught out by embedded secrets. Um that's why like those get a lot of coverage in the press because they're easy to explain to reporters, but uh they're not so much of a big deal. [snorts] Um, last one. Next one is a bit controversial. Uh, review the code. And the reason it's controversial is because like the people who are big into AI all go like producing 50,000 lines of code a day. It's amazing. It's like anyone like Bill Gates was saying lines of code was a crappy metric back in the 80s. It's like no, no. It's like code is a liability. Uh, and the reason

code is a liability is because if you have an application that does something in 500 lines and something that does it in 5,000, the 500 line one is better because it'll have less defects. Um, and the other thing is a lot of compliance frameworks that people are signed up to, they do actually mandate that your code has to be reviewed before it goes into production and AI doesn't get a pass. Um, so next one, LLMs are high-V value targets. Secure them like highv value targets. Document what access they have. Apply the principle of lease privilege. Lock down who can get in and change things on them. Lock down when things change on them. Um, and because they're

high value targets, go through configuration hardening. actually read the manuals and see what are all the different things that we can do to make this tool more secure. Um, for example, clawed code has a like it use has a git agent and that is a high-risk agent because it has access to your codebase. But it also has a mechanism where you can allow list what specific commands that agent is allowed to execute but and that immediately turns it into a much lower risk agent. So yeah, um, training your developers is very important because AI is designed to feel really intuitive and easy to use, but it's actually really complicated. It just hides all the complexity from you.

Like people sneer about, oh prompt engineering courses, but getting meaning over to a AI coding model through prompts and context is a lot more complicated than just saying build me the app. Uh this is also really good because if you train people to use AI, you're teaching them all those things and they don't think of it as a magic person in a box anymore. So that covers for that risk. Um, lastly, use AI yourself. Like, I mean, I am I am not the world's biggest AI fan, but there are genuine valid uses for it in rounding the edges off in security and just making life easier for yourself. Like, I definitely fall more into the

workman archetype, but the workman still gets value out of AI. And the simple fact of the matter is at the moment AI is on sale at fire sale prices. It's never going to be as cheap as it is right now because all the companies are desperately trying to drive uptake and growth. So take advantage of that. Uh so those are our mitigations. That's everything we can do to make our companies more secure in their use of AI coding tools. And none of this is really amazing. Like none of this is absolute rocket science. and uh nothing is revolutionary which leads nicely into the conclusion. Okay. So as I said at the start I've been in the software industry for over a

quarter of a century at this point which is and in that time I've seen so many times when everything has changed everything changed when quality became a concept. Everything changed with the internet. Everything changed with the agile revolution and digital transformation. Everything changed with web 2.0 and APIs. Everything changed with the move to the cloud. And then everything changed again with serverless. But the end of the day, our job doesn't change. All the non-code bits of the SDLC are still there and still need to be done. Um, like every time, security are still fighting the same battles, just maybe on a slightly different map. So, as Douglas Adams said, don't panic. [snorts] Uh, and yeah, that's the talk. Um, we

have time for questions, but uh, if you liked listening to me talk, um, I do have a podcast. It is not about cyber security. It is about, uh, weird Irish history, but I enjoy doing it, so I wanted to give it a shout out. So, yes, uh, that's me. Any questions? [applause]

Mhm.

>> Mhm.

Yeah. Yeah. >> Yeah. Yeah. Never is way too much of an absolute. But I would say like at the moment these companies are giving us access to these models at unprofitable rates for them. So yeah, it'll it'll never be cheaper relative to the cost of the companies providing it. But yeah, good. Yeah. Yes.

Yeah, that is kind of a Yeah, it's a I think it's like it's a good recognition because like 90% of our code isn't our code these days. It's coming from packages. But I feel like if it's an open I mean we've seen just this week with npm and quicks account like packages are inherently unreliable. It is something we just have to recognize. We need to use them, but we need to have the right security around them and the right responses around them. So yeah, I think there is also the risk of upstream dependencies having the same problems, but we just have to keep an eye on them. Yeah. [laughter] Any Oh, yep. Mark

Yeah, I possibly should have put that in the scope, but my job is application security. That's kind of where my attention was focused with this. I definitely think yeah, red teamers are having a lot of fun figuring out different ways they can get around this. I mean, I mention like it it is good fun just keeping an eye on the news stories coming in of some research team has done something mad and they want to talk about it. So, like I mentioned that Tel Aviv hacking the smart home thing. So, it's good fun. Yes.

Uh, not that I'm sorry. The question was for the recording like the generative monoulture produced code. Is there any tools going out and scanning and checking for that? Uh, not that I'm aware of, but it would be naive to assume that it's not happening. Like I think especially like it is just such a juicy way of like looking at this and again it's a it's just it's a force multiplier for other vulnerabilities. So yeah and that ties in with the open source stuff as well because all the it's open source the source code is out there makes it but I think open source in general is always being scanned for vans anyway. So, oh, yes. A few more. Uh,

yes. >> Mhm.

Again, I think it's just part of treating it all as as a high value target in your infrastructure and just securing it and locking it down. Um, MCPs are interesting because they can modify the access and behavior that other agents have. So, it's almost sort of an admin level access that you have to be concerned about. So, yeah, but it's fun. Yes, [snorts]

>> it's it's it's one of those things where it gets very complicated, especially if you're in the enterprise, you generally have centralized secrets management anyway. So that's why it's not. So that's another reason why like vibe coding doesn't work in the enterprise because you need to plug into these centralized things. But I think the examples people saw was either they were giving it the access tokens in the context and it wasn't treating them as securely as they told it to or there have been cases where it has just grabbed them from the code context so someone else had embedded the secret in GitHub and they just reused it. Yeah.

Yep. Okay. So, generally uh and I know at Rapid 7 this is something we do is we'll have a lot of patents on our code on different mechanisms that we're using in there. Those are defensive patents. Those are to stop our competitors copying our code and using it in their things. And so the risk is that AI won't like it'll have code coming in as part of its context and it'll just use that and not realize that it's infringing on a patent. Um the other potential is that if it starts using copyrighted or trademarked content in there as well, which is more of a broadly general, I think we've all seen like the various court cases that

are going on around that. But for us specifically, it's mostly about patents.

One um last one I think. Yeah, last question.

Absolutely none. They're all retro. Yeah, they're all grabbed. Most of them came off Pinterest just searching around like 50s and 60s science fiction magazine and novel covers. So yeah. Uh that's like I said it was more fun to do it that way. So all righty. Well, thank you everyone. Just hand you back to Michelle.