Beyond Vibe Coding: Building Reliable AI AppSec Tools

[Music] Hey everyone, Steve here, volunteer director at large with the Vancouver Island Security Research Society. We're gearing up for our annual Bsides Vancouver Island Security Conference happening October 3rd at the Victoria Conference Center in beautiful Victoria, BC. Bides VI is a community-driven grassroots cyber security event focused on accessibility, learning, and collaboration. and it's open to everyone passionate about security. Today I'm excited to introduce one of our amazing speakers, Emily Choy Green. Emily will be delivering a talk titled Beyond Vibe Coding: Building Reliable AI AppSacc Tools. Emily, thank you for being here. Would you please introduce yourself to the community and tell us what inspired you to explore this topic? >> Yeah, definitely. Great to be here. I'm

Emily. I'm the CEO and co-founder of Clearly AI. We automate security and privacy reviews for regulated industries. and my background is as a security engineer for AI systems such as Amazon Alexa. What really inspired us to investigate this area is the same sentiment that we got over and over again from customers and other connections in industry that it's really easy to prototype solutions with AI and really hard to productionize them. And a lot of the gaps that happen when teams try to move from prototype to production is around hallucinations, reliability, and normalized output. >> Thanks, Emily. Your session tackles some fascinating and critical challenges around the use of AI in application security. One of the key focuses is on

building AI tools that are not only powerful, but also reliable and trustworthy. AI can do amazing things, but it can also hallucinate or produce conflicting results, which is a huge risk in security. To help our audience get a flavor of your talk, I've got a question for you. Your talk mentions techniques for preventing AI hallucinations and handling conflicting documentation. Can you walk us through a practical example of how security teams can address these issues? >> Definitely. So, one of the main reasons that hallucinations happen is because large language models are working from their working memory instead of from the information and context that you give it. So, basically when language models are trained, there's three layers of

training. Layer one is how the language actually works. Um, you know, sentence structure vocabulary grammar things like that. Layer two is building in universal knowledge. things like, you know, what is the capital of the state of Washington, what is the largest country in the world, all of those pieces to help it really understand fundamental and foundational knowledge about the world. And then the third piece is task completion, being able to do different types of tasks. And often what we're asking large models to do is task completion. But if we give it insufficient information, it'll pull from its working memory in order to attempt to complete the task. Because models really like to to do the thing

you ask it to do. And so they will do whatever they can in order to complete the task, including making up information from their memory. So there's two major ways that you can engineer around these systems. Number one is to actually have that engineering, meaning don't expose the exact large language model outputs to your um kind of end audience. have a layer that does output normalization, puts it into very defined frameworks and have the ability to give the permission to the large language model to be wrong and to say I don't know the answer and sometimes as you might have experienced being in a classroom with the Socratic method if someone calls on you you feel

like you have to answer something and it's scary to say I don't know. So a big piece of this is to give large language models the ability and capability to say I don't know and to reward them for that. The last piece is that conflicting documentation may have different levels of trust. So when we kind of put in design documents versus code, we actually prompt the large language model to say, "Hey, these are both about the same system." And your job is to highlight those conflicts and don't take one as fact and the other not or add, you know, additional context to say what more trusted versus untrusted sources are um in order to help shape your

answer. >> Emily, thank you so very much for sharing your insights and taking the time to chat with us today. That was very uh insightful. For everyone watching, don't miss hearing from Emily and many other incredible speakers at bides Vancouver Island 2025. Tickets are available now at bidesvi.com. Be sure to follow us on social media. Links are on our website and join the conversation by using the hashtag besides vi. I love AI. We can't wait to welcome you at the Victoria Conference Center on October 3rd. See you there.