Secure AI Deployments: Building Compliant LLM Environments

hi I'm Tanya Jena from sura community and I am here with colum at bsides Vancouver Island by talking him and he is doing some cool stuff and I feel like if I even try to describe what you're doing I'm gonna not do a great job so tell me the cool thing you're building yeah my name's Callum I'm building Trent Trent is a secure environment where companies can deploy LMS in a compliant and safe way we work with companies of all sizes and scales but typically hardened or regulated organizations that require hardened software and we give them the AI policy to implement in their companies and understand what kind of data you should be able to upload to

certain systems and we also give them a private LM instance that's a green field for them to to use in a safe way so when you say an a safe way if I go use chat GPT and put lots of sensitive private data in there is that bad Kell terrible yeah don't do that so it depends you know there's Nuance right there there's a few different tears that open the eye has with chat the free and the personal tiers don't do that definitely don't team and Enterprise is a little bit more subtle depending on you know the threat model that your organization has maybe it's acceptable to upload certain kinds of information into those Pro

subscriptions the difference that they're not training necessarily public models where that ends up going and being leaked outside your organization however they still do do collect information for abuse monitoring and so if you have a company policy where certain levels of information just can't leave your network that's still not acceptable because ultimately open the eye employees or you know anthropic employees whatever company could still get access to that information and we all know that our providers never ever ever look at our data that's not a thing no never been an issue so what about Shadow AI is the idea that ad Dev goes and just adds an AI to your app that they're building without telling you

without permissions probably without licenses and then you might get sued that sounds really bad and general I just view anything where you're sending this information of any kind to NM in the same way you have to ask same fundamental questions you have to look into the terms of service of these companies and really understand what their data practices are regardless of whether that's through Shadow AI implementation like you're describing or whether it's through public product that's easily monitored like Chachi I feel like if you're going to help companies create policy first of all that would help them not have a ton of Shadow AI you could help them use llms more effectively we do a

lot of work to help companies do use case Discovery it tends to start with software developers that's a really common initial use case where it's like hey there are very clear ways that this can help with developers workflows and then a lot of times we work with data teams to help them you know maybe they ad hoc reports and they need to write some SQL but they're not actually familiar uh we help them do those sorts of things too so let's say I'm a software developer and I'm like o l lims those seem exciting and I want to start learning and I don't want to make bad habits do you have any suggestions of maybe where I could start learning or

maybe some things that I should not do yeah so I think it's important to be able to use these systems get familiar with them and learn I fully recommend people try tools like chat I don't think that it's bad innately and so I I recommend you try and experiment it with personal projects and learn that way I think that as a way of getting started with these things it's totally viable even with the knowledge they're going to use your data but you know Google's also monitoring your searches and we kind of accept that there's a level of privacy being broken with tools but sometimes that's worth the trade-off I still think it's worth experimenting with these

tools and highly recommend you do so thank you so much col this is really exciting and I'm very excited about what you're building people should go check it out [Music]