Sajeeb Lohani - Efficient Defence Turbocharging Security Workflows

Um I come from web security background. So I made the web security course at University of Melbourne. So come from security research, pentesting, red teaming, etc., etc. And I used to be a software engineer kind of like Aaron too. And uh I do a lot of bounty hunting and CV hunting. So uh why does all that matter? Um I run a team called the Shield Team. Um you can basically think of the Shield team as a team of engineers. Um they're trained sec uh software engineers that we turn into security engineers typically. Um why we actually go about doing this, it's cost-ffective and uh the gist of it is that it fosters innovation. It basically pushes people

to go and say cool um you can do x y and zed with your time. Maybe you know look at 100 sock alerts or whatever but you can also get a bot to go and do that for you which is typically more efficient. Um or you know it depends on how things go. Uh my team deals with everything technical security. So pretty much from anyone clicking on a link uh in a you know fishing email or something like that uh and trying to teach people not to do that all the way down to finding an ODA in need. Um so it's a lot of capacity and trying to do that between nine people is really really difficult.

So we turned our specialtity into automations. Now, um, what's quite fun is we actually turned this entire thing into the shield team purely because of the fact that a shield is typically defensive. So, you can use it to block attacks, but then again, you can also whack someone over the head with it and it hurts. So, it's quite effective offensively, too. And, uh, that's kind of the entire concept of how we believe purple teaming is. So, the story behind this entire team is actually kind of weird. Um my uh funny enough I was the first security hire in Bug Crowd post their mass exodus and um I basically started off and I was supposed to be there as a pen tester uh

for their applications and turns out I got heavily catfished um I came in and I was actually supposed to be doing a purple teaming role now but oh well you start off and try to figure it out. I was in the infrastructure team so cloud infra and uh if you know me at least from the 2019 phase uh my cloud infra is not very good. So it was a lot of learning a lot of figuring things out but then I um you know buckran ended up hiring a CISO. He came to Melbourne we sat down for a bunch of beers and got talking and then he realized that there was a really weird part of me which not

many people actually cared about which was the fact that I had been an engineer. So he turned around and said I'll fund you. let's get you two engineers and go show me some magic. And that's basically what we ended up doing. So the gist of it was that tools cost a lot of money and expertise costs money. However, human time is a lot easier to justify uh or a lot actually more lenient than um trying to go and purchase a vendor for $100,000. So what we ended up deciding was putting all of our money or a lot of our money into human hours. Now, I'm not saying we're going to sit down and rebuild something like Crowd Strike cuz they do

it a hell of a lot better than we ever will. However, there are things that we can do to make CrowdStrike more efficient and things like that. So, we basically provide our humans, our engineers to go do something more interesting and um try to build things in a very quick, agile, fastpaced way. So, uh we decided lambdas are cheap and that should be our basis. And um now, funnily enough, years after doing this, we decide AI is cheap and it's also a great way of doing things. So, we're going to touch on that. The real quick thing, the way we actually like to do things, um just to lay it out in case anyone ever wants to. Um we terraform

literally everything. So, it's all infrastructure as code. You can pick one up, spin it up within seconds, and you're good to go. Uh everything is a CI/CD pipeline. So, we use GitHub. So, GitHub actions, it's all, you know, literally an action. you uh pretty much merge it in a master and off it goes. Um quality checks are done as gates. So hypothetically you don't meet a gate, it kills everything and you die. Uh and all lambdas are created with specs or specifications stating that x y and zed must be achieved and you must actually properly fill out um all the different areas to be able to state what a lambda is supposed to do. So the real process

flow is you would start out with um you know the IM permissions. Now one big thing you are a security team. So make sure you do follow the principle of lease privilege. Preach you know do as you preach. Uh allow list your resources appropriately. Then move into building ECR images based off golden images. So it's literally just pumping whatever requirements you need on top of that. Um, we find typically lambdas very effective because a lambda can run for a second or two and it's really cheap anduling them for every minute is nice and easy and then spin up your lambda and associate the image digest. Everyone wins. Now, what exactly does my team do with

these types of things? Um, so we actually built our own custom anomaly detection cuz we hated vendors anomaly detection. Um, we've gone through and built logging sources because turns out some vendors charge you about $30,000 just to get seam logs and that's expensive. You can do that for like two bucks with a lambda. So, we did that instead. Um, alerting on specific types of tools, perfectly fine. Um, but we can do a little bit better. Um, we automate on specific types of behavior. So, things like snitching on people when they're doing something weird. Perfect example is when someone goes and posts something they shouldn't on Slack or something like that. We have bots that go through and find them, alert the

team, and then the team goes and says, "Hey, wake up. Don't do this." Um, and then the other part is syncing between tools. Some tools just don't sync properly. Now, I know there's a lot of no code solutions that can go and do this 100%. However, when you run as lean as we do, we don't necessarily want to use them in all capacities. So, my team runs about 80 different lambdas currently. Um, but the crazy thing is we don't actually have to maintain much of it because dependabot and things like that just kick in and help us out. Does anyone want to guess how much this actually cost us to run? Any guesses? >> 200 bucks. How much is that?

>> Less than one engineer. >> Less than one engineer, definitely. So, 200 bucks a month is the lowest we have so far. Anyone else any guesses? >> 10 bucks. >> 10 bucks. Oo, aggressive. It's less. Uh so um our lambda bill is currently so in August 2025 was $135 USD, sorry. Um but uh September it was $140. So we went up a little bit and that's cuz we deployed new lambda. Um but yeah, that's literally I'm not exaggerating. You can check our bill. That's actually how much it is. Now this does not include things like RDS. Obviously those are much more expensive. However, I'm talking about just the lambda. Um, so the good, the bad, the ugly. The

good is it's cheap and efficient. It's crazy crazy fast. Um, it's interesting for engineers so they don't get bored. We have crazy good retention rates in our team because we allow people to go and innovate, come up with new ideas and we actually have these six week kind of sprint cycles for a specific type of task which we call the innovation tasks where they basically turn around, we give a single engineer um 6 weeks, first two weeks is determine the task they want to do. They turn around and find the thing they hate the most in their job and then come and tell me. Uh next two weeks is turn around and build a plan. So build a plan for how to

automate it and the final two weeks is actually building it out and testing it. Um so it's quite literally just that. How do you make your job less [ __ ] Uh it's allow people to do it and uh it's fast-paced. It doesn't require a huge amount of approvals or anything. So we can spin up a lambda in less than a day kind of thing. Um all you need is approval from a peer and you're good to go. So nice and easy. Um, now the uglies and the bads are pretty, you know, significant though. It's hard to maintain if you do not build it properly. So you have to come in with a plan. You have to come in with

a spec and you need to know what you're doing. Um, it is quite dependent upon people. So hypothetically, if someone leaves um, and they were the only person who knew about it, it's a single point of failure, which is bad. Um, and the easiest way to fix that document. Um, but I mean all of us know how bad that is. And uh if processes aren't laid out, there's always going to be knowledge gaps. There's going to be one person on the team who knows everything and then you know no one knows anything about anything else. So is what it is though. So an example of the type of thing that we've actually gone and built um is the

security approval bot. Now, each of these uh different um bots that I'm going to show you today are kind of explained through the process flow are actually AI oriented and at the end I'm going to also tell you about our AI bill which is kind of fun. Um so the first AI bot is a security approval bot. Now this obviously screams terrible idea to everyone here because why on earth would you trust an LLM with it? And the truth is you shouldn't. Um, but the trick, there's a bit of a trick to it, and this is where you have to kind of change your way of thinking. Typically, security turns around and says no to literally

everything. However, there are certain efficiencies that you can always gain by utilizing tools to your advantage. So, um, you can have something like a lambda polling for new tickets. It'll then go and identify possible um, you know, uh, it'll start processing things, grab the different roles in this case. So we had um fields in our forms stating you know which uh role they actually want to take um and it will turn around and grab the authenticated user's email address. So it's something they can't spoof. The only thing they can really spoof is a predefined role. So it's from a Dropbox unless they can pop the actual form. There's nothing much they can really do there. So the bot will then go through

um previous tickets and documentation and then give some references to the engineer to make the decision. Now note that the bot is not providing any access and that's basically how we kind of limit the amount of risk that we have. The bot doesn't know how to provide access. It doesn't have that access itself. All it has is read only rights to tickets. Oh actually write to but so then after that the human goes in understands exactly what is needed and then says cool based off X Y and Zed this team is allowed to have AWS access or they're not. And then we basically change uh the entire flow. What this provides us is basically a standpoint

and you know the equivalent of Jira's related tickets just done in a way that it remembers what things are and doesn't keep [ __ ] itself up. So then um if you ever have to do vendor risk assessments that are terrible terrible and disgusting thing um typically they take like 4 hours if you're doing a proper technical assessment and digging into every bit of documentation. So our team turned around and tried to build a bot. Um and uh it was also one of the things that my team hated doing the most. Um like think about a company um you know with probably 100 vendors or so. You have to do an assessment for every single one of them. It's hell. Um

now what we ended up doing was uh basically we figured out that there was a consistent format that a GRC team love to give us all the details. So we grabbed that, pulled that into a lambda. um basically ran all of that in um and allowed the bot to actually reach the internet. So do web searches and things like that. It would go through analyze the vendor's documentation and then um with our master prompt break down things into the exact way that we would normally do an analysis and then it put in a private comment into our ticket. Um what does that do? It basically gives the engineer a little bit of a leg up

and decreases the amount of time. instead of 4 hours, it takes about half an hour to an hour because they don't have to go and look for the information all over the internet anymore. So saving time, but once again just efficiency, not necessarily taking over everything. Now the next one that we have, and this was one where it actually does take over everything a little bit more, but um we have a lambda that goes through and uh so we created a learning system for an automatic ask security a question type thing. Um, and all it does is it basically has a lambda that goes through all the tickets that don't already have embeddings. This is where

we have an RDS in the back. Um, which is basically a Postgress server. All it'll do is go through all the tickets, grab the embeddings based off the API, and pump it into PG vector, which is a Postgress database. Then after that all it's going to do is go through grab tickets that are already don't have a response and very very simply just create embeddings from the original question that they have pump that into the database do a string match or embedding match more rather and then try to get an approximate confidence. Um what we did was you actually just built in variables for thresholds and then kept tuning the thresholds until we found it working right. And for us it

was typically 70% confidence and 30% um kind of uh threshold on um matching which kind of you know is it'll depend on your actual uh company. And if you do want to know how we actually did the tokenizing um we used Tik Token and that was a 400 um car 400 token window with a 50 kind of rolling uh overlap um which worked well for us. once again need to be tuned and figured out. Um but yeah, the gist of it is that after that we pump everything into an LLM, let that go through and make the decisions again and then come back to the engineer and give a list of different references alongside their actual uh confidence scores and

figure everything out from there. So it decreases your work once again and um yeah, it kind of works out pretty well. So um this is actually our AI bill which I think is pretty cool. uh 27 cents uh for the uh you know last month and this month was 30 cents uh which you can tell is a hell of a lot less than engineers. So trying to decrease the amount of time we spend on engineers and instead let those engineers go and innovate and do something useful with their time um seemed to be a hell of a lot of a better idea for us. The gist of it here is that engineers are awesome. They build weird and

wonderful things and uh they can be cost effective if you allow them to be. So give him a chance please. Thank you

>> questions. >> Yep. >> Um so the you're using for your is that a custom or like >> No. Um, we actually just use GPT5 right now. Nice and easy.

>> How does the AI be so cheap in my personal experience? For example, um, using um, like a subscription with cursor for example, 30 bucks or whatever it is a month. Uh and then they'll add if you go over that they'll like it's very expensive. It's best to go over how you get so cheap internally. >> Uh so it's API based everything's done through APIs. You utilize it. So typically what we're trying to do is do pre-processing on everything. So squashing the data as much as possible using embeddings where possible cuz embeddings are cheaper than actual um text and that squishes down the amount significantly. Um, and we also do a lot of pre-processing on the data to get it

into the database and that is done through embeddings. So, um, it's kind of like small little wins that you can take. And if you can do everything through the embeddings database, um, then you're actually spending more money on RDS than AI. And RDS money is just consistent anyway because it's a monthly bill. So, we spend probably about 20 bucks a month on RDS. So, >> yeah.

Thank you. >> Awesome. First great

prestigious size. Thank you. >> Thank you.