I Let My Pi5 Hack: Lessons Learned in Building a 0$ AI Pentesting Agent - Nithin Ravi

I know we've just had lunch and I know you're all probably sleepy. I thought I'll start with a joke. Uh does anybody have a joke to get started with or Okay, if nobody has I'll just I'll just start. So one year ago I was sort of chasing a gold mine only to realize that I was playing with fire myself. Uh you'll sort of understand what the joke's about when I get to the topic itself. So um around last year about this time I I was sort of experimenting using a CPU only cluster to use local LLMs and then to sort of experiment with AI and all of that the fire analogies because at the end of the experiment and all of that

stuff my Raspberry Pi almost sort of burnt and then it was throwing up smoke and everything but if the joke didn't really land well I'll talk about myself. Um I am Nitan I have a double masters. I have been hacking for about 5 years in the space of web and API hacking. I've talked at a couple of conferences like besides defest etc. I hold a couple of certifications as well. But then the most interesting thing about me is I do both chess and photography. If you want to play chess after this event just reach out to me. We can definitely have a good game. I am also a very big reader. I am reading this book called

Wolf Tutm. Um, if anybody's interested in [snorts] drama, especially around Chinese myth, Chinese uh, fiction and all of that stuff, I definitely recommend you giving this book a start. Um, all right. Now, let's get directly into the topic. So, again, as I was talking about about roughly a year ago, I was sort of experimenting on how I can use um, Raspberry Pi as a VPS for myself. And then when we get into the topic of VPS again um we also have to talk about why do we need a VPS for pentesting or bug bounty hunting or anything that we do right and this journey started long back when I first started into this space of bug bounty

hunting um and like [clears throat] 5 years back almost every creator that I was following like VXS rat and then stoke and um Namcon sorry Hamsek and couple of other people that I was following and Reddit and But in all of these places, they were all talking about, oh, buy a VPS, you need a VPS to hug. Um, if you don't have a VPS, you don't really have an automation workflow. What are you going to do? Etc., etc. Things like that. And to tackle this issue, I mean, I I obviously fell into the marketing. Um, I did get a digital ocean. I had a digital ocean VPS. I was also having a um line VPS and

then I was spinning up I was spinning up my automation workflow and all of that and these different VPS. And of course you know with any cloud provider you have lots of cost. Um so I wanted to sort of cut this cost and also uh with Raspberry Pi I also fell into the marketing again. People are saying you can do a lot of cool things with Raspberry Pi which is very true but then only to use it for a [snorts] computer is not probably the wise choice but anyway it worked for me and this is the architecture that I used to fix it. It's very very simple. At the start I had a Raspberry Pi 5 cluster. I mean I didn't

really buy it. I I got it as a gift. So I think it makes sense because I didn't really spend anything. Um and then once you have a PI cluster, you would also need to sort of wire your architecture and then um use all of your applications and in all of your devices because of course what if you're traveling and then you have a new target and then you want to hack on them. You know, it's all about speed at the end of the day. Um so you need another um again you have your applications that's rely I mean that's inside your server and then you'd have to use all of these applications in your

tablet or in your secondary laptop or in your mobile phone etc. Um there are a couple of ways you can do this. You can use again VPNs like uh wire guard. The only reason that I sort of again use tunnels like wire guard is because they're peer-to-peer and again you would also need something that's open source. So say at the end of the day if you want to tweak fork play with it um you have that option. Um so the first thing that I did was again I I had a pi and then I installed wild card on top of it and then I ran into this issue and the issue was CGET. So I use I don't know if I can

name the ISP but anyway I have an ISP and then they have this issue called a CGA which basically lets me not access my wire guard tunnel outside of the internet right so I need some sort of fix to it and then the fix was inserting steroids on top of wire guard which is basically tail scale so tail scale really it basically works in the same principle mechanism of wire guard it's it's actually wire guard with a couple of additional powers so if you have to mitigate all of these issues that's related to N or in networking etc. Your best bet would be to use tail scale again open source free um very very easy

to use uh has a really neat UI and all of that stuff so you can manage all of your devices pretty easily. So Tails scale is the um the the tunnel that I use to secure all of my applications from my PI cluster to all of my other devices. Um so you have your server ready, you have your uh tunnel ready and the next thing is you need to make your domain cool, right? It's your I mean it's your local domain but then you say you don't want to remember numbers and then port numbers and etc. You you can bookmark your pages but then it's pretty cool to use DNS rewrites um and then yeah you get ad blocking with adgot as

well. So that's the green color symbol. Um you have DNS rewrite. So for example, say if I'm using a platform like N for example, I can just go use my laptop today do N don't hack my N this is Nathan's computer.com I get that N10 service. So it's very handy it works and then uh for any other self-hosting uh service or application that you're using it's very wise to containerize it. So you have Docker and then you also have my main automation sort of tool is which is in and I know we're all security folks we probably would have seen lot of innateish hacks but uh since we are again I'm hoping that since it's

all inside my tail net it's very safe but if anybody can prove me wrong please do and want to secure my um applications again. Cool. So that's the architecture we have at Pi. We have all the applications running inside Pi. We have a tunnel um using which I can access all of my applications in all devices. We have DNS reites. We have Docker. We have Net and so everything is fixed. So we have sort of defeated the uh the infrastructure problem. Uh we don't need a VPS anymore. We can anyway use all of these applications do all of my long running scans and um use any application that I want inside my compute. So next

problem is sort of understanding and to be very fair I I was doing pretty great with just using automation even if it was just chron jobs or um or or just scripts that that was running scans 24 hours I was doing fairly good but then there was this entire hype and fad around AI everybody has to use AI to make your things very very cool and then I obviously fell into it and then um when I fell into it I was sort of doing this thing which everybody did I especially when you're into the bug bounty space, you sort of want your your tool to just go and hack into whatever target you have. You just say hack this

thing for me and then you sort of expect um you know you you do this thing, you wake up next morning and then you want a million [snorts] dollars in a bank account. That's the sort of um expectation that everybody has but it's that's not how it works, right? Um there are things you can actually do without AI. And it's very very important to understand what you can do without AI because then you know also what you should be doing with AI and the things that you can do without AI is you can have you can definitely look for vulnerabilities uh that require um OB callbacks right for example if you're looking at class of vulnerabilities like

SSRF or anything that requires DNS re DNS rights or uh say even for bugs like blind access everything that you need to do is just spin up a call back in your um local compute and then uh have something hit it uh pray for something to hit it. Uh so that's one thing that you don't really need an AI for. And then you also have things where you can monitor files like JS files. I love monitoring JS files. I think there are JS is such a gold mine especially when you're into bug bounty hunting or hacking web applications, right? you can read code and then um you can do this weird hash comparisons and then see what

endpoints have changed what files have changed if there are new parameters etc and it's very very helpful again these are things you probably don't need AI or I I thought I we don't need AI initially uh I'll prove you wrong at the end and then even for class of vulnerabilities like idors um you only need to change numbers etc you don't need to analyze things right so you again don't really need AI if you're very very clever with automation And you can pull off all of these vulnerabilities without using any AI at all. So then the question is where do we need to use AI and where do we need to use automation? So all of the

points in red color that you see it's like red pill blue but then I sort of missed the blue pill thingy. I just use green. Um very sorry on that. But then yeah for all the red pill thingies these are things you need AI for. Again a red very very cool. So you need whenever you need lot of things to be triaged for example you I I feel that's where we need AI uh it's very very clever when it comes to triaging things say for example you have multiple endpoints which you've extracted even from the same JS example right um and then say for example if you have to prioritize different endpoints um and then if you have to like sort

analyze them AI is very very good at that that sort of stuff so anything which involves analyzing a massive chunk of data um AI is very very good that's not something that I believe or that's not something that I have been good at in terms of writing rules and then to sort of prioritize rank. Um so I can just ask K to do that for me. It's very very helpful that way. And again that brings to the big concept of contextual reasoning. Um contextual reasoning basically means you have huge huge massive pool of data especially if you're just running wrong scans and then you have you collect massive amount of data and then you want this data to be

interpreted and to be reasoned with. Say if you want to talk about this data to some platform uh it almost feels like I have a coach or a mentor that I can just sort of communicate back and forth and ask uh things about that's where AI is very very good at it's not very good at when you ask it to tool um we'll talk about that why it's also not a model issue and why it's probably hardness issue but then when you want to prioritize speed and then if you have like a very very defined structure again as we talked about previously if you have these chains of vulnerabilities for which you can absolutely clash with only

automation, you would probably be better off without using AI. You definitely don't need AI and especially if your bugs need no logic. If you're just going to do something like match and replace or um pretty much things like that, you don't really need AI um in these things. Um so now let's talk about the AI experiments that I did. So now I have a [snorts] compute I mean I have the infra and I can use that infra on all of these devices. Um and then I almost fell into the AI hub uh AI hype. I bought these uh I mean I sort of subscribed to all of these different um models and then using etc etc. So again, uh the thing with AI

is again it's it's very I mean it's it's pretty expensive if you're using a really good model and then if you're using especially I don't know what myths cost is going to be if it's going to be available for the public but then if it's not again your regular cloud code max is like what 20 $200 per month. Um so I don't know I didn't really feel like buying into it. So I I sort of was chasing the local LLM dream and then um so the first thing that I obviously did was research on how we can use local LMS and then how we can orchestrate and all of that stuff right so that's where your

comes in place um so with you can like pull all of these like local models and then use them within your compute but again uh as everybody knows with AI you can only do so much with a compute only. you definitely need GPU, right? Um that's why everybody's selling GPUs. But then um again I that's that's the point I was chasing this gold mine which I know is going to fail but I anyway wanted to chase it and then see what's going to happen. And then I sort of started experimenting with all of these different open source models like deepse we have uh metlama and then we have quen 3 [snorts] by Alibaba and then we have

Microsoft's 53 and all of these are like really local models which has u very very small parameters. So you can just pull them off and then run them in a pie which fairly work for some cases but then um it didn't really work for a lot. So now you have this local you know it's open source so no cost you've pulled them uh you just need to use them. So next you have to sort of understand how to use them and to sort of understand how to use them you also have to have some sort of base to evaluate them and sort of understand what sort of models you have to use at what instances right

um so when it comes to evaluation this is what I found um you need three things first is you need AI as we've talked about previously you need models that can reason and you need models that can craft things for you right so when it comes to reasoning I found that the deep sea car 1.5 dist works really good at reasoning uh and it even in my 16GB Raspberry Pi compute it gives me roughly about 32 40 tokens per second which is very very good when we're talking about a local model and when we're talking about something that needs crafting or scripting etc then I've um I've genuinely found quen 2.5 coder to be very very good um yeah quen 3 is also a

very good model but then um I've only played with it so much deepsee has worked for me a lot better. So now we know how different models are playing around. This is sort of experiment that I did um by putting my pie into different things that it wasn't ready to be put at first place. So now you have your models, you have your orchestrator, then you sort of need to understand how I can use my automation and where you can use my AI and sort of how to stitch them together to get something very very effective right? >> [snorts] >> So yeah, I was using nit fairly because it has a very very neat UI and of course

we are security people so we need really good UI right I suppose. Okay, cool. Probably not. But then it has really good um you know when you use things like NAN you have this visual workflows. Of course, somebody can ask me, you know, Nathan, you you anyway have chron jobs. Why not use chron jobs? You know, you can why do you want to do all of this stuff? It's very visually pleasing, right? Honestly, and especially if you want to use different um NA particularly, they have really good um um methods, modes, I don't know what it's called, but then you can sort of pull these different things and then if you want to have a web hook, for

example, um you can use them. Um again, u you can see what's failing, what's not failing. So NA10 has worked particularly well for me in that side of things. So first you have to sort of stitch a tool output and then you would need a workflow and the workflow that I had particularly with nit was um say you've pulled a massive chunk of JavaScript files again at first um you have it again depends upon your target and everything but then say if your file is very very huge you sort of need to understand what sort of requests or what sort of files are worth investigating at first place. So this is where I this is

like a very very simple triage or very very simple reasoning that I needed for uh so for tasks like these I found particularly that you can just use 1.5 or uh 1.7 or 3B parameter models and especially things like it again depends u I found quen to be very very successful in um understanding and investigating requests with with massive size and just saying yes or no um so once when you have these requests or these files and then you sort of have understood with AI what sort of files are very interesting to look at. You also have to sort of suggest attack vectors and this is where you need models with with coot uh which is train

of thought and uh these are again models you uh you can again use models like 53 or quen uh depends um or deepc carbon deep carbon has this ct model by by by its very nature and it's worked very good for me um historically and say once you've found a particular like you analyze this mass of files and then you found a very very um interesting request and you want to create some sort of P on top of it. So that's when your um models which are very good at writing scripts uh come into place and then that's where um I've particularly found again quen 2.5 to be very very good at finally to write reports you you can use any model

for that matter it doesn't really matter a lot but I've particularly enjoyed working with um llama 3.2 But again, these are all things that you probably would not be doing with just a Raspberry Pi compute. Uh again, because if you only do something with Raspberry Pi computer, you'll also find yourself not being able to sleep well. And the reason why I couldn't sleep well is because I am a student. Back when I was experimenting, at least I was a student. Um I live in a single room and then I have my Pi cluster right next to my bed. It was just spinning so badly every day. I just couldn't sleep. You know, it was the noise. But yeah, if you take all the

fluff out, it was not just noise, but then it just didn't work. It didn't perform like I thought it would. Um, it spun a lot of tokens, but then none of these really made sense. Um, so what I thought was, okay, so I've experimented my fascia with all of these local models. Um, yeah, experimented failed successfully. So now let's see what I can do next. So I started abusing all of the free tiles, right? So I built this automation thingy again so I can pull all of these uh free APIs from different uh inferences like rock or open router. They always have these free um APIs. They sell that from for promotional purposes. So I said okay cool. What if I

can fetch all of these APIs for my um for my own self and then use them just for experimenting, right? Uh and [snorts] that worked pretty well. I still use some APIs from grock till date and and uh yeah very very effectively abusing all of these free tires. Open code is a very very different scenario because they don't I I I think they don't sell APIs uh they are a harness themselves. So you can use them as an agent within. So say if you have analyzed all of these files and then you have to talk with these files or create something with these files, then you can use tools like open code and then use

their um freely available um models. Right? So okay so we have experimented with all of this. Uh now we have models, we know what a model is. We know what a harness is. Um, now we sort of have to understand what's the difference between using an open- source model versus using a process model like OPUS 4.7 or open uh or or uh or uh Gemini or GPD 5.5. Right? So these are purely my opinions. Um if anybody has any differences that's that's completely fine. But then I genuinely believe that open-source models are um not bad at tool calling. It's a very very hardness specific issue. Uh that's the thing with open source models when everybody talks about

these open source models like deepse uh deepsec v4 pro for example right I have had very very good hits with deepse v4 pro um and whenever I talk about this with someone um in the security space they always say it's very very bad at tool calling it doesn't do the things that it wants that I ask it to and then it sort of ends all of my um workflow in 2 hours whereas I asked it to run for 12 hours sort of things like that and [snorts] mostly all of these issues them because not of the model by itself. Again, it's just a huge text file. So the the the the sort of problem that

you're facing is a very very harness specific problem. Um and I after analyzing like all of these different requests what I found is uh if you don't post train your model uh be it whatever model on whatever harness if it's just directly calling its API I've seen um especially all of these open source models like five deepc um or quen etc whenever you have like a huge schema and then say when you have optional fields on it they always send a null you don't I mean uh if your automation involves relying on all of these is parsing all of these schemas properly and then you don't want words like null uh you sort of have to train it to not use words

like null and then especially when you're using arguments um and then when you're dealing with JSON strings and then arrays properly I've sort of seen that it no matter even if your input is an array it sort of puts it into a JSON string and things like that so you it only comes from practice I believe you sort of have to talk to your model you sort of also I believe I I genuinely believe that the more you appreciate your model for the the output that you give it, it sort of behaves itself differently. I don't know uh of late I've also seen my deepseek uh sort of have opinions on different things which I found pretty scary but then that's

that's the way it's behaving now. So I always tell it oh yeah wow you're doing good uh keep doing things like that. I know Sam Alman has said not to waste tokens on all of these things but I've genuinely found it to work. Um so that's the thing with all of these models and harnesses. Once you have a really good harness, when once when you sort of understand uh what things are working versus what things are not not working, um you would also sort of improve your tool calling in a in a in a much better sense. And um yeah, so okay, cool. Nathan, we've talked about all of this. What's the output? What have you found

out? Um right, so this is uh something that I did pretty recently. I was using and sort of experimenting with DeepS V4. Um, and then I submitted this book like last week and then I also got rewarded for it. So I'm assuming it works. Uh, and I also paid roughly about $1. It's still 0.99. So I think the my title to this talk still uh still is justified. Uh, but yeah, so I did this thing. I I have all of my automation scripts in in in N10 and then I sort of analyzed and then I had this um huge file and then I threw this file directly into Deepsec Vro and then uh this is a forked um

harness of the DeepSc DUI that I've used and uh with all of this I asked it I I I just baked a couple of skills and then I asked it to um um analyze this this chunk of JS uh files that I had and then it not only found a anonymous session plus Nar. It also made sure that to that if I specifically chain both of these vulnerabilities, you get a atto as well, which is something that it did. Um yeah, so so that that's I I found it very very impressive, especially if you face it properly, if you know how to use different sub aents and then if you sort of um um use all of these sub agents

with the context that they need. Again, it it all stems from um post- training a model over and over again. But once you have structured these very beautifully, you obviously with very very less um um um cost, you you get very very good outputs. Um, also, uh, one of the tips that that I would probably throw is if you're if you're chaining vulnerabilities like this, it always makes sense to have a verification sub agent at the very end which does which has about no context to any of the agents that you've had before. So, it has to sort of operate on its own and then when when you ask it to verify all of the vulnerabilities, you also defeat

u you also sort of circumvent all of the u true positives that you get. So here are my takeaways from or the things that I've learned from all of these AI experiments. Firstly, [clears throat] that's there's almost no gold mine. Uh so if you're definitely chasing some sort of gold mine uh and then saying, "Yeah, I'll do all of this by just paying nothing." You're probably not going to do get anything effective. And you definitely don't need to AI everything. A lot of my bugs before the age or hype of AI has sort of uh come without using AI itself. and definitely go abuse all of the free tests and also no I just thought I'll add another one

uh if you're in offsec and if you don't use deepse definitely not promoting deepse in any form but then if you don't or if you haven't been experimenting with it you are definitely missing out a lot [gasps] perfect thank you for coming to my talk questions