Pentesting ChatOps

so thank you for coming uh what i'm going to present today is a uh updated version actually of a presentation that i also gave uh previously at b-sides in las vegas so i'm going to be talking today of course about penetration testing but also about chat ops who here knows what chat ops is raise your hand okay about half the people good well you'll know what it is uh by the time this presentation is finished so let's see how do i actually change the slide yeah oh that works great so uh we're gonna start with the question what is chat ops so chat ups is a concept that i well mostly gratuitous gratuitously stole from the company github so we all know

and love github of course they created a chatbot called hubot so who here has ever used hubot raise your hand okay smaller number of hands so hubot is an open source project and it's a chatbot it's basically very similar to like an old school irc bot you know so except for the fact that it tends to run in more modern uh chat rooms so i'm sure most of you here probably use slack or something similar raise your hand if you use something similar to slack okay so at radically open security we use a open source self-hosted version of slack called rocket chat who here has ever heard of rocket chat yeah all right i highly recommend it by

the way uh you know i i don't really trust slack you know it's probably only a matter of time before they're acquired by some google facebook whatever you know uh and of course there is no cloud there only is other people's computers so uh yeah so and especially when you're dealing with customers pen test data i can tell you that i don't trust that kind of stuff uh to slack's uh cloud and to be quite honest i don't think you should either but that being said um so we use uh rocket chat and uh this is basically a uh screenshot sorry it's a little bit fuzzy uh with the projector but uh this is a

screenshot of uh what our chat room looks like with uh with rocket chat and yeah can you guys make this a little sharper by any chance no it is what it is okay if you were able to read this text what you would see is that uh i typed in a command called raw spot pug me okay so raw spot is the name of our chat bot so raw spot for radically open security okay and pug me okay so this is an example of one command that we can do using our chat bot so these chat commands essentially are providing any number of different commands that you can carry out as a penetration tester pog me of course is

like one of the most useless commands that exists you type in rosbot pug me and you get back a pug yeah i mean not really useful but definitely you know for a sort of like cultural development of your company this kind of stuff is great at one point in time i think we installed the cards against humanity but anyway so uh yeah so but but the truth is there's a whole lot of far more useful commands that we also can do uh via the chat bot and essentially what you're doing with chat ups is you are actually turning your chat room into a kind of command line i mean kind of like a unix command line

so it's like making the chat your command and control center for your entire pen testing operations and that is a super powerful thing which then brings us into uh the actual description of pen testing chat ops and again my apologies for i have never seen the quality this horrible ever but i'll have to explain basically what's on here so if you were able to read this what you would see is you would see that i am compiling a pdf of a penetration test report so you know reporting it's a necessary evil developers hate it pen testers hate it pretty much everybody hates it and uh like every pen testing company in the universe we have developed a framework

to essentially automate all of the most boring bits away so uh it is worth mentioning that we have open sourced our uh entire framework for everything i'm describing today by the way and not only have we open sourced it but we have made it an o wasp project so if you google it uh or duckduckgo it or whatnot uh look for owasp pen text so this is the system uh that i'm talking about so but what we do is uh the the core of it really is a uh pen test documentation uh automation framework so we're taking xml uh templates and compiling them using xslt style sheets into pretty pdfs which then of course we can deliver to our

customers and hopefully they're gonna they're gonna love um the way that we do it is basically i use raw spot i use our chat bot to be able to uh actually execute the command to compile the report so if you were able to see this actually i think in this example it's a little fuzzy i can't read it either but it looks like we're actually compiling a quotation not a not a report but uh i believe that it says uh raw spot quick scope off dot uh well basically customer name and so what happens is first we fill in so if we're creating quotations for our customers we fill in an a4's worth of xml then we do this quick scope and then

that expands the a4's worth of xml into a longer xml file containing all the boilerplate you know and all the stuff that we would need the next command that i'm uh doing via the chat bot is i am uh saying raw spot build quote and again off dash customer name off yeah i apologize we're a dutch company the dutch word for uh quotation is oferta we never really got that uh out of uh out of our system but uh but basically uh that's sort of the the naming scheme we use internally for distinguishing uh quotations from uh pen tests pen test reports nonetheless this build command at that point is invoking the entire backend tool chain that then is taking that xml

file compiling it using the style sheets and then essentially spitting out a clickable link to a pdf file and again sorry it's not readable but what you can see is that blue line that's kind of fuzzy there that actually is a clickable url to the pdf file now the really nice thing is everybody in this chat room can essentially see you know that pdf link that was just uh clicked out that was just spit out by the chat bot so basically you know a command invoked by one member of our pen testing staff also can be viewed by everybody else who happens to be in that chat room so now is a good time to mention that at

radically open security we have a really unusual way of pen testing that we call peek over our shoulder so what this is uh i in the past have railed a little bit against the lack of openness and transparency in the pen testing process uh usually perpetrated by big commercial companies so i i tried to come up with a new workflow for pen testing that really tried to involve the customer more and to optimize for knowledge transfer so what we do at radically open security is we actually invite the customers to join us in the chat rooms so this is actually really quite kind of interesting because uh well we radically open security we happen to be a 100

online company okay so all of my staff works remotely i have a critical mass of staff in the netherlands of course because we started in amsterdam but i've also got a lot of german hackers for example ccc of course but we've also got penetration testers in australia we've also got pen testers in india we've also got pen testers in latin america i just hired recently some people in south africa so i mean when we say we are a global team we really are a global team however uh this actually has uh advantages and what this means is that all of the communication of all of my team members happens via this chat room so essentially the chat room is our office

but then what we do is we invite customers to join us in our chat rooms so what that means is it's basically inviting the customers into our office so then during the course of an engagement the customers can overhear every single conversation that our penetration testers are having throughout the course of an engagement needless to say they love this the other thing also is when we're making use of a chat ops commands via the chat bot they can also see this too so if one of my penetration testers uh uses the raw spot build command and we get a pdf link for up for a pen test report spit out the customer can immediately click on it

so this is really powerful so what it does is it really assists with making a very tight and fast feedback loop with your customers which of course has a number of uh really great advantages so uh yeah so pentex itself i mean you can basically find uh bits of it in uh get lab sorry in github uh also more recently we have also put a lot of our chatbot code into gitlab uh this has also been because we've been integrating it with gitlab ci because we're actually in the middle right now of doing a big devops refactoring on our code to try and make it easier to deploy with containerization and everything but there's documentation if you were

able to read this you would see that this is what the an example of what an xml template looks like it then compiles via the chat bot into something that looks like a bug standard pen test report pretty much but of course because this is open source you can basically take this adopt it yourself steal it that's what open source is for replace the logo uh replace the company name and you too can use it for generating your own uh pen test reports automatically um oh man this is hard to read so this looks like the same slide as before all right so we internally use uh git lab uh this so we kind of have a a bit of a trifecta

of software that we tend to use uh within uh radically open security so we have uh rocket chat uh we have gitlab and then we also have uh well basically uh all the bits uh with the uh with the chat bot we used to use uh can board for uh project management purposes but at a certain point actually phased it out because well people didn't like clicking sorry like cultural quirk of my company nobody wants to take their hands off the keyboard but you get that with hackers but in either case so we're doing the majority of that actually via the command line now as well but this is uh an example of how the uh structure of

the pen test report looks uh in git lab the the workflow that we have actually is that the pen testers document uh the findings but also the the notes in get lab so we use git lab issues so every time uh a penetration tester updates an issue in gitlab for example saying you know we just tried this and this and this or otherwise if they are pushing for example let's say they ran some scan and they're pushing the the file with the scan results into the repository every single time this get lab repository is updated the chat bot uh spits out a update in the chat room that says this pen tester just did this

at this time stamp click here for more information this is also really nice for the customers you know because what it is is actually without any extra work on the pen testers part we're actually giving the customer a kind of blow by blow update on exactly what our pen testers are doing at any moment in time and again like a well-informed customer tends to be a happy customer there's other really nice things by the way about including your customers in the process in this way shape and form you know for the purposes of uh crystal box pen testing this is super efficient because essentially a lot of people tend to ask the question yeah but isn't having the

customer in your chat room annoying like aren't they always asking questions and getting in your way and you know requiring too much explanation yeah i get this question a lot but the truth of the matter is it's actually annoying for us not having an active customer there and the reason why is because basically the customer is an oracle for us so anytime we have questions like for example what does this function do or you know why is this code path like this or you know we we just did something did it create a file on your server can you check or are you using a complex password here so you know we want to know if it's worth running the password

cracker on this you know having the customer there actually is really really nice you know other things like can you reboot the server or we're having problems getting this to compile and basically what we do is we invite the customer to invite actually as many developers sysadmins and devops folks basically anybody involved with the product or service that we're testing to actually join us in the chat room to sort of you know join the party for us it doesn't even matter so it for us it's kind of like you know the more the merrier there is opsec of course which i'll i'll get to a bit later but uh but nonetheless for us i mean

operationally speaking it's not more difficult for us having 10 people in the chat room than it is having two people another thing that it also enables us to do is uh allow the customers even sometimes to uh invite their vendors to join us surprisingly enough and this would sound like a really odd thing like why would you want the vendors in your chat room while you're doing a pen test but if it turns out actually that the vendor is involved with either hosting or building whatever the thing is that you're testing it actually is super useful because again it's like a second set of oracles that you have in the chat room on top of

this it actually helps to improve the relationship between the customer and the vendor strangely enough because they're kind of working together and collaborating on this and also from the point of view of a pen test company uh this is also beneficial because i have to say sometimes those those vendors who got invited to work uh with a peek over our shoulder actually became direct customers of our slater just because they uh saw what we were doing and like the workflow so much that they decided they wanted to try us so for their own pen test next time so there's a whole lot of win really in doing things this way um so what you have here is a it's a

screenshot from a github repository of another open source tool that we built that was called the passive scanning tool so this was a tool that we had built as an assignment for sidn which is the uh uh company that owns the dot nl uh tld and i'm allowed to talk about this because we the report is published but we wrote essentially a tool that took the output of uh showdown and uh scans.io and uh i think one other thing and basically uh bundled uh all of these uh well the output together and just uh spit it out in a concise way so we could do uh passive scanning so this tool also has been open sourced

uh but the interesting part about this is this tool then also can be integrated into the chatbot so this is actually really nice because you can actually start taking a lot of the different kinds of tools that you use frequently and then starting to slowly integrate that you know uh into this kind of like single uh command line so you know what other kinds of things actually can we do using the chatbot so another formula that we really really enjoy doing uh with our customers is what we call a red blue pen test so what we do is we take a group of developers so software developers system administrators and devops folks okay so let's say we have a dozen of

them what we do is we gamify their pen test and we actually split them into two teams so we'll have say either a red team and a blue team of say six and six people or we can also do it with a red team and a red team what then happens is these two uh teams compete on hacking their own stuff and what we have done is we have actually created our own scoreboard application so that at the end of every day of this kind of you know ctf competition where they're hacking their own products then you can see for example and sorry again this is fuzzy but right here i typed in a command

called good job blue and then raw spot then at that point says incremented blue 24 points and then that spits out some motivational image which of course geeks love but i mean it really allows us you know in the same integrated command line to be able to put actually all kinds of applications you know that we might need for all different kinds of uh assignments uh with our customers so this uh this all is uh is really super cool the nice thing by the way about the red blue pen tests is uh it really takes the developers out of their usual position of developing for you know a few days to a week and they're really looking at their own

stuff from the eyes of an attacker the number one comment that we get after doing red blue pen tests is and i quote i will never look at code the same way again and this is why we do it so what other kinds of things actually can we integrate into this single unified command line well it turns out there's a whole lot of different things i mean start by thinking about scanning tools like take nmap for example nmap uh is handy in the sense that uh it all it allows us to uh get output from the tool in xml format that actually is a really easy for easy one for us because if we're then using xml for our

pen test reports themselves then integrating bits of xml directly from our scanning tools actually helps us you know to make things quite a bit faster and quite a bit more efficient so this uh this much we already have integrated uh enter into uh raw spot other scanning tools uh that uh possibly could be uh integrated uh well still right now we're still using it by hand but we could is essentially things like w3af uh sql map essentially any kinds of scanning tools that you might run uh on some kind of uh infrastructure uh hydra so uh brute forcing yeah i mean this is another thing also that we can invoke uh via the command line and why

not uh reconnaissance so things like who is google pass the passive scanning tool i told you about before i mean we've also got duckduckgo implemented in that i think at one point we put uh can i google that for you in there anyway other things like hash cracking now this is super cool what we've done is we have actually built some rainbow tables and put the rainbow tables on our back-end server the nice thing is we can actually uh invoke a hashcracker that makes use of these rainbow tables and we can invoke this via the chat how cool is that but another really nice thing about this is because we're using a web front end for this

basically it means that i can use this full set of hacking tools that we have available essentially with any kind of device that has a web browser so what that means is if for example i'm sitting in a bus you know traveling to some god forsaken corner of the country that i live in um what i can do is i can actually take out my cell phone and i can actually launch you know i i can basically crack somebody's password using rainbow tables from my cell phone how cool is that you know and it actually really helps i mean for for being efficient because uh i mean of course there's off there has to be there's multiple layers of auth

you know and this is another thing that we that we take seriously and we have to but uh what it does mean is basically you know it makes it allows you to be really flexible with the form factors of the device that you're using another nice thing also is that it makes system administration of the tool set really really easy because basically there is a a single server that is running the necessary tools that we need uh for raw spot and then um let's say you you hire a new pen tester and you have to onboard this new staff member so with uh most uh companies uh you know on day one you know the first

thing is always you know issuing new laptops and you know trying to get all the accounts set up and things like that well for us it's actually far easier because all we really have to do is arrange uh that they have uh off you know basically to be able to get in and at that point everything is set up right so it actually makes onboarding new folks super easy um you know similarly also uh with customers uh again i i'm talking about auth quite a lot but but do keep in mind that because a lot of these commands are there and a lot of these commands are powerful we also uh have basically access control lists that control which

user accounts are actually able to do which commands because of course we don't want our customers running you know hash cracking tools you know vr command line and trust me like a lot of customers that uh we on board i mean the first thing that they do is they try and hack us or they try and doing doing something that they that they're not supposed to do which is great because if it's if they succeed i'd probably be happy to give them a discount on their pen test so that is to be encouraged but uh yeah so uh so that that is quite interesting another thing also is with spearfishing so this is something else that we can do

uh using chat ups that i think is really quite unique so if we have um a spear well not spear fishing but a general purpose uh fishing assignment we have actually built a toolkit that by the way we have also open sourced um and with a one particular chat uh chat bot command what we can do is we can actually take either uh you know like a mailing list like a text file or a web page we can actually scrape it and then we can instrument those links automatically to basically point to uh our web server and then uh from there we use uh the slack api to basically take the hits on our uh on our

web server and then inject that basically in to our chat so that the hubot then can basically say this email address just clicked on this pretext name at this time stamp and that is super cool because again we invite the customers to be in our chat room so what happens is we you know we can have like multiple security officers uh from a particular customer sitting in our chat room and they can sit there and watch while their own stuff is getting fished in real time you know and we can also do exactly the same with uh for example uh web forms so if we create let's say a fake uh i don't know windows 365 login and we're

basically just trying to collect the names and passwords that people are silly enough to fill in the security officers can actually sit there and watch while people are filling things in and sometimes actually this is really quite entertaining there was one particular time where we were doing a phishing assignment for a hosting provider and at a certain point uh one of the software developers from the hosting provider figured out that the domain name we were using wasn't exactly right so at a certain point he started playing with the forum so we we started like you know seeing in the chat room you know all the sort of strange things happening so at a certain point it's just sort of

like username you know nice try password whatever you know and and my personal favorite was he i think it was username woohoo password sql injection so you know this was actually really entertaining to watch i mean of course that begs the question was this guy in a sandbox environment while he was doing this but still uh you know at which point the security officers were just like man lunchtime but thanks for the good entertainment so yeah so this stuff is really cool so the nice thing again with chat ops is not just that you know it centralizes all the tools that you're using but it actually allows you to kind of innovate new experiences you know that your

customers can have which is i think pretty cool so uh you know also uh for internal stuff uh we also have hooks uh into our chat system so um you know like every company in the universe uh we have project management i mean you know by now we're actually quite a large uh company you know i've got about 40 staff members we've had about 80 customers uh i now is a good time to mention that we actually are a preferred supplier for uh google also for mozilla uh in the moss program if you guys know we do a lot of open source audits uh for that open tech fund but also you know banks insurance

companies hosting providers telcos etc um but that's not the point the point is uh because we have so much going on uh we have project management tools that we need um for managing that the problem with project management tools is if you expect the pen testers to actually do anything good luck with that i mean pen pentesters are lazy so if you basically say you know when you're done could you please you know move the you know uh item on the can board you know from this to done i mean if you ask your pen testers do it they just won't do it because that's pen testers um but instead what you can do is uh you

can try and get them to update stuff uh sort of by side effect so we use this particular chat command uh that is called um raw spot ship it so who here knows what the ship it squirrel is raise your hand one person okay two people okay so basically this is another one of these incredibly useless uh commands that uh hubot makes available you what you do is you put in the chat command uh ship it basically anywhere in your uh in your chat you know and then you submit it and then it spits a squirrel like a picture of a squirrel so it's kind of useless but uh but it's like really cute squirrels and sometimes

they're wearing armor and sometimes they're on boats and like you know it's anyway maybe sometimes they're in a tree but geeks like this you know because essentially if you're shipping say a pen test report or a quotation i mean it's nice to have some kind of little ritual for being able to celebrate victories right you know that's good for the morale of your company but we've rigged it as such that when people type in raw spot ship it and they get the squirrel it automatically takes that kind board item and it moves it to done see that's that's a way to get like lazy pen testers to do stuff so uh just to kind of make it fun

so you know so this is actually really nice uh we've also got uh our own uh support desk software that we've developed ourselves uh called uh get notes so uh again we we tend not to use drag and drop uh too much within the company just because our you know we've tried it trust me we've tried a lot of different open source software packages we throw a lot of things against the wall to see what sticks and anything where people have to take their hands off the keyboard almost never sticks so um basically uh we came up with our own support desk software that works entirely via the chat so what happens is uh we have magic email addresses uh that

tend to be you know things like you know project name slash well customer name slash project name at magical email address and then what happens is um that so the chat bot at that point spits out an announcement of when new emails arrive that contain that magical email address so what happens is the a copy of that email is uh automatically put into the gitlab repository that corresponds to that particular assignment and then hubot says you know an email with from this sender with this subject name uh just arrived at this timestamp please click here to see it this is actually really handy because sometimes we work with customers on multiple projects and their security officers want us to

keep them abreast of the correspondence that we're having with our project teams so what's really nice about this is as long as this magical email address is being cc'd or bc seed then again automatically we're going to get these announcements in the channel so as long as the security officers are um basically in in those channels you know that that are involved with their own company and to be honest we've also written scripts to automatically add you know these security officers to every new chat room that uh is created that is for their particular company then basically we can keep them completely abreast on the correspondence that's happening without a single bit of extra work this is really nice

so again this is sort of the power of what you can do uh with uh with chat ops we've also got another command that's called a raw spot charge so okay as a consultancy company the bane of like every security consultant is declaring their hours you know everybody hates this we've tried to make it as painless as possible so what we do is we created this command called raw spot charge so a particular pen tester can just say raw spot charge n you know for the number of hours and then a really brief one line description of what they just did then rossbot says you know thank you name of pentester you had just charged n

hours with this description you now have uh you know m minus n hours left left in this pen test that that is like you know this percent of the pen test and then like a progress bar you know showing like the percent that's left in the pen test now that is super handy because uh it actually allows us in the course of the pen test as people are declaring their hours to actually see how much time is remaining and of course one of the challenges of working on pen tests is of course scoping and while you're doing the work trying to stay within the scope and this is actually really nice and also from the the point of view with the

customers they also really like this way of doing it because it gives them transparency into seeing exactly where their money is going because you know in security consultancy time is money so if they can see exactly where the time is going then they know exactly what they're paying for so customers really really appreciate this kind of transparency another thing that's really nice about it is let's say you you scoped wrong you know that happens sometimes you know we've gotten better with scoping but we still make some errors sometimes what it means is if there is a scoping error the customers can see it coming from like two miles away so what that means is uh around the 50

percent uh point of the test you can then say well you know dear customer we've just reached the 50 point of this pen test you know we've done xyz that you know was in our quotation but you know there's still abc left left to do we don't think it's going to fit into the remaining time so dear customer can you please help us to prioritize so we can make sure that out of a and a b and c that you can help us select which of these things are most important uh you know and then the rest can get put in as future work now customers really like this because a it eliminates the uh surprise factor

you know and and one of the easiest ways to upset your customers is to have well unpleasant surprises you know and and really the more openness and transparency you have the more they see these kinds of unpleasant surprises coming so this is actually really uh practical and useful for uh expectation management uh they also it also gives the customer some knobs to turn you know on making some decisions about what to prioritize and this is another thing that tends to make customers really quite happy so you know other things that we have uh integrated into our chat system are things like role-based access control so uh we do uh have uh particular uh teams and particular roles so our

company is managed uh in a decentralized fashion uh we use a system that is somewhat similar to uh hellocracy i don't know if any of you guys are familiar with holacracy but it's a kind of decentralized management methodology that tends to work with roles and circles uh anyway so uh the circles that we would have you know in the holocratic view of decentralized management translate into roles in our back system so you know that's basically how it's handled so and of course we create uh tiger teams for individual assignments uh but there's also larger more perpetual teams uh that are you know for example like the pentester team and the uh project management team and the itn

infrastructure team so and this all translates into our back and this also is how we stop you know customers from being able to run scans against their competitors or something using our infra so uh other things error logs so what's really nice is we can actually we've got a chat room that uh tails the error logs we've also got a chat another chat room that tails uh debug logs on things and what this means is that it allows people on our infra team to be able to debug stuff without needing shell on the server yay you know so this kind of thing is also uh really quite handy um of course with all these uh chat

commands uh you well it's a lot to keep straight in your head so we also built a sort of a help menu system uh so that our pen testers if they want to know what commands are available you know a little bit like unix man pages uh to sort of help them uh through you know knowing what commands uh you do what and what the syntax is so we've also built a system for that you know and then ultimately sort of i think the sky is the limit with what you can do with these chat bots you know there is research at some point that i would be sort of curious in doing and and there are questions like for example

you know could you use could you create ai chat bots you know and could these ai chat bots perhaps help with things like uh you know answering frequently asked questions for people or you know helping uh you know like if you're on boarding new staff members you know that kind of thing uh possibly so you know this is the kind of thing i mean i don't want to pump a whole lot of money into this research uh you know because it's not operational uh of course from a company perspective but i think uh this is this kind of stuff is very interesting like projects like master theses projects for ai students and things like that so uh so that's pretty cool

um yeah and other things like uh can we use uh ai chatbots for for example satisfaction surveys and of course satisfaction surveys in general are really incredibly annoying you know i tend to get them you know via email and i almost always ignore them but the nice thing is uh using chat ops you can sort of mix the routine and and kind of like methodologicalness you know of having a bot with the actual human touch of having humans there so for example if you were to use a chatbot to say hey customer you know are you satisfied with you know with whatever service you just got you know and then basically one through five or something like that if it turns

out then that the customer gave you a one because they're really not that happy instead of you know the chatbot or i'm you know just basically keeping that and they're only interfacing with a machine you're going to have live humans in the channel so if you see a response that surprises you and that requires more clarification then an actual human can sit there and ask questions you know as opposed to their interfacing with a automated system you know you perhaps could also use a similar system for things like also uh you know satisfaction surveys for the staff that you work with like are you happy you know i mean this kind of stuff is also uh useful to be fair we haven't

implemented this yet i mean this is future work but i mean this is uh what i see as being some of the promise of uh ai chatbots i'm sure there's more that you can do with this you know the sky's the limit but this is research and i think the more of us adopt this workflow you know the more innovate innovations we make and hopefully we can then open source all the innovations that we create so it turns out actually that uh our workflow is quite innovative uh actually my company is also quite innovative we have a uh a bit of a strange um business model we we're also a social enterprise which is worth saying and i set up

radically open security also as a non-profit company this requires more explanation and to be honest that's a whole different talk that i'm not going to give today but uh we have won a whole bunch of awards actually for our company and for the uh way that we work including this uh open and transparent workflow so uh the dutch chamber of commerce called radically open security the 50th most innovative sme in the netherlands um you know also uh we uh yeah have been involved with other our pen testers one of them won the pony award at black hat uh for his uh research um also we created a uh net aid kit project which was basically a non-profit

uh completely open source tour and open vpn router for journalists and activists that we built together with free press unlimited so that won some awards from the isoc and from the internet freedom festival um and also uh i won an award from uh cio magazine uh calling me the most innovative i.t leader in the netherlands so anyway it's not about me but it's i think more rather about the ideas so you know i i believe that uh what we're doing right now with uh with chat ups i mean this is really kind of a new innovation with penetration testing it's also i believe an innovation that uh can also assist you guys and help you

all raise the bar to do more open and transparent pen testing which i think ultimately will help you guys uh to also improve the relations with your own customers and also allow you to have better pen testing services i think also this chat ops is not only useful for uh pen testing i know actually hosting providers believe it or not who have done trials with our uh our pentax system and also had really good feedback uh from their customers uh you know for using hosting uh services uh with chat ops and believe it or not we even got approached by uh one of the dutch ministries talking about if they can use chat ups in a pilot for open government

so i think that there's actually a whole lot of really cool and different stuff that we can do with chat ops uh you know and it's just up to us to sort of explore the space and uh and try it anyway uh i am out of time so i think i'm going to wrap it up here uh so uh i'm happy to take any questions at this point thank you very very much

um it sounds like the chatbot can do a lot of things and so it must have permissions all over your system apart from your access control this you mentioned is there anything else that you have to do to make this safe yes good question so indeed uh it is allowing the uh chatbot to do a whole lot of stuff and uh indeed we have been thinking uh quite critically about the actual back-end art architecture of this system so i i mentioned before that we're in the middle right now of this big uh devops refactoring and a whole lot of that involves containerization so in the refactoring that we're doing right now we're busy actually uh trying

to create an individualized container basically for each uh customer and ultimately an individualized container for each assignment so basically if all of this is then running in its own virtual machine uh basically if there is a breach of some kind then it would actually only uh give access then to the to the vm that uh you know this particular assignment is uh is running in so yes we've thought of this and this is basically another way that we're trying to isolate things um there are still of course ways that uh you would need to uh federate this stuff i mean especially authentication wise so i mean this is definitely a work in progress on our

side but this is definitely why we're going the devops route you know with a lot of this stuff because really what we want is we want to uh create these containers and then when an assignment is over we want to destroy them again you know and this is sort of another opsec thing what i really want i'm not going to say that it's happening as much as i want it but this is the direction i want to move in is that once we deliver an assignment for a customer what i would want to do is we've we already have a raw spot command that can take the uh rocket chat transcript and then make a text file of that and then

check that into the corresponding git lab repository what i then want to do is we can basically get a tarball of that git lab repository and then what i can do is i can give that to the customer and then i can say dear customer i don't want this on my server take this tarball and if you keep it you're responsible for it you know i don't want it you know but but you take this tarball and if you ever want us to do a retest then you can basically give us that tar ball back later and then we can reconstitute the container basically using the tarball and then at that point then we have what

we need to be able to uh offer you uh re-tests and things like that in the future so that's basically another um sort of op tech thing that we're doing we also of course do regular regular pen testing on our own infrastructure and i can't tell you i'm probably not even allowed to tell you how many zero days we found in rocket chat and the associated software what i can say though is actually they've been quite responsive and quick about uh fixing them once we've reported them and in fact they became so enthusiastic with our reporting them that rocket chat eventually just hired us as their security vendor so we could just pen test for them directly

so that's pretty cool uh also um you know is the security of this perfect no of course not i mean but you know we're dealing with complex software here i mean also web-based software so you know we're like any other company in the universe we too of course are also making uh trade-offs uh opsec wise uh you know it's a constant process we're constantly trying to refactor and raise the bar but uh you know like any other system i mean i i obviously can't pretend that it's perfect either because well no software is so thank you any more questions oh that was very interesting thank you uh i have two questions actually so first is

i can see the value of having people from the uh company that's being contested and from having a lot of those people to answer your question and stuff like this uh but you know like the answering and checking things for you obviously takes time i'm not saying this is a wasted time but uh how do you deal with that uh do you do is there in a contract uh some kind of appointment of people who are supposed to help you or you are just hoping that people will be helpful yeah good good question so the question is do all customers actually want to take advantage of the peak over our shoulder given that it might be time consuming so

the answer is no not all customers want it and they're not required to do it either so basically it is a free additional service that we provide in the course of our normal pen testing if they want to take advantage of it we're super appreciative and we definitely encourage it if they say we don't care we're not interested we just want a report in a black box you can completely you know take this off our hands we don't want to see it we can do that for them too i mean it's not our preferred way of working but some customers are like that uh so uh yeah we can't accommodate that if need be so

okay uh so another question uh one of the biggest problem in my opinion uh with with chat ops is uh signal to noise ratio uh so uh i think that uh what you are doing being open is very helpful uh but not necessarily at the time it's happening uh then don't you have problems with this i mean what i mean is it's nice to go back in the history and see what was being done during the pentest but i don't need to know that right now i mean maybe sometimes i do but uh usually i don't even know that pentester at this particular moment is doing something so uh don't you have problems with with

having too much uh notifications and stuff yeah good question so uh i mean most slack-like systems do have a problem with what i would call chat dust you know i mean you basically uh you know if it's a very chatty room and there's people in there that are quite chatty then indeed you know the room can't fill up with some useless uh stuff um to be quite honest in most of our actual penetration test channels it tends to not be to not be that chatty uh just because we have uh other channels in our company that are chattier i mean like we've got for example ross off topic you know that's like our water cooler so if people want

to post idiotic memes that's where they go you know i mean we've also got ross ethics you know which is another channel for discussing ethical concerns around certain kinds of jobs and you know uh anyway there's a lot of different uh also ross pentesters which is kind of like the area just for the pen testers to ask each other questions about assignments and things like that so typically uh you know we do have an explosion of chat rooms that for sure but uh the only people that are actually in that chat room are the people that need to be in the chat room so the majority of the people actually who are on our system don't even get

access to that much which means that there isn't quite so much to read which means that they're not the ones typically who have the the chat dust problems however uh we are really careful to make it clear to people that what you write in the chat is not documentation because it just scrolls away and you're never going to find it again so anything that you would might need to reference later or that actually you know needs to be documented that needs to be done in gitlab you know because anything in rocket chat that's just completely worthless it's going to be gone like you know after a day or two um in terms of the customers uh you know

uh are they overwhelmed you know by all of this not typically i mean the thing is uh we can actually do like with chat uh with slack-like systems notifications uh for individual people so for example i can do you know um at sign uh username and if that person is not actively logged into the chat at that moment uh they'll actually receive an email saying that uh you know somebody on the system sent you a message please log in to uh to see what it is um this actually then notifies people of actually when they need to be logged in which means that they actually don't need to be logged in i mean the nice

thing about a slack-like environment is that it keeps a backlog of the chat so basically what you have is uh synchronous and asynchronous communication at the same time which basically means that people can do other things that they need to do they can like go to meetings or they can sleep or they can you know whatever just be distracted busy with other things and because this backlog is maintained when they log in again that that enables them to catch up on the backlog if they want to you know and then uh you know be able to continue basically where they left off this is also by the way really useful for coordinating people across multiple time zones because i mentioned

also previously that we have people in australia but we also have people in europe and we've you know other places and of course there's the whole uh time zone thing you know and sometimes i do have you know one european pen tester partnered with one australian pen tester and then the customer is is well probably in europe but uh you know but sometimes they're in the us i mean you know sometimes we have that um in that particular case then also the asynchronous uh nature of uh the chat also really works in our favor uh with that one so nobody has to be logged in all the time and nobody misses anything you know hypothetically speaking if the

chat backlog were that overwhelming people can just say i'm not going to read it just let you know give me a too long didn't read you know but uh that's basically how we handle things and for the most part i think it's pretty doable anyway thank you i think i'm out of town so thank you again [Applause]