Some Thoughts On Penetration Test Reports

uh hi everyone my name is Hans Martin MCH I work roughly about 20 years in the cyber security field most of the time I was doing penetration tests and offensive work during that time I wrote a lot of penetration test reports and I saw even more when I was on customer s sometimes they showed me stuff from other people and I'm always quite curious about that uh I'm the CEO of mck labs we are our no fluff security outfit uh we basically do one thing only and that's penetration tests and doing technical Security reviews so we basically create a lot of uh penetration test reports by our own I'm we are really really small so I'm not in the position of the CEO

that can just say hey please write the report I have to do this by my own so I can uh tell you from a lot of my own field experience uh um the agenda for today uh I'm not going to bother you with basic how should the report look like stuff uh I think you can also you already know that if not there is plenty of YouTube tutorials that show you the basics how how to structure general report instead I want just to focus on two main topics the first one is how you can do that in less time because I think that's everyone uh that are there that's the one thing that most people are

interested in and also how to REM uh improve report quality in general because I saw some key aspects in some reports from other people where I thought either why the heck are they doing that or oh this is really awesome I need to include that into my own reports okay so the task about uh reducing report time um I had some um personal struggles more or less this year and due to that I needed to become more efficient okay I need to be uh I have to do more stuff in less time so I looked at where can I actually optimize my work and you if you are in this field you quickly realize that uh penetration test reporting is

actually one of your main tasks in my case for example we roughly uh calculate about 20 to 30% of the re of the project time to actually reporting so if you think about a five days project I will spend one day just doing the report um the fact that uh I want to optimize this is also uh related to the fact that no one likes uh actually doing that stuff I no exception here so having some uh some ways to do this more efficiently is quite important to me I have to point out that some of the stuff that I will show to you are just ideas that I came up with I did not Implement

them on my own workflow because of various reasons because for example my tool set is something different but I think it might still have something for you that you can take with your work okay so let's talk about finding templates um they are the best way to reduce your time that you spend on reporting in ideal case you can basically create 80% of the actual report by just dropping in some finding templates and adjusting them to you need and they also have a different uh purpose they help you to quickly sync the reporting style between the different team members if you have multiple uh people that doing penetration tests it's quite important that even if you have a person that is

not as good at writing texts uh creates a more or less similar report than one that has a lot of experience and doing that with basically providing fighting templates is really really important I think we can all agree at that but this brings us to the following question which you all had if you ever worked uh in this field why don't we have a template for finding X okay um you then start to look in your older reports Maybe you know that you had a similar issue like two months ago or something like that and you then copi that over you forgot to remove the customer name and stuff like this this happens a lot happens to me happens to

others also the reason why you don't have finding templates was actually discovered by this man and this was discovered 50 years ago or 60 70 years ago actually uh this man is called seril Parkinson and he basically did a research on uh why bureaucracy is getting bigger and bigger and he came up with an ironic article and uh defined several laws and the most important one is works expans so as it as to fill the time available for its completion what does this mean in practice I know that I need to create uh slides for this presentation since several months when did I finish them this morning okay and the same applies to uh you when

you're doing reports the problem here is that the penetration test report normally create uh defines the end product of your testing scope so you have five days to do uh five days for the entire report or for the entire project the report will be done on day five it will not be done on day four if so then is more or less an exception but normally you always will be swu by Parkinson's law even if you create the final report on that stage it's not the real final report because it needs to pass QA first this normally takes up two or 3 days until some of your colleagues has the time so once you actually have the final

report uh done you're already working on a different project okay during that time you don't have any slots available for creating templates out of the report you already with your head into a different project and you are already working there no time to look at the stuff that you did one week ago here is how they solve uh Parkinson's law in uh in science fiction uh uh you have Scotty that comes up and says hey I need five days and then Kirk says hey you have only four days so he's basically shortening the timeline that is available for doing the project and um then Scotty says okay and he do does it in four days okay this is science

fiction this does not work in real life uh in real life you probably say okay I will apply shorter timelines but after or sh of deadlines about after two or 3 weeks you're falling back into your own H your old habits and you will create the report we will have to report done on your last day so I thought okay what can we actually do about it and I always think like okay I cannot be the first person who actually has this problem and how do other people solve this and one group that has a similar problems are actually developers they need to create unit tests uh so they also don't like to do that and what they came up with an

approach is what they call uh test rthm development so the idea is that you're creating the uh the unit test before and then you do the actual code that should be tested and you can apply apply a similar approach to uh to your reporting style the basic that basically Nars down to the following sentence you don't have a template for this and there is a non-zero chance that you will see this in a different environment at a different time so create the template first and use that uh to actually include that in your report this might be a little bit uh of bumper because you for example might say hey I don't like to my English is not that good but we

have ai for that I will cover that in a few minutes and uh this also has the nice little approach that you're actually going to do um your first QA uh already while you are including the the template that you just wrote in your actual report so you're already doing some quality control there and this actually works really really fine um the problem that you might have is that uh you think that changing habits is hard and this is definitely true if you're talking to developers about test driven development they say hey this is this is awesome hard this is we we basically tried it and but we uh we basically skipped it because it was too difficult

for us you are in actually in a better position uh or you actually in a similar position at first uh for example you have people in your team that are not good technical writers if you say to them hey please create now templates they will not create good temp uh technical templates okay so you end up with shitty templates so you don't want that however you're actually in a better approach or you have in a better position because not everyone in your team actually needs to follow this approach this is totally sufficient if you have two or three people that are really good at writing reports give them some extra time that they follow this

approach it's also uh important that you don't have to do this on every report if you are already in a hurry and you have a delay then maybe skip that just write it down but still give it a try if you can to uh I also want to point out that in Germany we often have this All or Nothing approach but even if you basically start with that and doing that during the actual test and you stop in the middle of doing that you have uh for example two or three templates two or three templates are better than none just give it a try believe me I basically applied this uh since the last four months and it really changed the

way I do reporting at the beginning it takes a a little bit longer but after a while you will really see that you got templates for nearly everything I now have a first world problem that I have too many templates or that I have two big templates um so in at M Labs we normally had the case that okay we have a template and we add everything into it uh for example like cross scripting and the idea was then that we actually remove the stuff that you don't need when you're writing the report removing is always easier than writing something so we had a one big template for example cross scripting and uh added that uh

everything together and then we had to cut out the stuff that we don't need it turns out that this is not the best approach what I highly recommend to you is that you uh create different templates based on the following two uh things does the recommendation differ or is the root cause actually a different one as an example we have one uh one template for reflected stored in dbased Cross scripting because the mitigation is always the same we totally have a different mitigation when it comes to uploading files okay there you cannot say hey please apply input validation you can provide more specific uh tasks on that so there is actually the really really benefit of having an own template

on that unfortunately most of the penetration testing reporting tools they don't provide you the possibility of um having um small templates that you can integrate into other ones but I will complain about pentest reporting Tools in a few minutes so let's talk about AI because you cannot talk about optimization today without talking about AI uh sometimes people are downgrading AI as a fancy autocomplete uh but in your case this actually what you want okay I want something that writes helps me to write stuff and my Approach here is if it saves me 10 minutes and I do it does this every time then it's a definitely a Time saer if you're thinking about AI in the

penetration testing world you have basically one problem or two the first one is if you want to have a non-generic text you actually need to provide some context okay so if you have a cross scripting vulnerability you need to provide like which function was affected you need to provide which parameter was affected maybe provide a proof of concept and then the your llm will actually create some valuable text that you could actually use the problem that you normally have is that you don't want your your data or your customer data uh to train a data model in California or something like that so there's no actual option for that at the moment the best way to solve this of course will be

private llms uh so it would be the perfect execute to buy me a new Macbook Pro and uh start doing that however I don't have the time to do that uh so this was not a real option for me at the moment I still believe that private llms are the the big way to go and in 5 years we will all use them but until then I need to or I use public llms like jet gbt to actually improve my templates I have uh friends in the advertising industry and they tell me hey the first stuff that basically got rid out by uh by um jgpt or by llms was uh textas because they don't need them

anymore they can create the basic text with an LM now and translators and and if you not heard it in my beautiful accent I'm not a native speaker so my English is not that good uh however I use this actual uh prompt to improve my texts when I do uh my report templates okay I don't do it for the reports I do it for the templates and this works quite quite well uh you don't have to copy it I will share the slides afterwards so no but you can of course if you want to um if you play around with that for a while and you play with jet gbt you end up with creating your

own gpts uh this is super easy and super efficient what I basically did I created some really really good templates five or six of them I crafted them made sure that the reporting style uh shows what I actually want and then I uploaded it to CET gbt okay no customer data was in there so no problem from that side um then I did some tweaking and after more or less than half an hour 20 minutes to half an hour I actually had a really really nice uh tool that uh allows me to create scaf holes for my actual template vorts how does this or for my findings how does this look like here as an

example uh let's say I give me a template for some XML uh signature validation and it basically creates a template that looks quite similar to what I uh want to use in my actual report it's it's not that I want to do this that I will use this right away I will do the validation I will copy it over in my templating Resort store um but it already helps me a lot even if I just have something to write on that I can then use and to change it and to tweak it to fit the actual need that I want uh but this is already quite there it even has to-dos where I should place

my screenshots and stuff like that so this works really really well it's always comes up with hallucinations regarding the recommendations there's always an oos cheat sheet uh template there that does not exist because my five templates always included an over cheap sheet so that's why you need to check on that but again the idea is here I don't need now 10 minutes or 15 minutes or 20 minutes to create the report template I only need 10 report uh 10 minutes especially if you have people in your uh in your team that often have a writing plade having them something to start to work with this is a really real timesaver let's talk about optimizing QA

and there's one big problem that I call uh quality assurance fatigue so that people actually getting bored and getting uh they don't want to do that the first thing that becomes important here is that you actually think about your pentesting reporting process as a build chain okay you're not building software you're building uh a report but the the idea in in the process itself is really really similar and if you think about what you would recommend to your customers when it comes to build chains you apply hey do automated tests and go shift left so try to identify the issue quite early ideally before the test is actually uh sent over so what you can basically do

you can create unit tests for your actual p uh for your reports here are some ideas the basic ones a lot of uh are already implemented maybe in your tools so like finding and searching for placeholders but you can do a lot of uh of other stuff too that's actually depending a bit that actually depends a little bit on the tool that you are using but this is actually some uh interesting contents to actually catch some copy and paste errors that can easily come up uh into your testing the goal here is that the pentester finds this before he basically sends it over to QA so shifting left finding this quite early but you can do this at every

step uh when you're basically shifting your uh your pentest reports from going from QA into the final release do a final test for example that there are no empty pages and stuff like that always helps what would be nice and I did not see this in any uh of the templ of the penetration testing reporting tools that I saw so far is actually a template diffing so you normally have a diffing feature when it comes from the changes that are done by the QA person to the actual report but what you want is actually uh having a similar feature a step earlier so that you can see as the QA person okay which stuff is actually

part of the template that was used and which part was actually or which parts were actually added by the penetration tester themselves the reason is that you don't have or it it reduces the time that you uh when you're doing your QA spent on stuff that is basically already part of the template that has been tested or read several times before uh so doing that can definitely reduce the time here unfortunately as I already told you this is basically a mockup that I did in word um but because I don't see or I don't I'm not aware of any tool that actually can do that it also requires a linking between the template and the version uh the

template version that you used and the actual finding but this actually is doable technically what you also want to do is um what I call reducing click fatigue so if you ever did uh uh spell check or something like that in Microsoft Word you often end up like doing this and then you are hitting some uh you did some spell check that basically just got clicked over um if you want to reduce that what you basically want to do is reducing the total number of clicks so that the entire process uh uh is less stressful and there are some stuff that you can actually do the first thing that I really recommend to you is that you

maintain a central spell check dictionary and actually update that okay a lot of people actually have something like this but they not really maintaining it so my Microsoft Word has no clue what open API is or it does not know what um sliver or something like that is uh or so adding these words there and actually Distributing it over the to all your colleagues will actually reduce the number of clicks that they need to do when doing the uh the test or the QA and also try to uh remove stuff that does not needs to be spell checked for example we in our report template we have a code section where we have code Snippets and you don't need to spell

check them and you can actually disable this in Microsoft Word okay so just reducing the number of time uh helps a lot on doing that now I will rant about reporting tools uh if you are looking up reporting tools on the internet I assume that this is basically the second most uh development project that every pentester does besides maybe writing a C2 Tool uh so you find a lot of different tools on the internet uh that um some of them are open source some of them are maintained some of them are not maintained uh but you find a lot of them okay you also find a lot of different uh in-house solutions that basically were built up

um by the the company or by the team and if you if I ask my colleagues or my my friends hey what are the key features you come up with stuff like template re uh management keeping report lay out in sync and stuff like that however pentest reports in my opinion fail a major task that is actually creating a nice uh environment where you actually want to write reports okay this for example is Pond do and if you tell me that I should spend eight hours a day in such a small window to create five pages of reporting for just one vulnerability I'm going to say no just ask yourself if you would like to

write a blog post in one of these tools and the answer will definitely no because they are not made for this why because the tools are actually created by penetration test report or by penetration testers themselves they are normally not considered a group that is a valid uh or that is a good uix designer okay so they they're not good at creating good user interfaces and unfortunately now you know that you will complain about this every time you use your report tool I'm going to be honest on that if you want to have an idea what's how this actually could look like so what you could get please take a look at obsidian obsidian is a notes taking

app uh and we use that to switch from wik JS internally to uh to obsidian uh for our internal knowledge base and we have to say hey the the number of entries and the quality of entries in our knowledge base increased a lot and if you ask people why they say hey because this is actually a good environment to write texts I love to writing texts there it's creating it's directly creating a new uh fancy or it's directly looking like something and people just love it okay if you want other examples have a look at different writer apps uh they basically have a lot of approaches that you actually want in your reporting task if you have a tool that is using

latex to actually create the PDF please stop uh there's a new uh shining star and it's called types and it got everything that you want and you hate about latex is basically removed it can can it is a 15 15 megabyte rust binary it works in your browser you can have in uh in your browser previews of the actual slides you don't you have meaningful error messages everything you hate about uh latex is basically gone if you want to see this this is basically inv visual code I can actually edit it here and the code basically changes over here so this works quite really well okay because we don't have much time we are in a hurry uh CVS s ratings

or severity ratings CVSs is the base okay that's the standard period you can not argue with that um we at mbet more or less used CVSs until more or less uh before we stopped because if we don't liked it then we said okay we down or upgrade it uh so we came up with our own uh little uh idea here but we still provide the CVSs base score if you ever uh come with the idea of creating your own custom scoring system uh you basically just creating a different ver version of the impact and likelihood grid you basically just have final grids and as if as soon as you apply final grids you have more complex stuff no scoring

system is uh perfect there will always be edge cases where you say okay this does not really match up if you think about how your customers actually treat that they actually don't uh think in numbers here they think about severity groups like is a low vulnerability is a high vulnerability is a critical vulnerability the reason is that severity groups often Define the response deadlines okay if they have a higher severity group they need to fix this in two weeks if they have a medium one they have two months and they don't care if this is a 5.6 or a 5.9 it's not important to them if they actually care about numbers they need deterministic numbers and then

you need to stay with CVSs because you want to have comparable numbers that you can compare with your partners so no one has actually deal uh time to deal with your nice little uh custom scoring system so just it um here is how Blaze security did it they did it right they have uh a vulnerability classification that fits on a half of a page does not have any technical descriptions and does not have any numbers that's really really nice to understand uh we basically use that so last but not least let's rant about charts penetration testers love charts because we assume that the management loves charts okay that's my quote what I mean is this uh you always have your

nice little uh vulnerability summary where you create your crafts and everything is fine uh the problem here is that you're basically comparing groups of data sets and but this does not any uh so your different vulnerability groups but this does not provide any value to the management no one actually says to you oh I really like how the ratio between my medium vulnerabilities and High vulnerabilities developed okay that's never coming up so what the management actually wants they want to know if there if they need to do something okay it's something on fire and your visualization should actually reflect that half s Consulting they basically showed me how to do that they added traffic likes in their executive

summary okay so you know oh I need to worry about something like that or you don't okay everything is fine uh pent reports did not show any serious vulnerabilities and everyone understands that okay I don't need to have a a pie chart that uh shows me the relationship that doesn't brought me anything summary penetration test reports provide a lot of optimation potential I basically only get started I have so many stuff I could fill up one and a half hours just doing that uh I highly recommend you to switch to template R reporting this will not change your life but the but the number of time or the time that you spend on reporting uh don't bother your customers

with things that they don't need we are no fluff uh uh company this is basically our philosophy um but this is really really important actually and if you don't know what your customers need then talk to them okay talk to them how they actually like the report what they took out of that maybe get some feedback on that will definitely help to improve that's all uh I will make the slides available uh hopefully tomorrow um if you have any questions feel free to ask [Applause] can you turn my microphone on okay check check okay thank you so much Hans Martin okay so I'm not going to ask if you have any questions I would like some

volunteers to ask me or ask hands some questions so anything that you could help the rest of the audience learn more give me a question anyone oh we've got one right here thanks for the presentation uh you said you use obsidian internally right yeah um why obsidian I mean there is also other not taking apps right is there anything specific I think obsidian was the one with the graph mapping or something yeah is there basically I I saw another block post where they said hey we switch to the obsidian and I tried it out and really turned out to work quite well so I basically copy copied that over I think uh it was from

trusted SEC there's a good article about from trusted SE how they use obsidian also how they made the structure and that's I basically copied that one to one made some small adjustments after that but yeah it turned out to be quite well okay thank you time for one more yeah thank you for the presentation uh you talk about AI that you use AI uh to write report not to support you uh when writing report I only use it at the moment I only use it to uh create my templates not the actual report but this also helps me to reduce my time I definitely will go for private llm here uh so I assume again in 5 years a lot of

the penetration test report will actually be created by an llm and you're just navigating the llm to uh adjusting the findings here but I think there's a lot of potential here are you about to do CH uh to not yet no time for that oh for one more okay question the next time yeah thank you much for the presentation I have just quick question and in the end the the the pentest reports you given to the customer so do you have any experience or or things you think important to reduce the the time that you need to again clarify everything with the customer because that cost lots of time if if it's not clear we don't

have that problem uh we write really really well no this is really really important uh I saw reports in from German companies they are honestly I'm going to be I please Inu the uh um please sorry for the language but they are really really not explaining well and we if you ever will look at a pentest report from our I spent sometimes three or four pages and just describing the vulnerability and the actual impact maybe it's also because we have quite awesome customers but uh they basically they they don't come back with a lot of questions regarding uh the understanding this might also be because they are quite uh we have a lot of application developers and they already

have some serious um knowledge in it security but uh normally I really spend a lot of time into creating reports and I actually think that this is a ratio if you want to measure your quality how many questions do you get back from your reports uh if you if you don't get any uh feedback on that there's one problem that the people did not ask so ask them or if everything was okay then you also have this as a small feedback but if there is a lot of confusion on the customer side and even if you think that your customer actually is a technical understanding person you actually know that you have a problem in your

reporting hope that answers your question thank you so much everyone another round of applause for Hans Martin