Write Your Damn Report

okay well hello everyone and thank you bides for giv me the opportunity to speak here and I've come to speak about write your damn report um just to introduce myself I'm one of the Security Consultants at pentest I've been a pentester for about 20 years now uh I'm also an open source developer and I'm also a chapter lead for OS Manchester just a little plug we still have a few slots for April meet if anyone wants to join us now you may be wondering why I chose to come and speak about this rather dull topic that's a bit of a antha to many pentesters well the seed was planted last year when in one of the wrapup

speak someone said no one's come and talked about pentest reporting and I thought well hang on is this relevant to us I mean perhaps just to get a feel for the audience I mean who here has ever written a pentest report and who here has read or received a pentest report okay and who here has got an opinion about what makes a good pentest report right fantastic so I think we're on board now um I've got quite a lot of experience building uh bespoke report generation systems basically I've worked at three different jobs and built three entirely different report generation systems to see the different workflows and what I thought I'd do today is just

share in chronologically my experience so my very first job in cyber was a vulnerability assessment firm so this um this place this was doing like large automated scans of Estates and they've got like an automated system with Central databas and scan nodes and there were scripts to take the tool output as XML and pass it into the database and there was like a workflow for the testers to go through and like confirm Mark St as foral positives and then finally all that data was used to generate HDML reports [Music] um now it was at this job that someone gave me a very good piece of advice and this is really stuck with me um because

what we were doing um we were mostly taking nest's output and mapping it to our own descriptions and the advice I was given was before writing the descript I don't just try and like paraphrase what's come out of the tool understand what the vulnerability is first read the description read read the supporting references when you understand it at that point you can in writing explain what you understand to the client and I think it really stands out when you read a report if what someone's writing is something they understand or something they're paraphrasing and then crucially if a client comes back to you and challenge es you what you do not want to do is

just say Ah that's just what the scanner said you want to have some ownership of it so this bit of advice must be 20 years ago I was given this but I think it's just as valid today as it was then now having all the um all the scan date in the database gave a number of advantages it meant some of these features were quite straightforward to implement so we can have suppressions so if a client says this is a false positive it's marked as a false positive and it stays that in future scam you can highlight what's new in a scam um and you can give reports that are organized in different ways you have it by host by

vulnerability and also um you can do trending reports um so I remember the front page of the scans looks something like this and which I thought is like a really like nice introduction it's like it's a very high level view um bit like a graphical exact summary and then what someone can do is they can drill into the detail that's particularly relevant to them one thing though that I would point out is whenever you do these kind of graphs the lowrisk vulnerabilities tend to sort of overwhelm everything else if you take an initial look at this you think what are our it teams doing you know here we are eight months into the scanning program and we've only like

reduced that much but the thing is no one really cares about low do they um so actually what is much more useful to a client is that and if if you wipe away the CR you see what's important and something that was quite quite cool about doing the reports is him out was that they were Dynamic so you could have little check boxes to like enable and disable different risk ratings um okay so that was job one dat Spas backs HML reports I can see some big benefits from doing it that way and then I came to job two and this was at an in-house pentest team and here the report template it was a Word document and and thus began a

somewhat LoveHate relationship with word in this in this context so you'll all be familiar with this so they report had your scope had exact summary it had a table of the findings and then it drilled down into the detail of the findings and I was thinking okay well where's the cool automated database back system well there was a shared document on a drive with some comment Ed text but of course no one even really updated that so the reality was is what what you did is if you wanted some boiler plate text you went on an old report and you copied it from that and and actually what I realized was it was in a lot of

ways this was a bit anti-team because everyone had kind of their own list of old reports that they'd used and they were different and I I thought it got away in the way of collaboration and then you send it to stakeholders as a PDF and I've realized that there are a lot of reasons people like PDFs for the reports because if your reports like in an HTML portal and 5 years down the line you have some incident and you want to show evidence that you had a pentest done and then you point at this portal and it's like oh well you know data retention policy or companies gone out of business having it all as a file that you can like save in

a shared folder and keep is actually really valuable it just means you miss out on all the cool stuff you can do in HTML the other thing I realized is having seen all the cool stuff you could do with trending data there was virtually no interest in it at all almost all the systems would be like pent tested like once in a blue moon maybe like once as a project and if we're lucky once in the next five years I mean maybe that was a shortcoming on their part but that's how it was there was just no interest in that trending data but despite this I thought come on guys I can do all this cool stuff with a

datase p reporting system so I built it was a kind of like Renegade sort of ser approved project um and it was it a pyth web app using turbo gears which was the cool framework at the time um but because they weren primarily a Microsoft house one of the things I had to agree to was to make it run on Microsoft tech even though it was python um so I actually ended up doing quite a lot of uh contributions to Turbo gear and SQL Alchemy to make them work well um on Microsoft tech and it had a cool online scoping form and because this was all internal I could expose this on the internet and it was all

single sign on um and then similarly we had scripts pass tool output testing workflow and then PDFs were generated from the data in the data space um now one massive benefit we had as an in-house team is we had access to really good tooling um so previously a lot a lot of like my infrastructure work had primarily just been nus now I did a lot of scanning credential scanning of uh Windows servers but as well as nesses we had access to security Expressions so th this was really helpful it meant we had two completely independent tools that were looking for the same classes of vulnerabilities and I was able to do some quite fun stuff with this

so this is a little intro to the database schema that was used so as tool output was passed and put in the database it would all go in a results table so we'd have a Target then the tool that found it and then all the tools they will output an ID for their specific finding and then separately we got our table of vulnerability descriptions and then what pieced it all together was this Vol map table so this is this would take the Tuple the tool and the Tool's ID and it would map it to our own ID now at first it took quite a bit of effort to build up this data but in time this became incredibly valuable

because what we could do is we could import both tools output into our testing workflow and for each finding we could see which tools had flaged it up and at this point I made a policy decision that if the two tools agreed that was just truth I wasn't going to do any manual investigation and if I ever got called on it I think that was justifiable but the really interesting part was when the tools disagreed and what that meant is we could focus the manual followup about to to cases where it really added value um we ended up finding a whole load of bugs in nessus actually um a thing Ness has recurrently did is if You' got like

patch a and then patch B that supersedes patch a and you've got a server that's got patch B and S not patch a now security Expressions was quite good at understanding this relationship now nessus kind of tried to and did maybe like 80% of the time but it often failed to um and of course we could then see that do a bit of manual followup the other thing of course in this example um only nus has found it unfortunately when it comes to third party software on Windows the reach of both tools was fairly limited so Microsoft stuff they were pretty solid when it came to SV dip or something in this case only NASA's was able to pick

it up now we also did a lot of web app testing um at this company and what I did was I I continued the database model so I just effectively treated manual testing as yet another tool so in the workflow you could add a manual results you could add you could add some extra text detail I think in hindsight we were probably pretty light on the detail that we did includes um and then this leads me to my third job where I works at a penss consultancy now the very first thing I noticed was that immediately there were high expectations around report quality and that everything that was reported had to have a lot of detail it had to

have like evidence uh screenshots request response blocks um what what the uh the CSO always said is we've got to be able to reproduce it whether it's a client or whether it's your colleague a year down the line to be able to reproduce it and that means quite a lot of detail also there was a peer QA process so you weren't allowed to just finish reporting send it off to a client you had to get it through the QA process and that originally used a latch based system um now this is where I learn I got another piece of good advice um because quite a number of Consultants including myself um would have a tendency to build up a

bit of a backlog of reporting do the work the report ends up being late and the solution to this is to report as you go don't leave it to the end do it as you go uh so built a database back system for this company and we did go back to using word and we found that there were a lot of benefits to word um especially around the QA process and track changes features very powerful there's not not a lot other than word that does start so fluidly um so we used to co call dockx templates and that's that's quite cool because it let you put tags right in a Word document and then it will then process

these and like fill in data but very soon we hit all sorts of problems um so docket templator can do some things but if you want to do fancy things it's limited uh to access the database you need to be online so if you're in a client data center you might be out of look but the biggest problem was this was a workflow pain in that uh in that once you generate this word document from the data space you were then a bit stuck because if you edited the word document if you then regenerated it you lost all your changes um and also things like it could get really annoying to say because it it's generated a graph and the table and

the description if you if while you're writing up you decide oh this isn't actually highrisk this is only median you've got to then go back and change the table and change the graph now one of my colleagues introduced me to a completely different architecture for this and it meant letting go of a lot of things because I'd seen a lot of benefit of using a database for everything um and his design was to throw throw this all away just have a word add in and one of the massive benefits of course is because you're using the official Microsoft interface is you've got access to everything I mean sometimes it's a bit Arcane uh but in general simple things

are easy and complex things are possible and we still have a knowledge base of vulnerability descriptions but it was cach locally so if you were in the middle of a data sensor you can still access it and I think what what Pro proved to be very beneficial is this really fitted well with this report as you go workpl process uh and I will just just to show you what this what this works like um

you can do things like ah okay it's yeah you can do things like you can add a port scan result and all it's doing is just directly passing the XL and dumping it in word and there so there's no day space involved and then another um feature colleague did was a burp extension um now burp of course it does look lovely syntax highlighting of your requests and responses but if you copy and paste it into a Word document you lose all the nice syntax highlighting this extension lets you copy and keep the syntax highlighting which was a really nice touch um if you want to have a nosy at other companies pentest reports there is

an online repository that's got many of them and I think my conclusion is um three jobs three very different reporting systems and this is possibly why so many pentest firms have a bespoke system and that the commercial systems don't get that widely used and I think my final point is considering there's all this cool stuff that we can do with databased back to HML reports and we don't do it because we're making PDFs I think we could do a bit more Innovation and Reporting

does anyone have any questions so the word add in is not and won't be that burp extension he's got a long-term intention to make it open S us

[Music]

yeah so Innovation I was thinking of um it would be really cool if we did actually do HTML reports you can stick all the HTML stuff in a single file so you've still got that feature that it's a single file that the client can keep and you could have Dynamic stuff um in terms of making the data open um I mean quite often I end up putting things in spreadsheets for clients um I think going further than that would be difficult because typically you're not engaging with the same client that much

yeah so I mean I something around that people sometimes asked like can you automatically create tickets and and that's normally turned out to be a bad idea um but I think what's quite a nice touch you see some tooling it's got a button so you can do a oneclick create a ticket so someone like a project manager or scrum Master they can look at the results and they can then make a human-based decision about if they're going to fix it or not and then if they are it's one click and at least that takes care of a lot of the clerical stuff but yeah I mean this this is exactly what I mean doing more stuff like this would be good

Innovation to

[Music]

see so so in terms of work experience I'm afraid at the minute pentest aren't able to offer that we have done in the past maybe we will in the future in terms of advice I think I'd go back to was it my second or third slide now where I said don't regurgitate understand work on understanding and I think when you're communicating with people in the industry the way you speak will prove your understanding and that'll take you a long way

please appuse