Document It, Or It Did Not Happen: Writing a Quality Pentest Report

hello good afternoon hope you all were able to get some lunch um thank you for coming so this talk is the just some advice on how to write a pen test report and are you guys able to hear me in the back okay awesome okay so uh my name is Shelby Pettit I work in application security and at Adobe I haven't really Niche role which is cool that gives me an opportunity to read pen test reports written from pen testers across the globe it's really fun um I've seen them from Big firms small firms India China us it's been really fun um so that's kind of where I'm drawing from my experience sharing some of my uh

my opinions on what makes a pen test report useful or clunky in my opinion or sparse um so hopefully that is helpful but I will say sorry I'm kind of struggling with the presentations format but I will say I am not a pen tester so I will not have tons of uh really practical tips on the actual Logistics of writing the pen test so apologies for that um who's this talk for mostly pen testers um there's going to be some overlap for red teamers but since their objectives are a little bit different in the in their uh what they're trying to achieve there'll be some overlap but that's not the main focus here um so maybe some of you I

if I ask people to raise their hands who went who decided to become a pen tester because they like writing no one's going to raise their hand um I get that but here are just a couple quotes that help us realize that the pen test report is important because um the people who are your clients um that's that's their only tangible deliverable you know out of this whole engagement all they have is the report to show for it so we've got to make sure that that represents the quality of the work that you're doing on the back end and everything that goes into that so let's get into the planning phase a bit um first thing we want to talk about is

what's the objective why were you hired to do this job um there there actually might be a couple of reasons the most common is usually for like compliance sake um so it's it's nice to know this background because if they hired you for one purpose and and you kind of focused on the assessment and you don't understand what their goal is you might miss it because they might have something different in mind you know maybe it's for compliance requirements it can help to have some familiarity with those compliance requirements um I definitely don't want to tell you to claim to be a consultant in in the compliance side if that's not where your expertise is

but a little bit of context can help knowing like if it's um if it's in scope for um what's the like the card payment requirements PCI DSS like maybe with that knowledge you know that usually the networking aspect needs to be in scope so just a little bit of background can help you out there but um yeah don't claim to consult in areas where you don't have expertise I'm not trying to say that um I'm trying to think of other ideas for why a pen test might happen they might have internal processes things that they're looking for or maybe they have a very specific objective in mind that you would only know by asking so

that's that's a question you should ask a person um that hires you as you go to prepare for the engagement

foreign

with my degrees in I.T I can just barely manage a PowerPoint okay so next thing you want to know is who is your audience once that report leaves your hands and goes to the customer who sees that and who cares um usually it's going to be a mix of technical people and non-technical people leadership is going to want to see this oftentimes so you kind of it's the challenge is you have to write to a mixed audience some people are going to read it from different perspectives um so it's kind of nice to know another thing I wrote in here is classification understand the data classification they might want you to write some a little

footer on your report or something the effect of like confidential only or things like that just make sure that you respect the confidences of your um of the person who hired you to do this job that will help instill some trust make sure that you handle the data according to a way that they feel comfortable with next thing is timing um it's I'm sure it's very challenging as a as a pen tester you've got a million and one tasks to do in the in the course of this assessment um and just don't forget to leave some time for that report um so yeah just maybe plan on it taking a little longer than you think

to make sure it's Polished in time to go even though we have these tight time-boxed assessments next thing is format um I have a lot of opinions on this I've seen pen test reports come in many different formats most typically it's like a Word document or a PDF either of those is really fine um my only my my big feeling is I need to be able to copy paste out of your report um some some clients might request a CVS um or sorry CSV um that way they can ingest it into whatever ticket managing system they're using or Data Tracking so they might give you specifications to please submit your data in this format separately from a CSV requirement I

personally do not like pen test reports in Excel format and unless if your client requests it right client is King but um the reason I have this bias against pen test reports in Excel is I don't want to have to carry around a screen this big with me to read all of the texts in every little cell that's that's kind of clunky for me not my favorite maybe some people say yeah but I like that I can tabulate the results quickly in the cells that's a point but um actually this is just kind of for kicks and Giggles uh I have seen visuals like this before in reports I'm like thank you they're really helpful

um it's not a bad idea to include some visuals in your report I think management their eyes tend to go toward the graphs and things like that that can be helpful uh don't use a 3D graph like I threw in here it's kind of corny but yeah yeah that's what I have to say for format moving on to the structure so you you have the Liberty to to structure the way you want this is my recommendation what I think is a nice flow you'll have a cover page and you're going to slap your firm's beautiful logo right on there nice and big and a title of what was assessed um on your executive summary so there are a number of things and I

don't know where my notes went let me see if I can find my notes that will help me so in the executive summary basically we want to keep this to a page or less this is a summary of what you found you even though it's showing up early in the report you're going to write this last most likely and um you so at a click at a quick glance the reader should be able to know a couple things right off the off the bat they should know how many of each findings were found categorized by severity they should know um that's not something you want to search for in the report if you have a

10 page or a 20 Page report you don't want to search about you they just want that to stand right out next thing is um you can you can show a distribution by category or um by severity whatever makes sense for you um also a summary if you saw some trends like maybe there's a particular area of security like um you need more input sanitation sanitization or um or more Network segmentation if you start to see some trends that popped up this would be a good place to write a sentence or two like hey here's some things we saw here's some areas for you to focus on as you go to improve the security posture of your organization so

that's good to know you can also throw in if there's like a very blaring or severe finding you can mention that here too specifically just quickly um so yeah we'll we'll add a little bit more details to the executive summary later moving to the table of contents personally if you can put a bullet point in the table of contents for each one of the findings so I can click on the link and it takes me to that part of the report I love that it just makes my heart Happy um the findings are the real meat of the report that's where the good stuff goes um so you want to list it from most severe to least severe so that's how

you're going to categorize them and we'll go into details for what to include per finding later is so so far at this method we've gone from General to more specific we're going um we're kind of drilling down so starting at summary and down into the Nitty Gritty details um so if if there's something that starts to get really bulky but it's relevant information you can pull that out separately and throw it as appendix a or appendix B that's no problem that helps the readability of your report um there's there's some judgment calls but I'm sure whatever you organize will be nice um one more comment about structure there's one particular style that kind of drives me crazy and that's when your

summary is big if you've got three pages of summary that's not a summary and I don't want to see it because all of a sudden half of your findings are half of the details of a finding are in the summary and then later on in the report we have the other half of it and that's messy to jump back and forth give me a succinct summary and then I can go find the D all of the details together in the findings section um so that's just the an opinion I have about that moving on to the next part

here are a couple more details of things that need to be included in the executive summary um and honestly if you want to include these details somewhere else because that makes sense to you that's fine my preference is to put in the executive summary except for maybe the definitions or framework if that starts to get detailed or Hefty you can toss that into the appendix no problem so you definitely need to include the scope that might be URLs IP ranges things like that or a scope of the the application that you covered a description is fine too whatever works makes sense for the assessment the objective you know the objective of the person who hired you did now you

want to make sure that that's communicated clearly to whoever the reader is so they understand that we're all on the same page here that you met that objective um you also want to have this the start state of the test and the end date the methodology did you use a white box assessment gray box black box and whatever else those definitions are kind of nebulous so whatever else is helpful context you can throw that in there too if you followed a particular pen test framework such as um like the O wasp guide to securing web apps or um like pcis or the CIS benchmarks those those are really helpful to put in there because that's a reference that people

can understand exactly what was tested um you might also have a set of definitions maybe the way that you define or use certain terms is specific and you want to get that cleared there that's really helpful I've seen tables of definitions included in dependencies and that's fine well that's really helpful can be with severity rankings especially because everyone might have different opinions about what should be considered a critical or a low or anything in between so definitions can be helpful there the last thing we have here is assumptions so um go back with me a few years to one or many years to when you were in high school physics and your textbook told you to imagine a world with no air

resistance and um and things like that you know no air resistance gravity is the same as Earth's those are assumptions these underlying assumptions that we make going into into this uh the assessment maybe it doesn't represent the real world entirely but these are assumptions that you've been told to or have incorporated into your test that's fine those would be appropriate to include in the test so they understand your approach so any questions so far keep going awesome

hello

okay so findings um each finding should include a title this can be simple cross-site scripting stored cross-site scripting or something like that you also want to have a tracking number um companies tend to I don't know how it's used internally to pen test firms but I have seen a trend of everyone assigning kind of like this uh tracking number you know five six one two three four five or something like that to every single finding they have that way there's there's no uh mistake about which finding we're referencing in our discussions and things like that um so they have a tracking number whatever works for you also severity ranking um different organizations have different preferences different risk

appetites and approaches so whatever they and you can agree on as a as an appropriate way to assess because you don't want to give just a report of your findings without this layer of analysis you need to prioritize for them do that analysis as a security professional I'm telling you to focus on this and this and then if you have time also that that and that right so you want to help them prioritize that's the point of the severity ranking um if you're taking a risk-based approach cvssb3 is great um or I've seen scales critical high medium low and informational that's great too you can Define what those look like from a risk impact perspective in

your appendix um priority one priority two I've seen Banks use this kind of status they've got those defined on their side and then they they slap that label on so everyone knows this is how we handle p1s this is how we handle P2 things like that and the slas kind of associated with that um so that's kind of severity oh I guess there's there's another approach there's also um the adversary based approach so there's like you can have a risk-based approach or an adversary-based approach um as long as you're consistent and you can defend why you put a certain label on there um that is just fine but there can be some some differences in opinions with with severity ranking

and it might depend on your organization and that's fine

next you're going to need a description of the volume so not everyone who reads your report in fact I would even say the majority of the people who read your pen test report probably are not Security Professionals some are but you're probably some at the end of the day assist admin or an Ops person is engineer is going to get this report and be told to fix things and so you can't take for granted that if you throw out these terms these jargon that we that we talk about in the security world that they're going to know exactly what you're saying so you just add a sentence or two that explains this is what this

finding means in just in simple terms because not everyone's going to know that but you can put it in terms that they understand um just this doesn't have to be huge a description of the volume can be a couple sentences maybe a small paragraph or two and that's I think that's really helpful even even for people that work in this it's nice to have a little refresher make sure we're just talking about the same thing on the same page next is

explanation of impact so I've explained what you found now so what why should we care about this um you make sure you differentiate what you're able to prove versus what is potentially um the impact here as much as you can as a pen tester it's great to show the impact actually demonstrated so that it becomes it comes out of this theoretical realm and into the look for real we actually did this here's a screenshot right um so that's what the the explanation of impact if you're not able to show that full attack chain you can't throw the theoretical stuff in there just make sure it's stated appropriately so they understand what was done and what was

really discovered steps to reproduce um I find that this is most often the part of a pen test that's lacking so this is once once again keeping in mind that an engineer who may not have a background in security is going to have to try to reproduce your finding in order to successfully patch against it right so they've got to do what you did so you have to give them instructions you've got to give them a recipe you use this tool you navigate to this tab of the tool and click this and type that and it's going to give you the result here and and you walk them through it I think that's so helpful it might feel

wordy to you or maybe unnecessary but I think as someone who consumes the reports and and has this back and forth with the engineers it's really helpful it helps us understand exactly what you did so that we can not waste too much time trying to rebuild what you've already discovered last thing is evidence um evidence kind of shows it could be a screenshot or a video share of what you managed to capture what you managed to do evidence is not always complete without those steps to reproduce I kind of think of the evidence as like the end point and the steps reproduces like the journey how you got there and I we are running low on time so

affected location the URL the endpoint the IP address the file name whatever it was list all of them if if you're if time if you have time enough in your assessment to identify every location where it's vulnerable within scope that's way more helpful than just saying it's all over um if it's a universal setting maybe they can go fix it but sometimes depending on the remediation that's required they might have to go and do it for each location so it's nice to have that all listed out every affected location recommended remediation if you're able to give more than one remediation that's so helpful because you know we get to the engineering side and we say here's our recommendation

they say we can't do that it's working as designed we're like okay well here's our backup recommendation and our Backup backup recommendation maybe you won't fix the whole maybe you won't patch the whole vulnerability with this third backup but it gives them options you know this is best better than doing nothing so I like to tier my recommendations have multiple approaches for fixing it you have your best practice and then maybe your backups now that you know I've given you like a whole laundry list of things that you need to be documenting you know ahead of time so you try to document as you go take notes of your progress of your process capture screenshots as you go so

you don't have to try to figure out at the end of the week what you did and I know some people do set up logging to capture their their traffic that helps them get that information later so here's just a quick example we're going to run through I borrowed this from Sans um you can ignore that red box that's just for the demonstration thing but I like that they have um here we can see their severity the their Threat Level the vulnerability analysis if if we followed the the template I gave you I would name analysis maybe like the description and then maybe add another header here toward the bottom that says like steps to reproduce

um but yeah here we have screenshots they've labeled their screenshots which is really nice they labeled this one figure six getting shell access so if you just look at the screenshot or the the label on the screenshot it's very helpful it tells you exactly what they did um one thing I would add if you want to just kind of make it a little bit more polished I would highlight after you take the screenshot put a little red box around the part of the screenshot that's interesting they don't you know people are going to look at this be like okay that's a lot of text what so if you put that little red box it draws their eyes to the right spot

that's really helpful for me once again I I do appreciate that they highlighted the right row here showing that they have net cat running that they're able to install that and get it going um and then here after the after their walk through they give an impact as well as a recommendation and they have external resources hey if you want to go do more reading here's a link very helpful as you are writing we don't want to use Scare Tactics right um try to try to take as evidence-based approach what does the data say um and don't go too far from that as you write reports you're going to find that a lot of it is copy paste so you can

feel free to build a template or use software that will do that for you just to save yourself some time I find that just a fun little tip if as you're filling out that template if you find pieces that you you're like oh I need to come back and fill that in later put special characters there like in the past when I'm working on big long reports I would highlight it but then you're trusting your visual check and that's not foolproof so put like three asterisks in a row or three um dollar signs in a row and then at before you submit your report do a text search for those characters uh because it should only come up in those spots

where you meant to come back later and filament just so you don't forget little things like insert customer name here in if you do a white box assessment you'll probably want to collaborate between the rough draft stage and the final report stage so that the engineers and the system um like designers on the customer side can have a chance to explain anything that they might disagree with and then you can negotiate maybe some finding severities might need to be adjusted or things like that based on new evidence that they that they show your firm also might have a QA process lasting letter of attestation um this is very similar to an executive summary you're going to have a lot of

the same details in there and this is as um this is basically proof that the assessment happened and kind of a high level summary of what the findings you might not release the name of the findings because they're not patched yet but this letter can be given to for for many use cases to show that the pen test happened maybe I don't know how that holds up in audits versus the full report but like um I know I work for a software company and so our customers want to see that proof that we have been doing our due diligence and are doing thorough Penta so they they examine our scope and all those things that are included in that

Loa that we're able to share with them externally without sharing the details of all the open volts right so that's really helpful to have that letterhead pretty Loa to share and then last thing there's lots of software that can help you write good reports but now that you I can't recommend any because I haven't used them but now that I've given you kind of the basics of the ingredients for a good report you can go out and search for things that make make sense for you and that work well for you um so yeah I am out of time sorry we don't have time for QA but thank you that is all [Laughter]