BSides PDX 2023 - A Gentle Introduction to Understanding Fuzzers (Allison Naaktgeboren)

[Music] all right hello everyone this is a general introduction to understanding fuzzers or Allison doesn't know how to name her talks but this is the track two and hopefully you all know you're at bsides although if you don't congratulations you're at bsides so we're going to do a few things today I'm going talk about introductions and goals we're going to talk about fuzzing isn't isn't we're going to break down a fuzzer sort of how you understand what these things are we're going to talk about who uses a fuzzer and why and then the second half is sort of craft I was asked for to give some specific advice about fuzzing but it's hard to do

that in a general case so we'll talk about mutational coverage guided fuzzers the most common of which is the AFL family AFL Plus+ AFL itself lib AFL the 30 plus research fuzzers we have written based on AFL and then we're talk about optimizing bits of it and then how it would fit into the life cycle this part's more craft than science so it's more like practical advice but it's also hardly airtight and then at the end not that I expect any of you to do this today because when I put this together I didn't know it was going to be the last talk but there is a lab it's available online you can do it whenever I'll have

my email and you can email me if you get stuck or have problems so I don't expect you guys to do it today even if you are surprisingly awake but it is there you can try it and I will help you so who am I I'm all this stuff I believe you can all can read even in the back but the important thing here is I'm an academic who studies fuzzers I'm not a vulnerability research person I don't use them in my Daily Grind beond like benchmarking them but if you are a VR person I would like to talk to you because I can't build a fuzzer to help you if I don't know what you want prior

career I was a compiled application Dev like Rowan I tend to build things that I don't actually use but other people do and besides being a bsides volunteer I'm also an instructor lecturer as all PhD students are we occupy this super position of what the end University considers convenient but if you um if something happened that was bad or made you feel bad even if it wasn't a violation of the COC I hope you all know you agreed to by being here uh the organizers would still like to know about it we'd like to make each year better I usually there's a survey that goes out where we ask you about how Stuff went and what you'd like and what

could maybe do better cool all right so who I think you are um I think you're probably fried from two days aone because I am fried from two days agone um I believe you are girl awesome for coming to the last talk I know that's really hard and I'm going to try to focus on some big picture ideas and the common stuff slides will be up so if you're like yeah they said some cool things I don't remember what it was the weekend it's already a blur you know I'll put that some up so I assume you've at least heard of fuzzing and that's why you're here although you may not know much about it I think you're probably

relatively new to intro infosec h I need to slow down because bides tries to balance content to have intro stuff medium stuff and kind of advanced stuff we don't always win at organizing the schedule to make that obvious but that is what we try to do so I assume you are all relatively new um You probably have some developer experience I hope you've had a CS class or equivalent maybe a boot camp if you're an expert you're probably going to be kind of bored but you can look at cute cat pictures on the internet I I won't hold it against you if you find some good ones let me know so the goal of this talk is is to try to

give you some ways to think about framing how you use fuzzers what a fuzzer is what it isn't when to reach for it when to not going to try to avoid being prescriptive do this do not that because I want you to think about the why why do we tell people to do these things with fuzzers why is that and does it apply to the situation at hand the greatest tool in CS is knowing when to reach for right tool and when to put it down and get a new one we don't really have silver bullets and we don't really have free lunches I hope you learned something to be honest at this point I'm grateful if you don't

fall asleep that's a win uh and I would like you to leave here understanding that fuzzing is still more craft than science I can tell you what it is where we got it Bunches of things that are useful for certain classes of fuzzers but we're still not at the point where it's a Well understood science it doesn't have the mathematical rigor of other areas in CS so that's why I want you to hopefully learn to apply some of this but keep it keep in the mind it is craft you may have to tune it it may not always apply it is not a universal law we don't really have fuzzing axioms though my adviser probably wishes

we did so then let's get into what a fuzzing isn't I hope by this point given this is a security conference I don't have to convince you that vulnerabilities are a problem anyone disagree with that okay cool you're all awake so why fuzzing well fuzzing is a form of security Automation and we kind of need Automation in this field because we have about one infosec expert for every I don't know 100 or 300 developers and each developer has a probably right to bug about every 75 to 100 lines of code regardless of age or experience they keep showing up and that simply does not scale even if you are bad at math you can see why this could be a problem so

we have software that we build tools to help us I mean the someone who builds software this is the part that's interesting to me I build tools to help people maybe not me but other people so where did fuzzing come from it was a dark and stormy night and they only had dialup in 1989 oo yeah okay well I mean it's sunny and 3:00 so I can't really blame you for not looking scared but anyway a young Professor was trying to work from home which you very few people could do in 1989 through a dialup line when lightning hits the pole scrambles the standard in going through the dialup line and promptly crashes his program he

doesn't get to finish what he's working on what's a poor CS Professor to do well your CS students always need final projects and having students solve your problems that you find kind of annoying but not quite enough annoying to fix yourself is a Time honor tradition so in 1990 we get the public invention of fuzzing it's an article in the um issue of communications the ACM it is a magazine article I have it with me actually very carefully didn't bring it to the talk because I am also fried from two days of con anyway he assigns it to students what actually isn't as well known is that two student groups attempted the project one half asked it

and one knocked it out of the park and gave it their all guess which group is famous the second one right so from all the students in the room go big on your final projects as someone who is a greater and a professor I will tell you that we will give you credit for trying a lot of things as long as you duck this is why having minions is great the invention of fuzzing I'll have it up here I'll have it up here for you to look at afterwards but yeah it they did that is their final project that is a pair of students and you know a professor helping them with the editing and providing guidance and

oversite however the rest of the field they are graduates in uh systems and operating systems as we now call the field and the reception kind of sucks it facts it sucks so much that other faculty told Dr Miller he should drop out of Cs because the research sucks so much wow when I say poor I mean mean um but he doesn't however the security community in particularly vulnerability researchers give it a warm reception albeit they don't tell people for a very long time that they have all adopted this thing so uh if you are interested in reading about it you there's a lovely interview with Dr Miller again I'll post the slid so you can click on the link

it's worth reading now the fuzzer that sort of is the first most people heard of is AFL this kind of raises to popularity late aies early teens but there many fuzzers that predate it but this is the one that everyone seems to have heard of first and there are various theories for that it's got an impressive track record it's got the screen that gives you the illusion of control uh it is fairly easy to set up and maybe it was just in the right place at the right time I don't espouse any of these I've just heard all of these over the year so please be aware and if this is interesting to which I'm hoping it is

the 30-year retrospective by Dr Miller and his current students is worth a read tldr classic fuzzing still annoyingly useful if you went to Rowan's talk yesterday you heard him mention that he piped Dev Rand into student projects and pretty much universally brought them all down works on production devs too even those of us with shiny titles so now I hopefully convince you that this thing is useful and a little bit of its delightful backstory what the hell is it well fuzzing and purist form aims to thoroughly explore the input space of a Target you can call it a victim a fuzzy a program under test software under test we suck at noming clature um looking for

inputs that cause interesting Behavior what do I mean by interesting Behavior well that is up to the fuzzer Creator and what makes interesting Behavior to a fuzzer is a key point of its taxonomy not all fuzzers consider the same things interesting it's sort of a key thing you have to understand with working with the fuzzer and ultimately fuzzing in its practical form is a stochastic or probabilistic technique it's Dynamic you have to run the code that's one of its potential downfalls and why it doesn't fit for every situation sometimes you can't or sometimes it's a really bad idea uh if you want to get into the bar fight about whether symbolic execution is a subset

of fuzzing or fuzzing is a subset of symbolic execution bars two blocks that way I'mma stay out of that fight but the other thing you need to know is that the original fuzzer didn't really have one of these because everything it worked with spoke standard in but there is sometime a need for a driver also known as a harness which acts as a glue between the fuzzer output and the fuzzy input uh not all pairs need one but if you do you reallya want to make sure you put the engineering effort into it if you are trying to knock on a thing that takes you know trying to knock on a 100 Doors you only actually knock on one are

you going to find the bugs behind the other 99 doors nah so if there are 100 ways into your program and a 100 ways to get inputs in there you kind of need to probe all hundred if you want to find bugs if you only probe one you probably going to miss stuff I won't buy that this is a thing that gets overlooked by new people fairly often don't underestimate the value of your driver if you need one all right so what does working with the fuzzer look like uh right uh I should explain that AFL stands for American fuzzy Lop we have this history of naming fuzzers after fuzzy furry things which is a pun that's immediately

obvious to English speakers and really confusing to the rest of the world uh and that's why there are lots of rabbit jokes and fuzzing is because the most famous fuzzer is named after a rabbit I've managed to convince Google that I don't want the Australian football league finally but anyway you're going to start with either this pictorial diagram uh I mention I'm not the artist in the family you're going to set up now setup can be non-trivial all these steps can be non-trivial but whatever your setup is you're going to do it and then you're probably going to debug it with some short fuzz jobs to make sure you have all the settings you need it's actually

doing what you want and you know just sort of testing the system that you're going to get what you want cuz nothing is worse than running the thing coming back 24 hours later and going let's see our glorious results and it' be like no permissions Exit Zero bye and you're like damn it all right so you once you have are you happy with the configuration and that everything's behaving itself at runtime you're going to fuzz for real then you're going to review and check your results what comes out of the fuzzer may not be gold we have What's called the crash triage problem or the reproducibly problem not all of the things that the fuzzer finds interesting

are what you find interesting and not all of them are going to be reproducible because a fuzzer does not always know why so your second thing which is sort of optional if you're on the defender side we'll talk about a little bit more you will need to analyze the result because I get messages hey I heard you're the fuzzing person uh I got this crash what's wrong with my code I'm like whatever could cause a seg fold is what could be wrong with your code they're like yeah but how do I get it to tell me I'm like it doesn't know it can't tell you uh you have to this is where you earn your salary uh so if you are going

someone who has to do something with the results analyzing it is non-trivial and I have seen ratios north of like a thousand to one for basically false crash reports to True bugs so do not underestimate four and five and that's an area of my own research which we'll probably not have time to talk about but Hope Springs Eternal um this will also called the root cause problem because not only can a fuzzer not tell you why it struggles to tell them apart so it uses a series of assumption and approximation that actually don't work so well that often to try to guess oh this crash was different than that crash these are two different bugs it

usually gets that wrong and it usually airs on the side of reporting more crashes and making more work for you than accidentally hiding something that might be new uh hopefully you'll do something with the results you spent all that time and energy you've used electricity hopefully you do something with the results nothing's more frustrating than people fuzz and they just like drop the results I like well why' you do all that then you paid for that server time aren't she going to do something uh sometimes you spend all that time and you don't get results in which case you might have to go back and retune the fuzzer is there something else you could use is there a fuzzer

that better fits this you know know and then you're going to repeat this cycle questions about that this is like what the Daily Grind of using a fuzer looks like okay I see some thumbs up all right so when is a fuzzer a bad idea right fuzzer is not like we have great tools it's the great tool don't get me wrong but not a great not every great tool is great for every job and one of the important things in CS is recognizing that and knowing what kind of tool to reach for so first thing to know is you should know that fuzzing is generally hard on your Hardware it's CPU intensive it's very Ram

intensive and if you're running that in the cloud that's expensive and even if you're not running in the cloud that's expensive you have to pay for your own electricity you have to buy computers you have to buy all that Ram um probability is not always your friend if you are rolling randomly on an even dice what are your odds of guessing 1 through 37 you say 8 by integer for before anyone gets too cheeky good yeah not good uh one and six one and two to the 64 which really is really not good so if you know you're going to have a thing where probability is not your friend those types of puzzles are traditionally not great for

fuzzing uh it's also not good for truly dangerous code if you're like I don't know if this implant's truly dangerous I don't know if it can escape like if you're worried about a VM Escape maybe don't fuzz um because that's a problem you have to run it it's not static analysis you might use static analysis to make your fuzzer smarter But ultimately you got to run the thing and if you're afraid to do that or you can't do that you kind of can't fuzz it and as I mentioned earlier it does not know why which means it can't tell you why if you want to know but it also means you often get these bad signal

noise to Signal ratios you get a lot of reports and you might have to go look in for that needle in the hay stack what is the one golden crash report among a thousand turds and you know don't underestimate how long it can take you to figure out the answer especially in large old sea programs and ultimately we're flirting with the halting problem we're going to kind of tow up to that line and batter eyelashes at it and that means we have to rely on approximations and assumptions which work 90% of the time but if you find yourself in that other 10% not great the other thing you have to know pragmatically is that a clean

fuzz job is not evidence that you don't have bugs it just means you didn't find them this round everyone cool with that we have this thing in CS where we really love like the newest Shiny Toy and it's great and it will solve all our problems except it never solves all of our problems so we're going to break down a fuzzer now that we sort talk about what fuzzing is and what it isn't I would say that there are sort of five key features of fuzzer taxonomy if you're like looking at this fuzzer and you're like how does this relate to other fuzzers does it do what I want here are ways to think about it what is

interesting Behavior I mentioned this earlier it's a key thing at the heart of the fuzzer what is it looking for how does it get its new inputs where do we enter that giant Loop that runs until someone says to stop get new inputs all right cool how what's the exit condition when does the sucker stop running and this one is sort of a little bit more specific but relates to how does the fuzzer know if the fuzzy did interesting things what does the fuzzer know about the source that usually determines how much you can know about interesting Behavior so it's kind of a proxy for that question and does that fuzzer have a specialty of focus magic pixie dust that

makes it super special the sales people will always focus on number five so starting with what's interesting Behavior right now one of the dominant paradigms is coverage guided or coverage based and this basically tries to approximate paths through a binary its underlying assumption is that the more you explore of a Target or a victim the more interesting things you're likely to find if you want to see a thing crash you have to get the code to execute first so that kind of makes sense right and so it's going to do that by using those insert like inserted code points and looking for pads and it's going to approximate that by tupal this is the AFL style of code coverage guidance

which seems to be the dominant one another thing you'll run in as to is directed and in case of directed you name a point in the bionaire you're like here flag do dxt get me this uh and then you look for things that get you there make sense so they're kind of related but not quite the same uh any new path is good for coverage base but directed we are looking for a way to a particular point and then taint analysis is a little bit of an older idea we get that from I think the 70s in that case you're looking to trace a path between input in this case colored red to something special like secret and anything that

gets you there is good make sense and then I have never encountered a fuzzer that doesn't think crashes are interesting so you can always count on crashing the program being interesting and the idea is that we accept universally as a field that program should never crash I have only encountered one piece of software where crashing was viewed as okay uh and for reasons you can ask me about that off the recording so the next question is where do you get your new inputs what's the engine that drives finding those things and I'm going to break down three big Ideas a lot of these can be sophisticated there are a lot of hybrids this is a big part of a fuzzer and so

I'm going to make it look really simplistic because it's you know 330 uh so in the classic mutation fuzzing which is the biggest Paradigm right now you're going to start with some input this case I started with this picture of the cat I gave him some whiskers people like the whiskered cat so I'm like okay let's take the cat picture that did really well and give it some more whiskers and see if people like it even more make sense we're basically taking things that did well last time and tweaking them to see if they'll do even better now generational fuzzers were sort of big in the late '90s early 2000s for reasons they fit really well with things that have uh

grammars and so you have some sort of starting algorithm in this case our pictorial version is a face has two eyes a nose and two ears and I'm going to grab from various buckets that it says are legal and put those together so if you've seen in your PL classes like those grammar languages light up top bottom in terms of binding you feed these into that kind of fuzzer those fuzzers are great at this but that's their algorithm you can see how it's kind of different from mutation in the sense that it has a series of set rules that guide how it constructs new seeds we say it generates rather than mutates and then the classic fuzzer the first

one from 1990 basically you got to make sure standard in and Dev random work and you just pipe that in you see what you get and you don't care what happened last time you're only interested in this time and this is sometimes called dumb fuzzing and I think that's a little mean because it's still effective and it's not dumb if it works so exit condition the most common one I see as a researcher is scheduled duration we run our tests and our bench Mark for like set amount of times and then we compare them to each other the most common one I see vulnerability researchers use is user initiated exit they'll like leave it running in the

garage wander away for a week or two and come back and check hey I'm done with that one turn it off um it is theoretically possible and I've only seen this once for a fuzzer to run out of inputs to test it has explored the entire space and it knows it uh that is because that program only took a single 16-bit integer and we can in fact root force that entire space uh you're probably not going to see that one but in theory it exists you could actually explore the whole space and declare yourself done and then there are fuzzers namely lib fuzzer which finds a single crash and it's like I'm out mic drop I found a crash go fix it and exits

uh that one's not popular with VR people for Fairly obvious reasons but is popular with developers because it's presumed you will go fix the bug so uh you may have heard the term white box gray box Black Box I hear the E department is responsible for these terrible terms but in a white box it means you have the source it's possible for you to know just about everything about your fuzzy that you could and then a black box is what the VR Community tends to work with that's a highly optimized compiled binary it's good to go you probably don't work there if you're working with blackbox binaries just saying uh and the thing that you

see very commonly now is gray box that's what's used in AFL and it's called that because it usually starts by taking the source heavily instrumenting it with code points and extra instrum and then running it so you get the speed of the black box but kind of approximate the information of white box so like white box is slow but you have lots of information black box is fast but you don't got a lot of information and this gray box seems to be this like sweet spot in the middle cool all right and the next question is what's your focus or specialty if you're searching for fuzzers you might actually start with this question because if

you're fuzzing a kernel or a car you kind of have different requirements I've encountered fuzzers for cars we have ones for web apis buas which is the workshop going on right now uh we have fuzzers for kernels I have met fuzzers for routers ones for websites you know we have different fuzzers meant for different jobs so you can think this is specialty all right so I've sprouted a lot of theory at you let's try applying some of that and hope that it sort of puts the pieces together so starting with AFL Plus+ because that's like the most common fuzzer that people encounter when they're starting out what is interesting Behavior well system crashes as represented by segals universally

interesting so I hope that's not a surprise and it's coverage guided so it's going to look for new coverage as approximated by tups make sense it's going to get new inputs because it's a mutational fuzzer it's going to mutate things it's going to potentially mutate them in various ways it's got a lot of stuff in there there but ultimately it's going to mutate prior inputs based on their priority if you heard Rowan mention power schedules that's a way of prioritizing seeds and going yo I think this one's a winner put that one at the top do that one next it's got a fancy name and a whole paper but ultimately that's the idea and then

it's exit condition well technically it's user initiated exit most people script it I use it in a benchmark so I just plug into the Benchmark parameter please run my test for 24 hours okay thanks bye uh but the other thing I often see that isn't talked about is your fuzz job gets killed for resource hogging and if you're going to fuzz in the cloud or on Hardware that's not yours you might see this one a lot more often so I'm going to warn you that that one's out there and then how much do the fuzzer know about the fuzzy Source it's a gray box so you're usually running a heavily instrumented binary which leads it's usually built from Source people

have spent some non-t effort trying to like retrofit instrumentation into compiled binaries that's a whole line of research unto itself and then does this fuzzer have a specialty it does it's compiled user land binary applications that is the intended target of AFL out of the box you Rowan mentioned those are heavily fuzzed because there a lot of fuzzers that do this right out of the box makes it easy and it has about TW well probably closer to 30 now research ideas rolled into the source code so this thing does a lot of tricks it's got a lot of mutational strategies it's got a lot of stuff you have to go in there and turn in all the knobs and decide if

they're right for what you're doing doesn't do them all out of the box a joke about is an hour to set up it's a lifetime to master and the documentation could use if you're looking for a volunteer research project I suggest that one all right so who uses a fuzzer and why if you remember nothing else from this talk if you are shopping for a fuzzer before you can do that you have to identify your needs your goals and things like your platform what are you fuzzing what do you want out of it and and all those things before you can go find it if you read fuzzing like oh you'll read blogs POS this fuzzer is

great that fuzzer is trash I don't know what you're talking about and the reason is is because they've got different goals so not everyone wants the same thing in fuzzing despite what people on the internet might tell you all right so we're going to start with the people who really kind of put fuzzing on the map although they didn't invent it it's the offensive folks they want to find vulnerabilities they want to find it fast they generally want to find stuff before everybody else and they give zero FS about why they just need to know that it's reproducible and stable right so they're looking for Fe fers that will be fast to find they want to find

vulnerabilities not bugs bugs are basically a wasted their time and they want to find tricky things and ideally source code not required because realistically most of the time they don't have it uh I'm going to skip that section but you know here's a list of fuzzers that people like that might like every slide has one of these the next group to sort of pick up fuzzing was kind of the defensive people they kind of want to find all the vulnerabilities and they kind of want to find them all before release so they have a slightly different time table and they do care about why because they have to convince people like me to fix them they're going to look for things

that probe as much of the product they kind of need it to fit into the rest of the security workflow works for their teams they're more interested in vulnerabilities but painful bugs are okay too the devs will yell at you if we know about them and didn't tell us true story all right so sort of the last group to pick up fuzzing in industry is the software owners that's people like me uh or was like me I guess I'm retired now uh you looking for things that fine vulnerabilities but all the bugs are your problem too so I I care about crashes and hangs in a way that the offensive people do not I need to find

them about release and I can't fix them until I know why the why is a very big part of the equation from the software owner's perspective uh you often look for different things too I'll be looking for things that Target a specific commit I might want to fuzz recent code I might want to fuzz a new parser we put in in there I'm looking for stability because my to-do list is already way too long and fixing software that should just be working already is not my idea of fun I need bugs and vulnerabilities as Rowan mentioned in his talk snapshotting and time travel are really helpful for that and code coverage is actually interesting to at least the QA people

because you know you want to make sure you can tell your leadership that yes all of the code is tested or all the code we can reach is tested and then the group I am in now is academic researchers which are much maligned but uh and part of that is is because our goals are not always your goals it might be one of those three it might be studying software testing as a field we might be looking at something else in CS and fuzzing just looks like a good venue to study it or at least to get funding to study it let's be honest uh I'm looking as a researcher for things like known benchmarks how will I

test this thing is it publishable is it citable is a good thing to build new stuff on because I need to build stuff to graduate so the two big fuzzers and I say big big might be a little bit of a wretch for vmf but our lib AFL and V vmf which is Vader module fuzzer and it came out this summer it's what I worked on over while I was an intern But ultimately what you're going to do is develop your own so that you can write a paper about it and graduate all right so let's talk about the craft side okay I didn't fix that slide great uh generally here's a large wall of

advice about General Behavior your driver matters again if you knock on one door and there are 100 you're not going to find the bugs behind the other 99 you might need more than one depending on the configuration you want need to understand what makes the fuzzer tick to figure out how to optimize it the stuff you would do to optimize a grammar fuzzer which uses generational techniques doesn't work on a mutational fuzzer and vice versa that's why I didn't want to be prescriptive in this talk mutational fuzzers are the big business right now but that doesn't mean that's going to be the case in 10 years so generally whatever Powers exploration if you can only spend time

optimizing one area focus on that right if you is mutation focus on the mutation strategies turning all the ones on that will help you if you know dictionaries will be great for the Target you're working on spend the time to turn those on but those are like really sort of practice tuning and getting to know things in general if nothing else remember that making more things crash is good every fuzzer likes crashes the more stuff you can make crash the better off you'll be and that means tldr use as many sanitizers as possible all right how many people in here know what sanitization is I don't mean the garbage mankind yeah okay okay good I I talked to someone yesterday and

they're like I don't know what that is so here's the slide where we explain it so standardization is really about making more things crash if you're familiar with many languages um C in particular is the one we like to pick on because it was written to allow you to write an OS and in an OS you need to do a bunch of terrible things that you should never do in regular code so the language will let you do them because it was designed to write unic and that is what C is meant to do it is not meant for lots of other programs that we use it for it was meant for operating systems and all the terrible things they

must do so we have this buffer here who knows what an overread is okay so buffer overread is when you go past the end of the array and you're not supposed to and in C nothing will stop you from doing that because you must have meant to do it and it's not the place of the language to tell you what to do that's the philosophy see it's not my place to tell you what to do and all the F like oh tell me what to do don't let me do bad things too late anyway so we have this buffer and we could write off of it and we wouldn't know because like over reads don't always result in crashes you

see something you shouldn't but that does not necessarily mean you will result in a seg fault everyone generally buy that what address sanitization does in particular is put poisoned memory all around your buffer so if you try to read off of it it will crash your program immediately if you try to write to it it will crash your program so this is how you turn things that are not innately destructive like over reads into crashes and there's versions for the Heap there's versions for runtime there's a whole bunch of other ones but the idea is fundamentally the same try to make non-destructive Behavior as destructive as you can in the hopes that you'll notice that you did it got the

idea cool all right I'm talking about seeds in particular this was a request from a board member so hopefully they're happy with this uh because mutational fuzzing is the dominant Paradigm seeds matter a lot because they are the thing you use to mutate if your seeds suck your mutation's probably going to suck makes kind of sense you know that's a thing that that if that's powering the heart of your fuzzer you want to make sure your heart's got is good material as possible so this is a big area uh and you can choose to mutate areas of seeds you can mutate whole seeds you can do a lot of different things but they are good news for some targets and bad news

for others so most of the cool features are off by default in most fuzzers and you have to go in and go oh no I want to fuzz just this section because that's the actionable section or no mutate the whole thing go blow it up uh that's what I want so when you're working mutational fuzzer you're going to want to know what should you microtune and in general things that are very structured SQL XML those sorts of languages you're going to want to use like a dictionary or some sort of thing that will fuzz those keywords themselves I think in AFL the feature is called Auto tokens right because you can fuzz the grammar as well

kind of now the biggest area of contention within mutational fuzzing at least in the AFL world is what to do about the initial Corpus because that sort of algorithm starts with what happened in the prior iteration what did well let me go mutate it so what happens when you start and there's nothing in there you need something right and that's usually provided by the user and there are sort of two schools of thought about how much effort you should put into that and the first one which is sort of the conventional one you'll see in vulnerability research and you'll see it in some bits of the literature which is that it's very important and you

should make sure that that starting Corpus is as awesome as possible the other form which you see actually more on the internet and more dominant blog posts is that it's not worth the bother if you just run the fuzzer longer if I'm going to r that fuzzer for three weeks I don't care and a lot of that comes back to an experiment that the AFL author did whereby he just put in hello world left it running and walked away and came back the next day like hey I got valid JPEG files Isn't that cool uh this blog post is highly cited and I'm picking on this particular author because I got referred to it five times

and they have more than 80 years of security experience across all of them so this particular blog post about Linux colonel fuzzing was super popular but you can see how he's like oh yeah they're supposed to be important E I don't feel like dealing with it it worked for that guy let's go um so who's right who thinks the first guy is right the first Theory I see one or two hands all right who thinks the second theory is Right brave souls okay who thinks this is a trick question ah you are awake yes it's a trick question um so actually what my research showed initially is that they're both right just not all of

the time I have my theories and this is where I tell you that this is hardly airtight this is the end this is sort of an error of craft here's what my data says can't tell you that this is axiomatic it could be the structural level size topology or something I don't know about but in general the more structured your data but more that initial Corpus seems to matter um if you were fuzzing something like SQL you really want to put the effort in my SQL job ran for nearly a week and the computer might as well have been off for all it accomplished uh however the job with the valid Corpus made of nice lots

of inputs stolen from their unit test did great uh so if you look like SQL or XML you want to spend time on that initial Corpus on the other hand the less structured you are I think the closer to Binary data if you look like jpeg or Tiff if you run for more than 24 hours you don't really care it the results looked about the same so I would say that in general you don't lose anything by spending 30 minutes to make that initial Corpus and make it good but you could lose stuff right you can only gain you can't lose so at the very least you can be lazy and run strings on your binary and

pipe that into the input directory if you're working on an open source project somebody did the work for you go take their test cases copy move done you know like you don't have to spend like a ton of effort mentally crafting things if you're working on an open source project or something that already has a test Suite go take the test Suite or you know if that doesn't work for you because you're working on something super leite and secret or God forbid you have developers who didn't write any tests somewhere along the way uh run the fuzzy look for some art some strings grab some interesting magic values if nothing else grab the globals they're usually

useful okay all right integrating fuzzing in a software life cycle the hard part this is also known as fuzz left this is what Rowan's team is working on and hopefully having making inroads and intet but there are no free runes and there are no super bullets and this is one of the places where that's true uh you're going to need a lot of server time you need to pay for that server time and you're going to need to do it more than once you're also probably going to need more than one fuzzer couple reasons for that finding all the bugs is harder than finding just one different parts of the system usually need different fuzzers if you're

fuzzing a binary you're going to want something different than if you're fuzzing a web client I mentioned that we fuzzers have specialty you need the right tool for the right job but the other reason is different fuzzers will find different bugs even on the same fuzzy on the same victim so we know that if you were really trying to thoroughly fuzz something you're could probably running like three four different fuzzers on the same thing which your Ops people may not want to hear if they're in charge of the bill also you're probably going to need to fuzz for a lot longer than you think you do a lot of blog toes talk about finding stuff after five six hours in

practice you probably need more than 24 hours you probably need closer to a week uh perhaps on the alpha branch and your adversary all the VR people I know just like leave their fuzzers running for weeks at a time so if that's what your adversaries are doing even if Ops does not like the bill you should think about doing it too they're also usually running more than one and you may one of the ways you can sort of save the bill a little bit is to rein reuse your seeds if you're using something like AFL there's always an output Corpus at the end here was the state of my fuzzing queue when I exited you can feed that yesterday's job into

today's job and try to sort of chain things so you're not revisiting the same thing all the time um it can take a little bit of like scripting to do that but I believe that's within everyone's skill set and don't underestimate the amount of time you're going to need for analysis triage and messaging with your Dev team if they are not understood they will not be fixed uh I will tell you that inable reports get ignored in my case if I got one of those it would then go into the bucket for later analysis which usually meant summer intern projects which if the bug is filed into December is not great so one of the things you need to

know if you own this side of the thing is communicating with the developer team about what it's going to do um and don't underestimate how much time and energy that takes all right final slide if anyone is brave enough to attempt the lab here are the links you select AFL it's going to use Docker so if you're professors and the people you're talking to for interviews are like I love Docker we love to see Docker experience get some Docker experience uh you should have basic introduction to systems or at least some familiarity with C assembly that sort of end of the world you can do them locally or on gcp I mentioned gcp in the the lecture

description it's about 30 minutes a lab I've never tried this on windows so if you have Windows questions don't ask me and and I can be found here my email is knock at pdx.edu I'm on the bsides Discord although we didn't seem to use that this year they listed my LinkedIn but I don't really use LinkedIn except when students need connections so that's not a great way to get my attention but email email us generally email is a great place to get me uh this slide will be uploaded at the talks directory and I guess the end of conference but if you are do vulnerability research I want to hear about what's hard to fuzz for

what's useful but non-destructive you know do you need Heap leaks do you need to you know what do you need as someone who does research it's nice to know that I will be doing something that people will actually use and find problems they care about likewise you know the VR Community complains we don't listen come tell me I will listen all right and that is the end of the [Applause] talk no way that one not mine uh which means we enter the scariest part of the talk for me which is open floor questions what do you got all right one in the front yeah so um let's say that like more engineering manager trying Buzz left kind of a person and I I'm

really curious about what your thoughts are on like strategy around Corpus right so like is it better to like spend a lot of time on a corpus and like sort of encode my prec consumptions about where my software is broken into that or like is that giving myself a blind like a blind spot and what the f could be searching for okay so the question is as an engineering manager if I put a lot into the work of the Corpus and try to pointed at areas that uh might be dangerous am I biasing the fuzzer in a bad way or a good way right uh well a very prominent vulnerability researcher who does not like to be named says that

a good harness and good seeds is how she blows things up uh she may or may not be mean that literally given who she works for uh and so what she usually does is she goes in the coach does reverse engineer and she identifies areas that she thinks look bad or fuzzy uh you know pardon the term the parsers in particular have been a long Target of fuzzing because they're often handc and contain bugs and so she'll identify that grab things from that area and she has great success with it so I'd say if she's doing that you should do that um because the adversaries will use it against you you're welcome and again if you're

running for a very long time I think you need to worry less about biasing the fuzzer towards areas and making sure it's got enough ammo to get to all the areas other questions I saw some other hands all right vmf looks really cool why should I use it VMS looks really cool why should I use it well VMS purpose is to be a laboratory fuzzer it's meant to test new ideas it's meant to be modular so it has some of the same goals of lib AFL of porting things unlike lib AFL it wasn't tied to AFL itself and so it's got some cleaner boundary lines I have had students spin up on vmf in 30 minutes and I can't say

I've ever had anyone spin up on afl's internals in 30 minutes uh Emily spent a lot of time making sure that it was well encapsulated and well well engineered and liba AFL and AFL have been phenomenal successes but they were someone's side project so not quite the same as like we're going to take an architect with 30 years experience and ask them to design the right thing so BMF is new it's kind of a baby fuzzer but it's a great laboratory so if you're like I want all the power cuz I want to go bug bug bounties bmf's probably not right but if you're like I would like to test my new ideas and be certain that it's actually

my idea and not comp log doing all the work vmf is better so the the targets that I had classified as semi structured were kind of in the middle you got some gain from using an initial Corpus that you actually pulled and filled with valid seeds but it wasn't as dramatic as like SQL or XML things that are highly structured so human structured or like half what call semi- structured you do get some gain like I don't think it's you know 30 minutes is not going to cost you a ton but your gains are not going to be as dramatic if as it were something like SQL so it's still a good idea for you

but it's not like you must do this the way it kind of is for SQL and XML make sense yes okay other questions or would you all like to go go go to closing ceremonies all right well thank you for being a great audience I don't think anyone fell asleep I'll see you all closing ceremonies [Music]