Fuzz Testing for Fun and Profit

all right hello everybody um i hope you all have had a wonderful time so far and uh to i hope you can hear me uh to kick things off i'm going to be introducing melissa here uh so melissa has worked in nearly every software development role dev tests and devops and program management at companies big and small and somewhere in between she's created and run high availability high quality services for playfab bing cortana and xbox one and now for mparticle's enormous data platform melissa discovered her love of massively scaled systems while growing the big the bing back end where she honed the art of keeping highly available complex systems up while undergoing significant code churn

now a director of engineering with mparticle she's passionate not only about maximizing efficiency both in her product code and in her developer tools but also about sharing the best practices among colleagues in the tech world at large now a little bit about melissa's presentation on uh fuzz testing for fun at profit a software test is no better than the data that drives it fuzz testing is a great way to find buggy exploitable or otherwise bad code and if you're working with a native application that operates on file input it's a solved problem grab afl or some other all-in-one suite hit go and profit but what about web services what about managed applications for a service based developer

fuzz testing your http serving a layer isn't all that interesting if you're running on the latest version of apache fuzz testing your application along its interface boundaries though very interesting and not a problem that can be solved with an all-in-one suite off the shelf in this technical talk melissa will demonstrate not just about what fuzz is testing but how to use it to test the service architecture at every level and interface as a key part of any continuous delivery strategy using real-life examples and experience she will demonstrate how easy it can be to integrate both mutation generation and generation buzzing into an existing test strategy for services that operate at any scale from micro to global every

services developer should be able to feel confident not only that they've sanitized their inputs but that they've been validated all right melissa are you ready to go i am ready awesome all right stage is all yours great thanks aiden uh so hi everybody my name is melissa benua and we're here to talk today about about a little bit of fuzzing uh fuzzing is fun a word of of warning ahead of time it's a very windstormy day so hopefully we don't run in any issues or flickers uh but bear with me should be fine you may or may not also see some background cast of characters coming behind me please pretend they're not there i try

my best to uh but they are very cute um so anyway we're here to talk about fuzz testing and i look like i'm doing something weird i have a clicker in my hand because it makes me feel like i can pretend that we're all in person together even though uh we all got we all know there's zoom going on uh so who am i so i am currently a director of engineering with a startup called on particle i run a lot of i run a number of teams ranging from back end engineers to uh test automation slash uh dev test sec ops um before i was doing that i was doing um doing those things directly myself as

an ic um and so i've got a lot of experience doing doing uh coding and engineering in general where various boundaries intersect so being a developer but also with a strong test background uh doing doing test work but with a strong security background um you get the feeling so what i'm going to talk to you today is is about testing but it's also about security and it's also about writing efficient code and it's something that i first became really passionate about when i was working as an ass debt so i started my career at a very very large west coast company that uh had s debts back that back in the day on a very very large web service that

i'm sure many of you will figure out before the end of the talk exactly what i'm talking about uh but so years ago i was working on the service i was in estat and one of the very first things that was handed to me when i joined the team was this thing called the forking service i'm going to be very careful when i say the name the forking service because we often called it other names that i'm sure you can extrapolate about especially when it did things like break people's test environments and so what the forking service was it's um sometimes otherwise known as t traffic but basically it was something that took real production data sanitized it a

little bit right anonymized some of the identifiers um etc and then made it available for replay at a certain percentage into any of our test environments so you could run a production-like set of traffic against your parallel prod environment for example or you had a debugging enabled or against your release candidates um or against a highly tuned performance environment so you could look for you know performance issues um but anyway so it would take a small percentage of real traffic um it would send it you know sends it to to our test pads and everything was great uh it was used uh to validate any release again like you had to use it to validate a release candidate any build

had to sit for 24 to 48 hours without any major crashes or memory leaks or weird things and then at that point it could be considered staple enough for production it was a pretty important service everybody relied on everybody knew what it was more or less it was pretty stable um but you know it wasn't perfect and we would still miss issues and so as a as a young and aggressive asset i said great i own the service and everybody knows that this large west coast company the best way to get promoted is to own and or make major updates to a testing service uh so i said how can we make this better how can

we drive out a whole new class of issues that basically are being untouched or you know very lightly covered right now and so i went to a couple of internal security talks they were talking about in particular this fuzzing library this native code fuzzing library that they had published internally for people people to use i had a bunch of attacks it was pretty cool um i was like oh this is interesting and i proposed to my manager um i want to see if i can integrate some of this some security testing into the forking service with maybe maybe a little fuzzing and he said sure i guess whatever i didn't really know what i was talking

about but he gave me a couple you know a couple a couple weeks and said go to town so i took this library and i took the f the forking service is the forking services existing model of what our traffic looked like because it had to have some basic model in order to do any sanitization so i combined this this library with our model and i created this thing called the fuzzing service and it was really just a layer that sat on top of the forking service um but it it just made the traffic a little more interesting so if you had you know if you're losing an http request and you had something to you know foo dot com

slash uh slash search uh in the quarry param line you had foo equals bar uh it would take your bar out and insert like a certain alert box uh you know alert i got you um and so i did this thing i i got it ready to go it got pr by my piers and uh finally we hit the day where we were ready ready to turn it on um nobody had been paying that much attention to what i was doing but we did have a big release coming up in the next couple of days and so um i sent an email out to my team and i said hey guys just so you know um make

pushing out an update to the forking service you might see some changes in your traffic i don't think anything terrible will happen famous last words i don't think anything terrible will happen but it'll probably be fine let me know if you run into any issues and i can make a change and i um sent the email i hit push right hit the deploy button on my on the on the new version of the working service and then i got up walked away from my desk i went to get a coffee and as i was walking back to my desk and getting a coffee um i could hear variations of people talking about that forking service the beds are forked

somebody help the beds are forked we can't validate anything what is going on i'm trying to test my code fork me it's not working uh you get the idea uh so i hear all these people and by the time i sat down and unlocked my my computer i had like five or six items for people saying hey whatever you did stop turn it off you're blocking validation for the release um you need to stop immediately so we can validate and send stuff out and i said oh you know oh maybe i must have you know sent too much traffic or whatever um so i turned it off and as we start investigating what's happened here

um bad things have happened but not bad things because i did bad things bad things because it turned out that our stable code that had been sitting under you know the release candidates have been sitting for a couple days by now in the release candidate environment getting ready to go out uh had immediately cratered to the ground with a bunch of different crashing issues and every time it tried to came come back up it immediately hit a crash inquiry again cratered to the ground um and so everybody went really quiet uh and all the dabs went off and scrambled to fix fix the bugs they fixed like you know three or four crashing bugs that day

pushed it up live and then they came to me and said hey whatever you did the other day can you turn that back on again please and so i did i turned it on and then i hear i hear more fork fork as the beds go down again and then whoops there were four more crashing issues so they go back i turn the traffic off again and then they frantically fix for more traffic issues um and then we repeat the cycle a couple of times until finally at least the bed will stabilize and i can leave the fork and leave the forked traffic up um there were there were more issues we started once we got through the p0 oh my

gosh crashing issues there were other interesting issues we'll talk about how we found those later um but there was a lot of scrambling and fixing on code that had been quote unquote stable and ready for release and the really interesting part was not all of the bugs were in net new code a bunch of them were in code that had been in production for you know three months six months in some cases a couple years uh they have been you know every now and then we would see these stack traces but you see it once every four months and so it's never a priority right one crash every four months that's not hard that's hard to reproduce whatever people don't

care um but all of a sudden when we could reproduce it quickly and easily they cared a lot fixed it immediately and we had so many interesting bugs they were from issues in the c plus web service to javascript vulnerabilities rendering we had crashes and memory leaks and deadlocks we found all these issues just by making our traffic look super super super weird um we had things that were easy to find you just pulled up the stack you pulled up the core dump combined with the traffic uh the traffic log we made things that were really hard to find one including a cross-process memory contamination that luckily never went out into production right that's horrifying you can get uh

if you leak somebody else's personal data with a specific set of queries you know thank goodness that never made it live so how did we do this this is what we're going to talk about today first i want to lay down some pretty basic definitions because software as an industry has definition problems um when i say something like we're going to do integration testing it probably doesn't mean in your head the same thing that it means in mine uh so i want to lay down some very basic definitions here uh starting with yeah you're going to see the dog keep going back and forth uh starting with fuzz testing so fuzz testing if we use a

more academic definition is just generating random input to explore the balance of our system under test pretty simple and so when i talk about an attack i'm going to talk about the specific data in that input meant to drive out a specific class of bug for example a sql injection attack and then we're going to talk about manipulations which are the combination of your input combined with an attack and of the three manipulation types we're going to talk about today um the simplest one is a creation manipulation which basically means substituting in a de novo brand new generated input from an attack so imagine replacing your entire query string with an alert box that would be a creation manipulation

whereas an assertion manipulation is modifying a known input to include an attack so imagine if you had a search thing and you said you know q equals ponies would then be you know q equals p o semicolon alert box and ies um and finally we have a genetic manipulation which we're not going to talk about too much today but which the really cool fuzzers especially the local ones do and that's successively modifying your input based on system behavior it generally requires white box buzzing which means you need um code coverage or some way of of the for the program to see what branches it has explored and what branches it hasn't in order for it to

intelligently continue trying to fork its way all the way down to the end of your to every every branch in your logic tree so these are the definitions we're going to use today and here's here's the the keys of what how to do this in a good and efficient way right because it's great to have a good set of attack sets it's fantastic um and it's really good to know your model right if you the more you know about your model the more intelligently you can apply those attack sets but the important keyword there is intelligently because in order to be really efficient with your time and resources you need a plan of how you're going to apply those

attack sets to your model because not everything is important to fuzz um especially if you're in something like a cicd world where you don't have three or four days of stabilization on a six-month release or whatever you want that code to go out immediately you wanted to go out as fast as possible um so we want to find the balance in between yeah if we ran for five days we would eventually explore you know we'd cover 99 of cases um get something closer to if we ran for 15 minutes we'll explore you know 97 percent of cases and that's good enough um because you know there's going to be things you care about and things you don't right your

model in particular here is the things that you actually care about right you probably if you're running a web service you probably don't care what's happening at the tcp layer because your web server does that for you um right you don't necessarily care about fuzzing your web server i'm not saying nobody should fuzz a web server because somebody is closing a web server but probably that should be the people who publish the web server what you care about is your application um more specifically right if you're only responsible for a part of the application you really just want to validate the part of the application for which you're responsible for there's not a lot of value in testing

your you know your off system if what you're assigned to do is testing your commerce part of your website um a little bit of you know incidental testing of the auth system is fine but what you really want to be hammering on is the best shopping pages and the cart pages and the checkout pages um you want to be able to do the things that are important to you right because that's what we get paid for we get paid to do our jobs

so let's talk about what data models are so here's where we start we're going to start with the model and if we're talking about web services in particular we are probably very interested at the http level and so this is where you know this is the foundation this is the raw a raw http request it's a get request it goes to some um some totally random website um but what we want to do here is not it's not just random strings here this actually has a model right http request has a format that it needs to look like otherwise it's going to get thrown out by your web server and it's not going to get very interesting

um i'm not particularly i mean i'm interested in what apache is doing but i'm extra interested in what my application is doing on top of that um so right i need to know that this web service has right has a model has it has to have a uri the uri has a specific format uri has a query parameters the query parameters having a different format layered on top of that um that format is separate from headers which always have to look a certain way or they'll be rejected um even within headers right cookie the cookie header has a specific format on top of that right the cookie format is um ampersands and equal signs separated by semicolons

which is different necessarily than like an accept header um and so knowing the data model of all of these things um is what's going to let you be really really effective right if you look at json it's the same deal json based if you have a json based http service what you care about is probably most of what's happening in your post body um maybe your application doesn't use courier parameters at all if your application is never reading off the query parameters and you configure your fuzzing run to spend 30 of the time buzzing query parameters that time is wasted time not because nobody should ever test a query parameter ever but because that's not really relevant to you

so again right if we're looking at a json based http request parts of the model still apply right obviously http requests are still requests they still have methods and uris and versions and headers but what you may care about would be the json post body which has a specific format both at the http request level and then json itself has a format right you have to have your braces and your quotes and everything in the right place json has the concept of objects right so if you have if you have arrays they should be valid arrays um and so the more you can respect your model the deeper you're going to get in your fuzzing attempt

right if you never ever send a valid json array because your fuzzer doesn't understand anything about it then you're never going to do anything interesting like fuzzing say a timestamp within there fuzzing if you can't figure out what a json object looks like you may not be able to intelligently fuzz a customer id or other things that may be surfaced in an interesting place in your application

so let's jump ahead to attacks and this is not an exhaustive list this is not meant to be exhaustive um but i sort of want to give a high level overview um so attacks can be anything from very very simple to extremely complex custom custom handcrafted manipu exploits i personally get a lot of distance out of uh out of a's out of random strings you'd be surprised at how much how far you can go with you know with a couple of a's and i don't mean like three or four a's or ten a's like they're like uh uh are on the slide but i mean 10 24 a's 10 23 a's 40 96 a's uh what happens if you append a

kilobyte of a's five kilobytes of a's into your query line that's interesting um you don't know how many times i've found memory leaks um in places that things weren't being exposed by stuffing so many a's in there that it's really hard to miss when the a's inevitably uh show up in other places um but of course you have you know you've got your sql injection you've got control characters which i'm always amazed that powerpoint doesn't crash every time i show these because uh it was it took probably 20 minutes just to get the control characters um and some arabic pasted in uh you can find xss vulnerabilities you can do integer overflow you can do

redirects etc etc i'm sure you get the point um something to keep in mind though is if you are looking to invent all of your attacks by hand from scratch please stop uh this is this is not uh probably the best use of your time and i'm not saying you should never custom craft any of your attacks but i understand that there are a lot of them that are already published um and collected i personally like looking at buzzdb though i know it's getting a little aged um but you can collect existing exploits that have been used existing expositive use to take advantage of real vulnerabilities and just maintain that in your list instead and that will be a lot

simpler for you than trying to invent everything all on your own there's no need it's a solved problem so if we consider that we have a set of attacks and we have some models let's look at what happens if we apply these things without any intelligence and we'll hopefully look at why why a plan is important uh so this fuzz request is it's super fuzzed definitely um but it's not going to be useful and why isn't going to be useful um because it's never in a million years going to get past our web server and i'm again i'm not saying never fuzz your web server but if you're trying to test your actual application and your code in your application it's

not going to be super useful and why is that so if we look at our our method right our method is invalid and even if we got past our method our our host name is invalid um and even if we got past the host name a bunch of our headers are also invalid so um right we don't have a cookie header so if we're trying to be interested in something that's in the cookie header it's never gonna get anywhere um and right if we're interested in working on a specific path of our web service and we've completely fuzzed the path then we're not even going to get there and so what i'm saying is you need to be

very deliberate and careful about what you're fuzzing in order especially if you're doing black box closing because the system isn't going to be able to adjust itself to get to the areas that are interesting you're going to have to provide that intelligence for it to find what is interesting to you because it's going to vary from run to run from team to team from person to person on the same time if you fuzz too many things it's going to be a wasted effort right because you're not going to explore any interesting paths you're going to have a hard time finding a repo of what exactly caused the problem and it's going to be very much more

difficult to make your data fit into a model that the system will accept on the other side if you buzz not enough things you're going to be wasting time right if you have only one fifth of your requests fuzzed in a way that's that's applicable or if you're only ever fuzzing the q equals in your query parameter and you're in your query line and nothing else it's going to take you an awful long time to find something interesting or you may have excluded areas that you didn't think of at the time that might still be valid for testing

so what we need to do is to make a plan and this is uh pseudoxml i guess it's xml-ish um but you need to make a plan to figure out what you care about so if you never care about fuzzing fuzzing your http method because your web service only ever accepts a get and you're confident that that part has already been tested then you probably should be testing you're testing your method very little you know maybe five percent if you're not confident it's been covered maybe zero percent if you're like nah we got this i don't care i don't care about the method in my http uh in my http request um you should figure out what attack sets

are valuable to you right in depending on your application so if you're a hundred percent json and you don't have xml anywhere in your system you don't have any code to parse xml might not be super useful to you to have a whole bunch of xml based exploit xml based attacks if you never ever ever use the cookie header might not be super useful to you to fuzz the cookie header on the flip side if your application 100 requires authorization every time and you're going to throw away every request without authorization um then you may not want to fuzz your authorization header too often um you may want to fuzz it a little bit to ensure that you've got coverage on

the parsing of your authorization header but realistically if you're going to throw away 100 of traffic if it doesn't have a valid off header then you're just going to throw away 100 of your traffic and again you're never going to explore any of the interesting branches in your application so let's go back and look at our http request again so this one is uh is a very effective fuzzing please use your imagination imagine there is like 50 times more a's in that query line because that's not very interesting um right we've inserted some some parse characters there's a percent 20 in there um and we have intelligently fuzzed the cookie header such that if you go and look look down

through it um you'll see in this specific mood cookie we've got we're injecting um we're injecting to see if we get an alert box popped up that's interesting especially if you know your application you know ah yeah that you that's a user id we use it all over the place sometimes we echo it on the page or sometimes we use it in a lot of our javascript code um you're likely to find something this way or else to ensure that you haven't found something this way right this is interesting so what we've done is go from that you know terrible http request that's not gonna it's not gonna find anything nothing useful is gonna come here it's going to

get thrown away and we've instead started exploring the actually interesting bounds of our application here's another example of an also good http request you can see here that we didn't touch actually any parts uh we didn't touch the method we didn't touch our authorization we didn't touch any of those pieces but what we've got is intelligently said ah this is a post body and more specifically this post body is json json has a format let's respect that format and it's gone and found that ah our timestamp is a number let's put in a different number a bigger number see what happens likewise you can see down in the customer id field it that we've fuzzed within the bounds of

valid json again right you fuzzed inside the two quotes and it's been smart enough to know ah we need to escape back we need to escape single quotes because all of our json has double quotes and so this is totally perfectly valid valid http request valid body but it may make something very interesting happen and so this is what happens when you apply a plan you go from randomized chaos that may be one in a million will turn up something interesting to still randomize chaos but something where maybe one in ten and so if you can imagine that you're doing say 10 20 30 requests per second when you're testing your service one in ten is going to turn up an issue

very quickly um and one in a million might be one of those sporadic crashing issues that is hard to debug and everyone's like ah it's low priority we never see it and it can be the difference between getting a very serious vulnerability identified and fixed and having one just live for years forever because nobody views it as very high priority

so let's talk about uh how we found something maybe change tracks a little so in some cases it's very very easy to know you found something in some cases it can be a little harder but you're going to want to use your harness your powers of observability just like you would with any other production issue or investigating investigating any other type of bug the most important thing is going to be understanding what normal looks like for your application so that you can spot if something is abnormal when i was doing fuzzing i found it always very useful to compare the baseline of say a production environment or an environment that wasn't under fuzz traffic against one that was

especially if one had forked traffic and then one had fuzz traffic uh because then that made it very clear that these were net new um or something specifically driven out by fuzzing um so if you start looking at your counters right not just your set of exceptions or crash jumps which of course are an easy place to go um but go look at your memory are you leaking memory is your memory allocation you know is it is it running slow and steady like this or are you constantly uh constantly trending up to the sky um uh the other thing you may want to take advantage of is side effects especially if you're looking for sql vulnerabilities or xss or csrf or

code execution or something um you may be able to find it with metrics um but it's going to be a lot easier if you have a side effect for example popping up an alert box may not work if you're running a disjoint headless task but if you have instead of hopping an alert box you do you know make your javascript increment a counter right if you set a public counter up on your on your testing you'll make your javascript income and a counter you can check if that counter has been incremented and then say oh i've got something same thing right if you've got a if you've got a if you've got your test sql table looked

up and you've got a function that would just just write something to your table ah then you know you found something um finding things that are more complicated than crash dumps are are is difficult it's a little bit of an art um it usually requires experience in knowing what your application does and what it can do um you can also use the concept of an oracle and an oracle will help you take those judgment calls that you might make you know personally as a single person and scale that out to make automated judgment calls so it can say ah i know that if my error rate and my fuzz bed goes over this amount i need to fire an alert

or here's my my threshold for memory and if i go above this memory um you know set of allocated bytes i need to fire an alert um so an oracle can look at what's happening in your environment whether that's from a simple alert threshold for your learning system whether that's a separate application you run periodically to check you know to call a bunch of you know maybe calls a bunch of databases and checks to see what's going on in there to see queries for specific data sets the oracle can help you take some of the load off of investigating and keep your at least take the low hanging fruit off so that you can look from spend time

looking for the more interesting or the more esoteric things so let's talk about what you can actually find so you can find crashes of course especially if you're in an unmanaged system we used to find c plus crash dumps all day and all night it was really easy if you're an imagined system we generally um we had metrics on our top level exception count so we could find how many exceptions had been thrown what you know what the exception what the exception type was etc those are you know those are very common those are easy um oh no we've got you know all of a sudden we've caught a hundred exceptions today under fuzz traffic under the new change

and we didn't catch any yesterday somebody probably introduced something bad oh no the service is crater and we can't bring it back up somebody probably introduced something bad um as i've mentioned before you can find memory leaks you can find thread contamination um you can find places whoops missing a null terminator on your string you can find places where you're allocating very messily and not deallocating um all those things that would be very difficult to find with traffic that seems sane right for because the the biggest advantage of fuzzing and um the reason i tend to go to it as you know part test tool and part security tool is our human brains are limited by our

human imaginations if you ask me to sit down and look at a web service and say go test that i'll probably come up with 20 or 30 interesting test cases if i'm really really experienced and comfortable maybe i'll come up with 50 or 100 um but they're gonna start getting repetitive because there's limits to our human imagination in a given point in time um when you start rolling in like t traffic forked traffic um then then it gets a little more interesting right because then you have the collective imaginations of all your customers and the people that you're serving um and those are great too but again they're bounded by human imagination when you start programmatically applying

um combinations meanwhile via a via a fuzzer you are no longer bounded by those by those limitations and you will get patterns that you never would have thought to do together because your brain instinctively rejected oh that's unreasonable that won't work um uh whatever and so that's how you can end up with a lot of things found that you never would have found otherwise

of course you can find a lot of exploits especially depending on what exploits you've rolled in so i've got some requests for some examples and i have good news for you we're going to play we're going to play with some examples now um so everything that i'm showing you here today is available on the public github repo it's just open source license it's no big deal um it is in.net core i do not promise you that it is a production ready system but it is certainly a fun proof of concept you're welcome of course to clone it modify it make issues make a pull request et cetera et cetera et cetera standard open source and disclaimers

apply so don't mind me while i pretend that i can see this um so here's here's some code um the code lives here it is dependent on fuzz db we can look at some of these um vulnerabilities in a few minutes but basically what it does is it wraps both fuzz db fuzz db attacks with a couple of other the ability to have custom attacks and then does some some cleverness to apply either generational or insertion manipulations apply that with the combination of a model and a strategy to do something really interesting so let's take a look we're gonna hit some random website on the internet no big deal um so i've asked it to

generate a couple of http requests this should look it should look pretty similar pretty familiar um a couple of interesting things to note a dot net thing is that it's breaking up the accept encoding header which is comma separated into it's pretending that they're three they're not uh same thing with accept language all the except headers do that so just keep that in mind that if they seem to be fuzzed three times it's not it's a display issue because of.net so again here's a bunch of requests there's five of them they all look exactly the same and now we're going to make them very interesting they became interesting oh look at all those a's so here let's take a look at this so

you'll see none of them are fuzzed outrageously um in this case we've got two attacks applied here in the query line we fuzzed the path you can see we've got an insertion manipulation we've inserted a whole bunch of things into the path likewise we've inserted similar things into the middle of the query line but you see it didn't break the format of query line here we've got the query line is fine the path is fine but we have gone and put so many a's inside the cookie and not just inside the cookie oh no it's the accept header we've replaced the name of the content type header with many many many a's but that's all we did uh here's another

one and oh man we put exciting things just in the middle of our query parameters format's still valid um here we did some a different attack same concept and here it looks like we've got some uh some xss injection going on again in in the query line so we can send these requests see what happens pretty interesting they were all accepted except one that was not found which is not unexpected because we fuzzed the um we fuzzed the path everything else was accepted that's interesting a couple things to note here is if you're doing automated fuzzing you need to make sure that you uh have predictability so here we use a random seed so that we can always reproduce whatever

insanity it was that we generated from run to run to run if i change this random seed i don't i don't know what's going to happen here something else will happen so here's those same looking requests look at all those a's [Laughter] that's great there's so many um here we fuzz them again this one doesn't look super fast i don't see what happened to it could have been nothing could have been one thing that i'm having a hard time picking out here we've got some more xss [Laughter] did something really bad with the cash control header that'll be interesting i'm really interested to see what the server does with that um anyways you can see ah see if we can run

some pearl sure why not so we execute these again come on ah our a's generated a bad request that's interesting we got a 500 level error this time from sending a whole bunch of data um the other against other ones again we're all accepted and i'm not going to send a lot more of these because i'm going to get flagged as a crazy bot um but you can see interesting things happen here and so the next thing i would do right is i would go and check my observability go and look and see do all my metrics look normal do i have any side effects that i've triggered do i have any crashes that 500 error

especially could have could have generated a crash i'm pretty sure it didn't because i know the service that i'm testing but could have generated a crash any time that you get a 5xx uh is suspicious um and so so there's a lot of interesting things that could have happened and this was just 10 requests so imagine that you're generating 10 requests like these every second and you're running for an hour imagine imagine you're generating 50 requests like these and you're running for an hour even an experienced security tester isn't going to come up with all these possible combinations that your buzzer can automatically handle for you

and so that's what meant that's what that's why this is important so here we've got um you can see we've got a model we've got strategy i'm just using the defaults that are included we could change the probability of fuzzing the query parameter if i change this to zero percent god willing i didn't write a terrible bug um we should see no query parameters get fuzzed uh yeah so this one didn't get fuzzed this one didn't get fuzzed other interesting things happens instead look at all these a's just get right past them um you seen in no point because my model i said to my model i don't care about i don't care with chore parameter i

don't care about any query parameters i don't care about the whole query line just ignore it test the things that actually matter to me and it said okay again we triggered another bad request probably the same one we triggered earlier with all those a's and the header but by having control control this way you can adjust to the scenarios that matter to you at any given point in time if i really wanted to fuzz today i wanted to really explore what my web server was doing i couldn't i could make it go a whole ham on web server things if i wanted to say i don't care i know that my web server rejects bad methods i don't care turn it off

fuzz only the things that matter to you fuzz them for a long time buzz them at scale um and see the you see the difference

so the the point here is that you can fuzz what is important to you and ignore or min you know or have it be a knock on effect the things that are not important to you you're generating so much garbage looking traffic so fast that it doesn't really matter if you're all curious we can look at some of the actual attacks that's being sent so here's some of those hex values we start with stuffing in there um let's see something interesting some no sequel ejection attacks into these are all and these are all this is from fuzzdb i didn't invent these but these are all attacks that have been used to exploit an actual vulnerability

so they are they are real all right some path traversal best reversal with bizarre encodings i'm sure you saw at least a few of these come by in our example some exciting sql injections a lot of xss i kind of get the idea i hope um so yeah a couple other resources for you i've got some some references and further reading if you really want to go deep i recommend using fuzzingbook.org um is a as a way to guide yourself through it uses python it helps you craft your own in-depth fuzzing runs um a lot of their concept they may use slightly different terminology than i do but they have a lot of the concepts are

very similar afl is great if it works for your use cases if you can just use afl then you should absolutely 100 just use afl if you can't because it doesn't work for your scenario as it does you know because there's a lot of places where it won't then you might want to use something else like the service that i've built here um fuzz db i've got a link to the fuzz db project which i love very much and i hope very very very much that they continue to maintain it microsoft has recently published uh an open source hdb fuzzing tool called wrestler i haven't personally used it yet but i have a couple of colleagues who have who

are really excited about the possibilities and of course kali linux has a bunch of various fuzzing tools built in if you just want to grab something off the shelf otherwise i had a link up in the beginning i'll post it later on in in the channel to my website which has slides um and examples if you go to my github repo it also links two slides and examples queen of code slash dot net model fuzzing a little more legible um this has links to um the slides as well and the fuzz db and to fuzzing book etc usually usually the easiest place to get all that is from the github repo though which i will also post in the discord

um that said thank you all for bearing with me today on this very windy saturday morning and i would love to answer questions or show off some more fuzzing runs i get a lot of joy out of just watching the buzzing runs and seeing what it does um it's very entertaining can i kick off with like a really light question yeah um do you always laugh maniacally when you see the a's does that mean i do actually i actually do because it gives me so much joy to see them because i found so many ridiculous bugs that was how i spent um i once spent a solid eight or nine hours trying to drive out that um

cross-processed red contamination issue and what we did it with it was so many a's because then it would how we could finally trigger enough we could rewrite enough of the memory buffer that we could see it show up in weird places it is my literally my favorite way of driving out some of those vulnerabilities as as other people maybe formulate some questions to throw into the chat or unmute themselves and say i'm out loud i do have one so in terms of in terms of a fuzzing plan mm-hmm um what do you how do you suggest how do you approach what are your thoughts on stuff that's not yours but really important to your security

aka i think of my hype for my question um third-party components open source things stuff you would consider parties your supply chain that's important to make your your service work or your deliver your product yeah so there's really two two types of fuzzing runs one is a steady state um sort of baseline maintaining one and the other is like as a phishing expedition looking for something something specific so if we're talking about your steady state buzzing run you should keep all of those so if you look at my plan here my example plan oh my gosh i have so many things open um definition um the default strategy here has some things at a high probability and some

things at a very low probability so those things that are not your responsibility that are important rather than setting the probability to zero set it small so that you don't never hit it but you spend a lot more time on the things you care about realistically speaking if one out of a thousand of my requests is attempting to you know to fuzz something that's low level in my http service and i'm running again i'm running 60 requests per second it's not going to take me very long to find that issue but i will be much more likely to find the issues i actually care about so it's a matter of balancing the time and how long you're going to do the run

if you're doing it 24 7 um you don't need a super high probability in order to drive that stuff out um whereas if you're on a phishing expedition for something specific in your application um you only want to run it for five minutes and you're limited in the traffic you can send you may want to say i'll get to that other stuff later i really just want to dig at the specific class of issues or the specific chunk of the code so it's about min maxing your time specifically um and your your your human effort because that that part's the expensive part machine time's really cheap thank you yeah i like that and into the

way you answered it really changed my perspective on it too so i appreciate that yeah no no problem anyone else have any questions any other questions i don't try to see if i missed anything i actually i threw up two of them but that was a pretty good answer that first one that was pretty good my other question wasn't more we dig into that what the idea of an oracle is a little bit how to get there like just expand on that just for a minute if you don't mind it was a really cool concept yeah and it's uh it's a concept we could talk about for kind of for for a really long time

it's almost almost writing a good oracle is almost an hour talk all by itself um so the very simplest oracles most of us are probably familiar with are your alerts right things that trigger you know that call you on page or duty if we're talking about data dog terminology right you got your graph and you've got your threshold and when you're when your data point goes above the threshold triggers an alert that's the simplest possible oracle it says if value greater than x ah something bad happens um and that's you know the kind of simple orbital you can get out of your observability tools um but you if you're willing to write you know write some code or write

some scripts you can get much more complex things you can say things like if my memory threshold exceeds this value and my traffic account is like this and my exception counts are like that then something bad happened um is a slightly more complex one and then you can get to oracles that are very application specific you can say things like if i'm writing to this ta you know if i go and scan this table for things that were written to it and for in the last five minutes and i see that i'm getting something that looks like x then i know something bad happened because i wrote my javascript to write to this table if i triggered an

xss and so your oracles can get very uh very specific to your application um basically think of anything you would do to check if something bad happened if you can do it programmatically it's an oracle yeah i hear hearing you say that out loud makes it sound a little better like even in my question all right it makes me understand more like even my question my ques was more about where do i get one and that makes no sense at all now that i hear you explain it it's it's an oracle for your thing so for your thing yeah yeah it's not like you can uh build on something or it's more of a way

you track those things exactly and so you can get some of them off the shelf right like most observability tools will come with some metrics basically out of the board um you know oh no your cpu's been pegged at 100 for five minutes alert alert um but you know they're probably not going to be all that useful to you they're better than nothing but they're not exactly highly tuned and customized to you know what you expect your steady state to be and what does what's what does a deviation from that look like and when is it bad to deviate

all right if anyone else has questions you could always go to the discord uh there's uh you can do a talk in the lobby there's a text chat in the lobby and there's also a voice chat called talks q a where you can continue the talks q a um but yeah i mean i just a thousand times thank you yeah no problem my pleasure sharing this information i think anyone who hears that is definitely smarter for it i don't know what else to say is i'm trying to find the biggest compliment i can give wonderful glad to be here all right um our host aiden i guess you can end the recording please all right