What the Fuzz

hello everybody good morning um so welcome to uh what the fuzz uh introducing fuzz testing by myself Josh Grant um oh it works oh this is going to be a good talk you're gonna love it okay um so uh just a little bit about me so I am Josh Grant um I am um I have been working in test automation for about 13 years um doing lots of stuff if you've heard of uh Selen web driver worked a lot with that if you've heard of if you've heard of apis I've worked with that um all this kind of stuff I'm a pythonista um all this fun stuff and I live in uh wonderful Toronto Ontario

it's across the lake um yeah so you're familiar with it now here's a beautiful picture of it um so yeah so uh and today I'll be talking about fuzzing so so quick show of hands how many people have heard of fuzzing or fuzz testing oh okay almost all of you that's super and how many people have have actually uh done it or under undertaken some kind of fuzzing okay almost everyone okay so hopefully this talk will be good okay um all right great so okay so so today's topic is is fuzz testing what is it how does it work um and how to use it for fun and for profit so um I guess I'll be honest I'm I'm I'm

coming at this from a test automation perspective uh I think it's super cool use of test automation um I I'm less familiar with it as kind of a security or um application security uh approach so uh hopefully you'll learned something good today so what is fuzzing it all started on a dark and stormy night in Wisconsin this is 100% true so I love this introduction so um there is a professor at the University of Wisconsin um uh Professor Barton Miller uh in the computer science department and in 1988 1989 he was working one night again uh definitely a progressive person working from home and he was working from his home office and connecting to the university network uh via a modem uh

you know which was not something everyone did in the late ' 80s and one night he was working so he would connect uh retrieve files or or send data to and from the University Network where he's working in his home office but it was a dark and stormy night this is a true so it was actually raining uh there was a lot of precipitation and he was actually experiencing some networking issues U during this storm and he did some testing did some debugging and realized it was actually from the storm and either it was an electrical storm or some kind of um like precipitation and it was actually causing some issues in in the connection

it was um causing some extra fuzzy fuzziness in his connection so there was extra static going over the wire or something like this and so what he found is he would drop his connection during the storm only during the storm other nights when it was clear and there was there was no um no storm or anything it was fine so he figured out it was actually because of the storm so it was actually that he realized the network connection wasn't as as solid as he thought it would be and that if you added this extra fuzzy fuzziness or fuzzy input um you could actually you know trigger bugs in your network software so he could actually he actually realized

that his his network connection wasn't as as solid as I thought it would be surprise surprise there's bugs in software um so so this is pretty cool this is pretty interesting um kind of reminds me of a story where I was I was just thinking about this um I was working at this company and we used to have an all hands and um it was it was small company we would all stand up and so one time the IT guy literally came behind me put his hands on me and moved me over like about a foot and said people block Wi-Fi so and then everything worked so it's kind of like that so so Professor

Barton got his grad students looking at these kind of fuzzy inputs and trying to figure out how to input fuzzy data semi valid data uh to find bugs and networking programs so that was that was kind of the the Genesis if you will um of fuzz testing or fuzzing so so what is fuzzing so that's kind of the story I like the story I like any story that starts on a dark and stormy night especially when it actually starts on a dark and stormy night um so fuzzing is an automated testing technique uh where you send in semi valid or or kind of semi- random quasi Rand inputs fuzzy inputs um and these are generated and passed into a piece of

software to find bugs or security vulnerabilities okay so and you're all very familiar with this this is this is good so so thinking of things in terms of strings if you have a string type like this could be a fuzzy input right it's kind of semi- random um you've got some kind of semi invalid things you got this Open Bracket could that cause some problems you have some special characters mixed in with alpha numeric characters something like this this is how I think about it like it's not I'm sure there are more sophisticated ways to think about it but kind of fuzzy strings fuzzy inputs so so what is fuzzing so a fuzzer is a tool that uh generates fuzzy data

uh so it generates fuzzy data automatically somehow and then passes it into the system under test or the kind of software under test um so a fuzzer does those two things it will generate and call your system under test so sort of pass in this data so does those two things uh another term that comes up fuzz Target so fuzz Target is a function that's actually called by the fuzzer so in the bad old days of software where you only had C++ and everything was kind of you didn't have all the nice tooling that you have now for finding issues uh with you know C++ and you know memory validation all this kind of thing um you

would just write a function in C or C++ and your compiler or your uh fuzzer would would call that and execute that function so we call that a fuzz Target um you could also call this like a test case um if you're outside of kind of the SeaWorld uh yeah you call it like a test case or a fuzzy test case or something like this but that's what a fuzz Target is okay so fuzzers fuzz things it's what they do like um it took me a long time to kind of learn this um you know I'd hear about fuzzing browser fuzzing for example where you'd fuzz a whole browser like Chrome or the chromium project uh

you fuzz a C library or C++ Library like what does that mean what do you do um well fuzzers fuzz things that's what they do um so and the simplest fuzzers uh just Generate random input they just Generate random input um just like an alpha numeric string or something like this these are these have been called by some people dumb fuzzers uh or simple fuzzers um you just Generate random inputs uh and then you call a piece of software with these and that away you go so I have an example of this um so I'm a python person so I get to use Python here so I'm excited um so an example uh using a fuzzing with this

tool hypothesis it's a component testing library but you can also use it for fuzzing um to test an encoding function so I'll kind of show that I hope

okay so here it is okay so here it is um so this is a very very simple function uh use it basically you're testing that if you have some kind of Json uh JavaScript object notation so it's a way to kind of write um like a list of things or an object of things you you want to verify that if you can if you load up a string as Json convert it into Json and then you take that Json and convert it back into a string you get the same thing all right so it's kind of this invariant um like if you're kind of math or computer science inclined it's like you're looking for an invariant you're looking

for an invariant property um but if you're not thinking that way just I have a string has some brackets and some stuff I want to convert it into this format then I want to convert it back and just verify get the same thing right so it's like you input output nothing changes um and I want to verify that like I can try that with like basic Json like what about an empty Json what about like with one entry what about with nested entries you could think of all sorts of things right you could think of all sorts of things um but you could just let test automation do it for you so um so I'm going to use this so I have

this given text so it just says pass in some text um that looks kind of right um and I'll do this so I create a string uh load it into Json and then I'm going to um uh or sorry load string yeah I'm going to load it up as I'm going to load it up as a string as a Json and then convert it back okay so all right

okay so I run this I use py test uh it's a great testr Runner tool for python it's excellent I recommend it um with hypothesis also cool for component testing if you're into that um so I run it and I get so I do get an error so what happens I get an error so the test failed so so hypothesis generates all these cases and tries to find something but it's kind of just random it's just going to like throw in some alpha numeric things it's not going to do a whole lot of sophisticated thinking but it is going to do all the stuff for me um Better Living Through automation so it fails with fails with

this input turns out so it tries this empty string so if I have an empty string something happens where if I try to load that up convert it back I don't get the same thing I can't gracefully handle that edge case that's pretty cool maybe I didn't think about that right maybe I didn't think about like trying an empty string maybe I try an empty uh Json object but not an empty string so that's pretty neat so so fuzzing can kind of do this for you so you can find bugs this is an example of finding a bug um just using sort of dumb fuzzing I don't like that term but it's kind of simple fuzzing just generally random 10

minutes okay all right

so computers are pretty good at actually looping through things um so we can go beyond uh simple fuzzing so instead of just generating things and executing things we can do some other stuff so mutational fuzzers um are interesting because they take a known good State and they'll mutate it so maybe you have a string and the string works it says hello world everything works well and it changes one character at a time one bite at a time it adds or subtracts characters and it mutates that string to generate test cases um so that's one approach where you start with a known seed and then you kind of change it once one thing at a time to generate new

inputs you can also do generational fuzzer so generational fuzzer says okay we're going to start with hello world runs okay now we're going to have hello Josh okay runs now we're going to do something else um you know ABCD EFG okay and so we'll just generate from scratch every time and computers can do this because they can just generate stuff very quickly and kind of loop through things so that's nice those are two approaches for like generating or sorry creating more interesting fuzzing techniques Beyond just random fuzzing but people are pretty good too people are pretty smart with computers so we've developed things such as feedback based fuzzing so feedback based fuzzing is even more

sophisticated where you actually you sort of peak inside of your system under test and you look for um some interesting test cases so maybe you start to see oh like this these inputs are slowing down like the execution time or these inputs are returning longer and longer strings or something like this we should should go in that direction and we should try to figure out like you know or we should try to generate strings in that known Direction so and I'm giving a very very basic explanation um I have colleagues that are much smarter at this and know a lot more about this but that's basically the idea of like trying to guide your fuzzer

into interesting places uh for test cases so this approach uses instrumentation so you kind of it's kind of a gray gray box approach where you know a little bit about the system but not too much um in order to test it so uh so this approach feedback based fuzzing uses instrumentation in the software under test to guide the fuzzer towards something interesting so now I'm going to show an example um so this is this is where the code intelligence part comes in um so I'm going to show a method that finds a SQL injection uh using a feedback base fuzzer so and this will be in Java unfortunately so bear with me

that's not right clearly as you see um it is it's very fuzzy um yeah so it looks like inell isn't working um wow that sucks so I know live this has never happened to me um like I mean jet brains intell smart people wonderful people um in conclusion so um what was supposed to what was supposed to happen is I show this this fun function I'll walk through it I guess I'll just explain it um so it's and I have the source you can find it I'll I'll I'll share it um so it's this function and it just takes this Java object and and um passes it into a SQL database um pretty standard stuff pretty

basic things there's there's some extra logic you kind of use a use an object you kind of wrap it into an object in Java and so this fuzzer calls this function that's all it does it just calls this function with um a data a fuzz data provider input um there's really a lot of thinking it just it just happens and it will generate a string automatically that will trigger a SQL injection so it will trigger like some kind of like um malicious select statement um just from from scratch from nothing like it doesn't know anything about the system it just knows this is a function it's in Java that's all it knows um which I think is pretty cool

like it's just you can fuzz a system and get like pretty sophisticated um or you can find pretty sophisticated bugs and security vulnerabilities just from nothing um because the fuzzers are very good they're very smart they're intelligent even um so

um so yeah so please come find me for this example um please don't tell my boss this happened uh this is really awkward um it is a really cool tool though like I I do want to stress like um so the fuzzer that's used is um CIF fuzz it's freely available uh it's based on a fuzzer that my colleagues put together called jazzer um jazzer is a Java based fuzzer um doesn't even require a fuzz Target it's pretty cool um but see fuzz is really neat um strongly recommend it um so so how to use fuzzing um kind of to wrap up a little bit so bug detection uh that's what I think of when I think thing of fuzzing so like

unusual edge cases like what happens when you have an empty string what happens when you have an empty um bite or uninitialized bite or something um memory mismanagement if you're in the C++ world this happens a lot you have to manage me um you have to manage memory manually um you can have some memory leaks and all kinds of problems null pointer issues you can also find these in Java which is pretty neat because these can be kind of annoying to deal with as a Java developer and finding security vulnerabilities so this is kind of more of the the topic for today um so finding things like SQL injection attacks injection attacks cross-site scripting memory

mismanagement again like you could kind of think of this as a category of security testing um you know can I pass in uh like can I use undefined Behavior to crash a system that's written at C++ or something uh and program crashes uh which is still a problem um one of the um one of the uses for C++ is an embedded systems programming so controlling Hardware controlling industrial Machinery um so you want to you want to verify that no one can for example um you know break into your agricultural facility and crash your cow brushers which are automated and which do brush cows to keep them clean um and run on C++ in part um

completely real so yeah um a couple of high-profile bugs uh that were found using um fuzzing heart bleed 2016 uh you're probably pretty familiar with this um in the op SSL Library like there's a cryptographic bug um huge vulnerability this was this was actually or can be found with fuzzing I don't know if it was found in fuzzing in 2016 but it can be pretty effectively um and log for Shell this is the logo I found for it I don't know um I don't know I thought there'd be a cool logo and I got this um yeah log for Shell now two years ago log for J log for Shell you can use fuzzing to find

this as well you can use fuzzing to find uh log for J uh exploits um what did I see somewhere like your light bulbs might be susceptible to this if you have Smart light bulbs and I was like oh my God I don't want to deal with this um so anyway so so yeah another another so these are two very very high-profile exploits um but both can can be found using fuzzing um and may have been found using fuzzing all right so in in conclusion sorry um so in conclusion so fuzzing is really cool I think it's really cool it's an interesting uh it's an interesting use of test automation it's really a neat use of uh computer

programming um and you can use it for a whole array of finding bugs or finding security exploits um and if you'd like to actually see that example that I wasn't able to show uh you know you can go to this GitHub uh project Road smart it's a very simple kind of um crud application uh definitely play around with it um you can scan this to download CIF fuzz if you want um or whatever so uh so thank you very much uh I'm here to around I have some stickers I'll take questions uh yeah thank you for coming [Applause]