IOCs in your APIs - Jason Kent

[Music] my name is Jason and I do things security wise with a company called sequence security um they fly me around and I speak into all these conferences I recently spoke in Riyadh last week um I speak all over the place I do a lot of oasp events and I teach for OAS I've been doing this for about 24 years you give me URLs I give you error messages that's how this works um and then you go fix those error messages a few years ago I bought a new garage door opener for my garage and it didn't have a button you know how you used to have a button and you push it and your garage would open uh and I got

this garage door opener and I said there's no remotes in here and they said there's an app use the app on your phone and you can open your garage and I went all right uh I guess I can do that and I open the garage at the app of my phone proxied through my laptop and then I opened the app I open the door with my laptop then I called a friend of mine and I said didn't you just buy one of these and he said yeah he lives about 2 000 miles away I said can you read me the number off the back of your garage door opener and he goes sure and he read

it to me and I said is your door opening yes excellent uh I was just a few lines of python away from opening all the doors they had a insecure direct object reference problem um I do a lot of with graphql and I I can't stand here in Australia and not give a shout out to the asset now guys and The Kite Runner folks um really great amazing stuff I'll actually put up a word list that I think is a little bit more accurate than theirs but I collect graphql endpoints API open API files these sorts of things I build tools and teach I work for oauth summer of code last year it was the most popular class

I taught people how to tear apart APK files for Android apps whole API endpoints out of them and then go attack those API endpoints um so it's a really interesting thing this is my LinkedIn page I don't know why we've all moved to QR codes but it seems easier um you'll see a whole bunch of them later if you guys want to connect me on LinkedIn that'd be great um if you don't want to you don't have to so what is the problem well we're starting to see more and more organizations get attacked through your API platform uh there's a lot of different reasons for this it used to be we instrumented everybody that came in

the front door you go through the web app right you go through our website that means I can run JavaScript on your machine unless you run a no script or something I can see a lot more about that end user apis are meant for computer to computer communication they're not meant for an end user to be interacting with it and then we went well wait I can make a mobile app and it'll just do the same thing that a web app will and we'll just call these same endpoints and we'll make it all easier and then graphql came along and we're starting to see apis are everywhere but we're also seeing that the indicators are compromising these apis

are happening in ways that seem a little different right Optus you guys are all quite aware that Optus had a problem right what was their indicator it's the Brian Krebs notification right where you get somebody calling you saying uh guys here's three million accounts that we have a bunch of information for the indicators there were after breach right um and so we're going to talk about some kinds of indicators that are out there things like if you work in retail and you use gift cards does that gift card check-in Point get hit a lot well of course it does people are checking to see if their gift card has any value on it what if you're guessing numbers and

trying to figure out what gift cards might be out there your indicator might just be activity right strange activity it might be but there it is so what are we going to talk about today well we're going to talk about what API attacks look like my company that I work for this is all we do we sit in line with apis and watch the attacks fly in and mitigate them and so we're going to talk about what are the things that we use as part of this solution um what is an ioc what really is it right um what are the common IL season apis where can you instrument to find them what kind of data can you use and then

how do you put this together so if we look at an API attack this is usually what happens and this doesn't really matter what it is whether it's a hype sale where somebody's trying to buy shoes shoes is one of those weird things that I have to go protect a lot of and don't understand right why people buy Nike Dunks I really don't get why Yeezys were something that was important I don't know keeps me employed though as they keep getting attacked to buy the things right what happens in the first phase of this is just simply reconnaissance just like any attack I need to see what's there what can I do what are the interesting things I might

find what is it that I'm going to be trying uh in a few minutes when I move on to the next phase which is testing and in the testing phase I'm just going to see can I get this to work right and this is where you spend all your time pen testers in the room it's a few right and this is where you spend all your time right I wonder no I I won't no I dang everybody thinks that you're like in The Matrix constantly and it always works but yeah I've been there you know you throw a million things at it and none of them work and then finally you rip something out it does

um testing right but once you get it figured out then you move to scripting right we're gonna start running this thing a little bit harder and then maybe somebody's gonna notice it right you get a little loud with what you're doing and they shut you off then you got to retool right you got to move to the next phase and this is what happens over and over and over again and that reconnaissance phase I've got to figure out where your API endpoints are right so uh anybody that wants to can pull their phone out right now and they can pull out a web browser and they can type api.twitter.com and they'll end up on the API subdomain

for Twitter if you throw slash graphql on the end of it you'll get some error message about a malformed graphql request how hard is that to figure out well Elon said something about a bunch of rpcs failing the other day and somebody posted back to them you even understand how graphql works and I went oh there's a graphql endpoint on that API I'm pointing you know and I just started enumerating through it and there it is right I don't need that I don't need all these other tools because ilano just tell me we're there but the rest of us are going to try to find things and there's lots of things Out There apk mining is something that I

do a lot of tearing apart Android applications because it's basically just a zip file and inside that zip file there's a couple of different XML files that just point at all of the paths that you might be willing to go after and so that makes things really easy but if you're attacking an API endpoint that's maybe East-West traffic right isn't North South traffic isn't something that's generated by an end user but is actually coming off of another thing then maybe we got to look at DNS searches or we've got to go to tracking pages right to figure these things out my friends and I are starting to do a bunch of research on free food right

sign up for the app and you get free French fries in McDonald's it's an interesting thing so I had my son sit down with a hundred cell phones and create a hundred Gmail accounts as part of a class project that I funded and uh 100 free French fries right that's an easy thing to do but what happens when it gets a little bit more complicated what happens if you've got to buy a bunch of things and Associate it with all those accounts well then we have to dig into the apis and look a little bit harder and cracking will help us with that it'll help us build config files so after we get our reconnaissance done

then we have to move into testing and the iocs for testing are really hard to see in your app environment when you start to look at the traffic that's going to come in on this it's going to look like what we call single request attacks that comes from one IP address one user agent one username for a request so it looks like it's spread all over the world using all these weird different user agents each request looks like it's Unique so it's really hard to Trend unless you Trend all of the data right how many data science folks one it's the future of cyber security for sure because looking at this data lets you see things like these single request

attacks but you'll also see stuff like this does anybody know what xmlrpc.php is anybody here run WordPress this is the WordPress RPC um defines what rpcs can happen against a WordPress site if you see somebody trying to go to this endpoint on your apis they're running a scanner it's pretty easy to kick them away right just look for that UA and kick them off um let's talk about versioning though right V1 login versus V2 login where I recently had a customer that said we accidentally left our V16 login on and they were on v37 or something and V16 was standard username password authentication and v37 was saml right so I'm bypassing your saml authentication by using an older endpoint that you've

got to shut off I'm now going to say something that I'm writing a talk about but this is the most important thing in API Security today if you have API 2 API 3 and API 9 in your environment you are step functions further down the line to be hacked than anyone else is we've done analysis on over a trillion transactions several different hacks this comes from our data but I also did a birds of a feather talk and at RSA conference last year and I sat with a bunch of Banks and I said what's it look like when you get attacked we piled this all together and we realized if you don't have authentication you have

excessive error messages and you have endpoints you don't know about you're doomed right um and if you look at breaches that may have been in the news lately you'll see a pattern there so once we get past we did some Recon once we get past we test to make sure that it's work now we're going to start firing the tools right and if we look at the history over time command line tools work first we use Curl or something like that right to just ping the site or make sure that things are up or I can get to the end point but then we rolled into using tools that were framework attack tools so it understands I'm going to make a

web request and it understands going to get a response back and I've got to parse data maybe you have to point it at the login page so tools like open bullet or Sentry MBA came out if you're a retailer you battle these tools all the time if you're in Banks you probably spend a lot of time with these things but basically it's a framework tool you create a config file you drop the config file onto the framework and it'll go get you free pizza from Domino's right I use Domino's as an example all the time because if you go to cracking.org Domino's is the first one that comes up um I don't know if it works here but in the

US it does right scripting the ilcs on this are going to be volumetric you're going to see a lot right it's going to look like computers coming after you um the robots are really hitting you hard um so you're going to see things like in 40 minutes 25 million requests when you normally don't have any requests or your volume is much lower than that as it comes through you we've seen things like 35 or 350 000 account takeover attempts in 25 minutes that's a slow attack for us we we usually see them much faster than that right and now with graphql becoming more and more adopted graphql has this really cool featuring called batching which lets you

put a whole huge pile of requests together as one request you send in one request so if you're logging and you look and see what happened when this transpired you're just going to see one request but if you open the Json object and that request it's going to have all of the stuff inside of it right and hopefully it didn't get you a whole bunch back so you're going to go in there and figure this out and stop them and then what's going to happen well I'm going to go find a meme generator for retooling um and tell you all about what happens when people start to retool right so you're going to see this initially

you're going to see this coming from Amazon gcp Oracle Cloud Google Cloud right it's cloud services are going to be attacking do your end users typically come from Amazon IP address space probably not right um it might be depends on your business but normally it doesn't so then we got to move into residential proxies right we need to reflect this stuff off of residential IP addresses that way it looks more legitimate right so where can I go get a bunch of residential IP addresses anybody know

Illuminati changed her name to Bright but yeah so there's a whole bunch of these anybody know how they get them how did they get the residential IP addresses how many of you believe that when you use Starbucks Network you need to put a turn on a VPN or insert whatever free Wi-Fi network here you have to turn a VPN on to be safe those YouTubers are all Italian nordvpn is the way to go or whatever right but you got to pay for all that stuff what if you want a VPN that's free I question efficacy here but okay so there's a VPN out there if you guys want to look for it the it's the Spanish word

for hello it's Ola VPN hola VPN when you read the terms and conditions says that you become an endpoint in Illuminati Network you're now a residential IP address that some botter is going to go use now we're only going to use a couple right we're only going to be on there for a few transactions but here I am reflecting IP addresses off of what look like real legitimate end users in cell phone networks using set-top boxes and that kind of stuff right so what's that mean I gotta spend more money the infrastructure gets more expensive the better this goes and so as you see these attackers retool they're going to retool to more expensive options they're going

to realize that their user agent strings are all three months old and they need to update them right they're going to realize that these IP addresses are bad and so you're going to see this but if you're using behavioral analytics and data science doing things like what's the first thing this end user did is going to tell you an awful lot right um also doing a bunch of analysis on all of the iocs so what are the iocs and apis is it the same as everywhere else is it a bit different well we just had the forensics guys up here on stage talking about uh malware attack ransomware attacks but iocs are just a piece of digital

forensics right think about what it is you're logging or what might be inside of your environment that's going to tell me a little bit about what's happening when an attacker attacks your apis just like physical evidence there's digital Clues and it's going to take you further and further down the line so what are the common ones well you see them in login endpoints and here's where behavioral analysis becomes really important if you have an exposed login endpoint so if you're organization has people log in from outside and you look at that login endpoint look at how often someone succeeds in logging in all day right so how many successes did we get today how many failures did we get and then

you sort of set high water marks in that and when you start to see that login failure rate climb up you're under attack from a credential stuck being attacked right as that failure rate climbs you realize somebody's trying to get into that login endpoint we also see things like user agents right browser make yeah uh if you're talking about apis it's pretty rare to see somebody with Firefox touching you right normally it's your app if you work with your development team and have them set versioning inside of the app user agent stream you can use that to track which users are touching your endpoints right um it makes it nice and simple a lot of this comes from doing direct

analysis but what we like to do is to do some data science and understand where the data pivots might on might you know unmask things that are happening so if you look at things like login request for user you know Bob comes from France and then it comes from China Bob probably didn't go from China to France or France to China uh in the middle of this transaction right so just simply understanding the IP addresses that have come in and doing a pivot off the user and the IP address is going to help you similarly user agents things like volume right behavioral analytics is going to do a lot I took the build out of the slide it

came back so what does it look like when somebody's doing things like scraping right is scraping illegal here I'm not up on laws uh in Australia is it is it legal to scrape websites here probably okay it's legal in the U.S you're allowed to do it um you can stop somebody from doing it but it's legal for them to do it uh so in the beginning right wget style scraping would be something that somebody would do they'd fire a script at you and they would pull down every element of the page right kind of indiscriminately but then we start to see things like well if they're scraping you like that they're noisy they're easy to see you

can shut them right off so then they move right we start to use things like inexpensive IPS somebody figures out that they can get a proxy list right and start running through that as we move down through this though you're going to see things like single request attacks really high volume rotating the user agent strings as well as the IP addresses um and they don't look like anything is going on except there's a huge amount of volume in in the requests that are coming through I work with a lot of organizations that do things like they resell cars right and it's if I sell the car I get the commission so if I go scrape someone

else's website that's reselling a car and I list it and I sell the car I win right we work with a lot of organizations that are kind of underneath that and it's really fun to watch you take a picture of one of the cars that they're trying to sell and pull all the watermarks off of it and this picture has gone to like 27 different sites and everybody throws their watermark on top of the old Watermark to say it's theirs only all right this is a really weird um economic thing that happens but scraping still exists today people are still doing it so an account takeover though you see things a little bit more sophisticated

there's a little less rattling the doors um what happens in the attack setup for an account takeover this is I've got a list of usernames and passwords or list of usernames that I want to go develop password lists off of and then take the accounts over you're going to see they're going to need a bunch of this stuff they're going to do a bunch of these single initial requests but eventually they need to get through the list right if I have your username but I don't really know what your password is how many combinations is that it's a lot right and it's going to be a big password list unless I dump the database and I actually have your username and

password if I just know your username it's a long time to figure this out I recently did a responsible disclosure with a camera company I've got a greenhouse out back in in behind my house I grow vegetables in and I came out one day and something had eaten all my new cucumber plants and I was super annoyed by this because they take forever right and if you bite the top off that's the plant dies and so I want to know what it was what got in I mean so I put a little camera out there I bought and then I looked at the camera and how it worked and how the authentication flow worked and because I am who I am

and I do what I do uh I realize that if you type in the wrong username it would say error bad username and if you type in the wrong password it would say error bad password what does that do for an attacker right I now can validate usernames and password lists very easily just by using the error messages which one is that of the three I said was going to be terrible for you successive data exposure right API three um you don't want that to happen so we're going to see this we're looking for error messages we want some information you may see successes in here but you're going to see a lot of failures login

failure rates going to float up right um but usually in this phase where we're doing initial testing we're not ready to to really hit it hard yet because we're going to need to have better infrastructure we're going to need to have good tools out there we're going to have to have a lot of things in place in order to make this work but it's going to be fast and it's going to be in the middle of the night typically similarly we see payment card enumeration right um I find a gift card endpoint if I can check a gift card and I don't have to do anything extra if I don't have to be logged in to do it if I don't need a PIN

right if I could just type in a credit card number or a gift card number and see how much values on it I work with a small organization in Canada they were having somebody check 250 000 a month off their cards right now lining that up with theft is pretty easy because I can say well somebody tried this gift card number and look somebody used the gift card number um so put a pin on it it's not that hard um add a little bit to it but then what's going to end up happening is well we'll just move to accounts right I'll do an account takeover and use online purchasing gift cards that are available

there whatever that might be sometimes we'll do things like credit card guessing my son has showed me this where he wants to buy some game right and he's like well Dad I can just generate a credit card number and try it and I'm like don't do that all right I've been doing this for 24 years the way that you do it for 24 years is you don't break the law right uh you can break the law and get rich one time easy right a bunch of people do it it's just when you want to repeat that when you run out of money and stuff so what we'll do is we'll try with what we call zero cost testing I'll buy

something from your website that costs zero dollars just to see if the credit card stuff will work right I can regenerate bamora tokens this way um and a whole bunch of other things like that so where are the iocs in that well zero cost transactions are rare right it's an easy one to find so how are you going to figure out where all this stuff is because remember I said we got rid of all that client-side stuff when we got rid of the web apps right everything's happening on the mobile app now well we can tie up an IMEI number we can do uh extra things you need to work with your development staff to get that kind of stuff created

right so that the app is actually feeding the system but you're not going to be able to do this with your wax right wax typically don't sit in line with API transactions and they typically don't sit in a place where SSL is stripped off or TLS um so what we end up seeing is a lot of instrumentation into some place and that's usually nginx or whatever that equivalent layer is for everybody else if you don't know this creates a whole huge pile of logs that you can go mine but it also lets you do things like a Kafka pipeline so you can have the logs flow through and then immediately make decisions on those things this is

exactly how my company's products work I'm giving away all of our secrets here because this is how you have to do it um it's really not that hard but what you want to try to do is create as much signal as possible so that you can do analytics on it data science is really important here doing things like just simple counts is also going to get you pretty far right successes and failures as I've talked about a few times we doing all right anybody hungry yet it's almost lunch we're close the guy before lunch so understand what endpoints are important to you right and this takes a little bit of analysis and a little bit of figurine right is this

endpoint important to me is this something that's going to have a data element that I care about log in sign in authenticator obvious right cart checkout order however you check out flow works for your e-commerce site but things like item availability right you're gonna see more and more organizations that want to be uh friendly to their customers and create as little friction as possible so if I need to go buy a hammer I pull up my local hardware store website and I see do they have the hammer how many of you have done this type of activity right everybody has these websites now and I'm going to tell you a story about stopping theft um at a brick and mortar through API

ilcs really fun little thing here but let's talk about the birth of data science let's talk about the birth of this analytics so we can understand how we can apply this because those people that don't study history something something repeats itself um so in 1854 the very first data study was ever performed it was performed by a guy named Jon Snow who wanted to figure out why his neighbors were dying at the time uh cholera epidemic was spreading through London but it seemed to be only in certain places and they couldn't figure out what the thing was like why are people dying from cholera but only in these buildings and so John pulled up a map and he took a

pen probably one with a big quill on it and colored in every time someone died a color in that building if you look at this map from where you're sitting right now you'll probably identify there's a place where it's Central where everyone was dying and you can see the big line here and there's a lot of big lines around it and you can look at the roads and say well the roads all seem to lead into the same place and right up there by that longest bar it says on this map pump didn't have indoor plumbing in 1854 they had to go down to the fountain in the Square or a pump somewhere nearby and

carry that water home and what John realized was this pump had cholera in it do you know what the solution to this problem was they unbolted the handle people couldn't pump water anymore and people stopped dying right using data analytics doesn't necessarily mean we have to do something really complicated right I don't have to have a bunch of machine learning to figure out that we're seeing a lot of failures on the login flow right it helps you get there but it doesn't necessarily need to be really complicated so sifting through this noise you're going to see things like behavior on on user agents right I'm going to be Safari Chrome I'm going to be Edge I'm going to

be all these different things should they be on your API flows of the very first question you should ask the second question is should we see these old user agents all this spread right you typically don't see that especially with chrome users because that update button is annoying right you want to push it and get the update um and so you want to go through that monitor your top end though look at what's out there and what is making the most noise headers are you getting all of the headers because one of the first things I do is rip out off headers a lot of times systems on the back end with no authead or present just assume that it's

computer to computer communication and let me in right oftentimes 256 A's is a pretty good off token right so look at these things right Canary headers put headers in that if they rip them out you'll notice right they don't need to do anything and you can just randomize them but if they pull them out send them into the honey pot right let them go play in there for a while looking at rates is going to be important so that's a behavioral indication stack well let's move over to infrastructure can you tell what these two IP addresses can you tell that they're part of some kind of attack flow whenever I look at a tax happening

at my customer sites I'll break it out into IP addresses like this as the data pivot and you can just see it 3400 requests 3 400 requests 3 400 requests 3 400 requests 3 400 that's the machine doing that right that's not people it would be very I don't know randomized if it was just people so you have to look at how these things are going to show up in your systems IPS versus username is a really great way to do this right how many of you use the same device for all the things that you do probably in your hand I think someone just took a picture of me right so cell phone how many IP addresses is that thing

going to have this month probably five maybe seven how many today maybe two maybe three if you see an end user showing up with six or seven or eight different IP addresses that's probably an odd behavior it might be something that you'll want to look at username versus user agent how many user agents do you have right I use Chrome pretty much all the time unless I'm attacking something and then I use Firefox just because that's how I have my tool chain right and I also don't want to mess with sessions and stuff um so I that's just how I build it right user agents by population should we be seeing all these old user agents should

we see all these different organizations out here if you're a small company in Canada should your users be coming from this wide of a spectrum right if you're here in Australia and you only deal with Australians and you never have to deal with anyone else you probably shouldn't see traffic from Russia just block it right at the edge and get rid of it scanners are one of those things that we're seeing more and more of simply because scanners make it easy to identify things this is a big chunk of what assetnote did with Kite Runner but I've broken this down into what I'm seeing as actual successes it's a lot of successes and you can see in these

endpoints what I'm looking for right why am I looking for graphql so much anybody know because if I can find the graphql playground on that playground is the database schema how to make correct requests it's like a Tac path mapping right it's nice and easy for me to do but there's a lot of other stuff that happens too subdomain enumeration look at your outside footprint do you have your staging sites or your Dev sites just hanging off the internet and easy to get to maybe behind basic auth um we see this awful lot uat sites and staging sites tend to have very interesting databases behind them because when a developer wants to create one of these things they want to work

correctly and so they want data that looks real and you know where you can get a good copy of a database that looks real prod right there's a good database in there it looks just like the one we need put that over Endeavor uat or whatever so you have to look for people poking around on these things because they're trying to get to them right graphql on your uat site for instance is a really nice attack path so let's throw this all together customer comes to me and they say we have a problem we use a third party API solution to help customers find things that they want now the thing in this case and in this

story is right here anybody know what this is a guy with a hair so it's a hair dryer [Laughter] uh I I'm not you know hair dryers aren't really a thing that I deal with a lot um it's you know it all fell out when I was 25. anyway what happened was they called and they said we get 50 000 requests to this system that will tell people where they can find things a month and on day two of the month we're at 500 000. something is going on please help us right so we instrumented and we looked and we said well it turns out this is going to be a pretty easy problem to

solve because they're only looking for two mailing codes right two zip codes and they're only looking for one item but they're really looking a lot uh and so what we did was we worked with the organization and we said go put six of these in your database in this store right so we saw the API Behavior change when they did that now they were just banging on that store all day right they want to know are you selling them are they selling out because the next thing they're going to do is run into the store grab them and run out of the store and not pay for them and so this company dispatched the police

uh we cranked up the number of hair dryers available at the store and sure enough it was about three hours later this team came running in they grabbed the hair dryers and ran for the door these air dryers are six hundred dollars right they're fairly spendy piece of equipment and it's not because they have you know an overly large amount of hair they're putting them on eBay right they're reselling these things so the police stopped them on the way out of the store and they arrested them this is a sophisticated Smash and grab robbery team using API endpoints to their advantage tape sales the most confusing thing to me in the world is people that really

really really really really really want to buy these shoes um I I don't understand it so this is a Time spread I think this is 2100 and it goes to oh 400. what happened it turns out a bunch of people showed up right you guys heard about Taylor Swift and her Ticketmaster Fiasco that's been going on lately very similar right all of these people showed up to buy the shoes three minutes before the hype sale was to launch this Spike right here happens we're creating carts the way Bots buy these things is they go by or they add to their cart something no one wants that they have plenty of stock in kids socks is usually what it is for hype

sales on shoes because they'll be kids socks that they sell um so then I put the kids sock in a cart what this is doing is I'm establishing the flow to check out but I'm getting a step ahead of everyone that's trying to check out with me right so imagine this if you will I run into the store and grab a PS5 and go stand in front of the register and you still have to run in the store I'm gonna get my PS5 first so I add something in the cart that's innocuous right so now I have my checkout Channel established then what I do is I start to look for the item that's going to drop right so in this

case Nike Dunks so they don't release the item numbers ahead of time right they only go live when you say you know search for Nike Dunks search for Nike Dunks search for Nike Dunks eventually they'll say yeah here it is and here's the skew so now I have my cart built and I know the SKU what's my next step update the cart with the new SKU and checkout well you're still searching for Nike Dunks this is how Bots beat you right so what we do is we sit in this and we throw the traffic away so all of this red traffic that came in and the white line there 99.1 percent of the traffic on this

particular sale we threw out so the humans could buy the stuff right um so then they get really mad the bot guys they get mad they threaten our researchers and all this but they do this thing where they say I couldn't cop any of these shoes today um but they usually have them pre-sold if you go look at one of these hype sales Supreme super dry it doesn't matter right you look at one of these really high volume items that somebody's wanting to buy go look on eBay for it they've already got it listed they know they're going to buy it right they know they're going to succeed the Bots know that they're going to win

all right four minutes left I'm going to show you example here of account takeover if you look at just simply the isps that are touching a login endpoint when it goes to boxes that start attacking you the isps are going to be inexpensive infrastructure it's an immediate shift from what looks like normal in the United States if I were to look at traffic on an e-commerce website that developed like this a t T-Mobile Verizon I would think this is all normal traffic cogent shows up no those are all bad infrastructure right notice how easy it is to find the box it's just simply crunching the data science crunching the numbers looking at paths applying a little bit of machine

learning and Viola right easy enough shut it all down the ilcs that are sitting in your apis are there for you to find they're easy to find and they're there for you to take action on does anybody have any questions what happened to my cucumbers that has a groundhog I wrote a blog post about it and I got a responsible disclosure out of it so I built a little better fence and kept the groundhog out

so I work for a company that makes one of those so obviously I'm gonna say we got one um what you have to be able to do is tell nginx to throw it out right uh you have to be able to do analysis in flight um so you're going to do a Kafka pipeline analysis and a mitigation flow inside of nginx it's going to be all hand scripted if you don't have the tools already built we do it mostly in Python yep anybody else who's hungry it's lunchtime kids [Music] [Applause]