BG - JARVIS Never Saw It Coming: Hacking machine learning (ML) in Speech, Text and Face Recognition

okay thank you everyone for attending our talk I'm going to start with a couple of introductory slides that we're probably already aware from somewhere else so our employer asked us to put this legal notice and we wait we will wait while you read it that's it so now seriously when we constructed this talk we haven't harmed any horses flamingos sentient inai turtles and also this will make sense during the talk my name is guy and this is Ezra we're both from Israel both presenting our work that our team is doing back in Israel around hacking machine learning and hacking AI in general and we are going to go into much deeper depth into the technical aspects of what it actually

means to hekia we haven't come here ourselves bound based on our own knowledge so always always when anybody is speaking on stage that means that they've done research and they've built on top of someone else work of someone else's research someone else's efforts so I want to acknowledge those people we will mention them along the way you will usually be able to see at the bottom right screen if you're taking pictures greyed out link that has some more information if you want to follow up on that and we will have an end aknowledge minute slide at the end so first of all is how did you even get to be here on in besides today so about a year ago in

DEFCON 25 during one of the let's say whisky fueled lively conversations a couple of us got into us a talk about what it would actually look like if you were hacking AI what is me what does that does it actually mean to hack a AI and from that talk that conversation came a research about a six month long research into what exactly that means which we are going to present here today so what can you expect first of all we're going to talk about what it means to hack hey I am going to give you a couple of background background slides to make sure that you understand what you need to understand for the actual

attacks then we will cover the attacks the couple of different variants and we'll wrap it up with a couple of their key points whenever you see something highlighted in yellow I hope it kind of looks yellow on this screen that means that something you need to pay attention to we're not going to release zero days today we are not going to give you a framework for hacking AI but we are going to teach you what it need what you'd need to do in order to get there so for us to begin I want to tell you a story about the horse so this is a horse from 1903 is her name is Hans usually known as clever Hans and he was

known as clever Hans because it was pretty smart horse he knew how to do simple arithmetic questions like 2 plus 3 1 plus 5 stuff like that he could spell words in German which is amazing both for horse Po and for anybody to spell anything in German and psychologist was dispatched to try to understand how come a horse is able to solve arithmetic questions or even spell words in German and that psychologists invented something that we know today as a double-blind test and that means that when he checked and looked into this matter of the horse he found out that his handler that you can see here in the center holding his out his end was

actually cueing the horse into the right answers so he would ask the horse how much is 2 plus 3 and the horse would tap his hoof 1 2 3 4 5 and he would read the body language of his handler to know when he should stop and the reason I'm telling you this is because in machine learning or AI the current state of the art is very much like that and that means AI is very good at solving very specific problems whenever you try to generalize or try to give it something that it's not expecting to get it breaks down it breaks down horribly and we're going to walk you through a couple of those examples so what do you need to

know first and foremost the difference between the very large password buzzwords that you're usually getting all over the place so everybody is saying AI artificial intelligence I want to break it down into a three basic concepts that we will build upon next the first one is machine learning machine learning is the task of actually taking a bunch of images at this use case a bunch of inputs and giving them to the system along with some metadata telling them what this picture or this input contains so for us in this picture there is an object which is the flamingo's object which is a girl also an object which is a hedgehog at the bottom so we are feeding the sister and

a bunch of images and a bunch of the input and the system learns to classify or to discern between the different inputs according to the different metadata the next thing is deep learning deep learning is exactly the same thing but we're not giving any metadata we're just shoving all of the inputs into the system and letting it decide and discern on itself the main difference between normal machine learning or legacy machine learning versus deep learning which is like the cutting edge new stuff is that in deep learning you need orders of magnitude more of inputs and data for the system to make correct decisions so if I needed 10000 inputs to be the machine learning

system I will tell you I would need million inputs to build a deep learning system and when we are saying AI what we really mean is the machine that knows how to understand context so it's not just this is an image with a flamingo and a girl and a hedgehog but also that she is going to play cricket in a moment and we all know the story of Alice in Wonderland that this is the context for that image and that is artificial intelligence and we are very far away from that and I must apologize during the talk I won't say AI and I'll say ml and I always always always I mean machine learning there is no AI

there's nothing like artificial intelligence yet only only in the movies so the main thing that I want you to keep in mind is that AI systems or machine learning systems are designed to solve specific problems keep that in mind this is going to be a very important thread throughout our discussion today what do you need to know to understand what machine learning is first of all you need a couple of basic math understanding we're not going to go there now but I do want to give you some notion of what a machine learning is so I'm going to use a neural network in this example so in your network you can think of it as a graph

with a couple of nodes and each arc on that graph is a holding weight so what does that mean it means that I'm going to multiply the input by a weight which is depicted by a line here and then I'm going to take the output modify it with a function multiply it again modified it with the function multiply it a grenade sum it together and then I have enough so what I have here is the input layer which is the three nodes on the left side and I have the output layer on the right side and these are the things that give me flexibility in how the machine learning system is learning what it needs to learn the innermost layers the

one in the middle here are called the hidden layers the hidden layers are the ones that encode the actual learning in the system so the the input layer and the output layer in code the way that we represent the data both when it's coming into the system and both when it comes out of the system but the hidden layers are the one that do all of the heavy computations all of the heavy understanding what we're trying to learn we have weights which are as I said are at the lines that we have here and we have the outputs the outputs are not human readable in the sense there are just a number we have a bunch of

integers which we are multiplying together a bunch of floats we are multiplying together in the end we get a number that number means nothing to no one except the machine learning system and we have to convert it into something that humans can understand in reality the neural networks look more like something like this so they don't have to be well balanced they don't have to match each other and they actually have many more inputs than you can see here so a normal input normal neural network that we are working more in the world today like an image classification network will have maybe a thousand nodes and maybe a hundred thousand parameters so nothing that I can show on the screen

but to give you like a sense of what it is so the way that we are building these models we are training these models is that we take that all all of those inputs that I mentioned earlier all of that metadata we push it into the system and we make those calculations whenever the system is correct we increase the weights so it knows it went the right path whenever it's incorrect it will decrease the weights so it will know that that computation didn't work out and we do this in an iterative process again and again and again and again a couple of million times in the end we will reach a steady state where the

network has learned something about the inputs in our world when we are saying a model a machine learning model we mean a couple of 3ds a couple of different distinct things the first one is the topology how does the network look like how many layers how many nodes in the lair this is the topology the other thing is the weights so what is the multiplier coefficient that you're going to multiply each input when it moves for the network and the last thing is what kind of function we are going to use usually you're using nonlinear functions but which nonlinear function that is very important the bottom line is that whenever we are talking about machine learning or neural

networks or deep learning or whatever we're talking about encoding some information some learning information from the wild world into that system this is the IP this is the intellectual property of whom whomever is building this usually the companies and that's what they want to protect and we will touch on that point later so to make this more interesting let's do a bit of a deeper dive into what matrix multiplication is well not really sorry just to stress you out we're not really going to do a linear multiplications here but basically speaking when we are multiplying two matrices together what we get is a matrix and when we're saying matrix and we also mean a vector but

both of these from our point of view we don't care about the math behind this we really care about this and that means when I'm saying this is a vector I mean as a single dimensional array of bytes or floats or integers somewhere in memory and when I'm saying a matrix I'm saying an array of bytes or floats or integers somewhere in memory so from our perspective we don't really care what the matrix look like what the vector looks like we care about how it's represented in memory how we can retrieve it how we can modify it so generally speaking these machine systems are very much like voodoo you put inputs in you get predictions or

classifications out and that's kind of suck because you don't really understand how this works and I'll spoil it for you nobody really understand how this works not even data scientists so most of the study or most of the research area is completely ignoring the middle where everything is happening and focusing on the results that you're getting but we're very much interesting what happens in the middle so a quick run-through of what happens when you're doing these predictions or going through this machine learning phase you will take your input and you will encode it in some sort of intermediate representation you're taking an image you have to put it in a vector in order to multiply by a

matrix so something has to be done to encode it as a vector that's the first step the second step is to actually do all of those algorithmic computations and then you have to take the output and modify it or change it or check it or do something with it in order to get a classification that's that part we call the mapping where we decode it back into human readable format when we are training the system we're doing this part iteratively again and again and again and again but in the end when we are deploying a real-world system we're not pushing the train system we are pushing a deployment model the deployment model is the same thing like

the train model basically but here we are moving linearly from the left to the right or from inputs to predictions and we don't retrain the system all the time in most cases so the thing that I want to impose on you now or to clarify now that when we're talking about machine learning models we're not talking about code it's not the same thing and there are a couple of very large distinctions between the two the first one is that when we are doing a reverse engineering or code review or something like that we can trace the code flow or the whether the program is behaving according to different inputs we'll put a single input and we'll do maybe a dynamic

analysis or static analysis and we'll know how the system will change over time and how it's going to look when we are doing a machine learning model we have to do the math operations to get some sort of intuition about what it means and we have to do this about each and every different input because when we are having 10,000 strong matrix it's very hard to get an intuition about what the system is going to do the second thing is that most executables have data structures in machine learning world we don't have data structures we have a sort of like vector representations of stuff very hard to decode very hard to understand what it is but the most

important thing is we can't really take the matrix and do reverse engineering on the matrix understand what information it encodes so whenever you're training a system and you get that matrix that's it's done there's no way to go back into the training and to understand what it is that the system was trained on and this is by the way an unopened problem in academia how to reverse engineer a model to understand how it was trained when we were talking about the model in the end we're talking about this this is just a couple of the header file for the header section for a machine learning model specifically called ResNet resident 50 and I just want to show you

in the end this is just a couple of bytes bytes in memory bytes in disk and they pose a problem because if you want to validate we want to make sure from a security perspective that whatever we're getting is secure it's safe we should use it we don't have tooling we don't have understanding of how we're going to do this we can't really do code reviews when we get a matrix there's nothing to review when you're doing a matrix and we don't really have code here it's not really like we're running code there's a framework that gets a matrix is an input but that's it there's nothing to review here so an open question it is how do you even

secure these kinds of machine learning systems against attacks or against a malicious adversary who change those models okay so that was like the first part of the background and now let's get into a bit more interesting things as I said the side of the models is pretty large it can get up to bytes in size so we're talking like very very heavy computations a lot of data in memory the second thing is or the most important thing is is that machine learning systems optimized to get the best or the strongest signals that they can detect and that means that if I'm feeling it an inputs that have some signal in them some of those signals might be different

than what you imagine when you have a one set of thinking about what your import represents and the reality is that it represents something else we call that a bias and I want to show what the bias is with an example so I'm not expecting you to read this I'll do it like a short introduction a dermatologist did a study of skin tumors and he found out that when he trained his system and invested a lot of time like the thing he took him like eight months something like that to train his system he trained his system and in the end he understood that he has a very horrified system to detect rulers and the reason that it was that most of the

images that they used had rulers in them for scale so they it was easiest for the system to understand okay I can see a ruler in this image if I have a ruler and the ruler is this big or this scale that means that this is the tumor if I don't ever believe in this image this is probably not a tumor so bias is a really very real effect and give you another example this is also from health care they try to build a system to detect cancer from blood samples and they ran all of the different blood samples in the database through the machine learning system it was very nice very high accuracy and

they tried to validate the system by taking some information the decision has never seen before and it felled miserably this is pretty often in Vegas science is pretty common and the reason for that was that they saw that it had locked onto a very strong signal which was the hospital name why was the hospital name such a strong signal well if you get a blood sample for somewhere called John Hopkins center for cancer this is probably a cancer a blood sample so getting bias out of your inputs is very difficult when we're trying to describe the security aspect of what we're doing we try to tie down into a common language to usually use like CBS

s scoring and the reason that I want to tie together is that we can have like a common understanding of what it means to attack those systems and what are strong attacks versus weak attacks so we are going to focus on five different attacks today those are one depicted in red backdoors denial of service IP theft misprediction and model tampering there are other texts here on the on the screen which we are not going to discuss today all of them very interesting we don't have enough time so we did our CBS s analysis for those attacks and we you can see them here prioritize from denial of service of the first place down to back doors at the bottom according to

the CBS s course so we are going to show you a couple of examples of how this works in the real world in like three slides but what I want to show you here is that when we had conversations with customers and partners but what they care about when they're thinking about their machine learning systems they really cared about these two things and the first one was IP theft and the second one was model tampering so denial of service while being very high on CB SS was not very interesting to companies deploying machine learning models they were scared that somebody's going to steal their IP and stealing the IP here means that if I'm building a smart security camera for

your home to recognize your face and unlock the door if somebody steals that IP goes to China and builds a cheaper model and with your IP in it and then sells it to understand you you really don't have any protections against that and that what makes people very scared so here going to discuss the attacks and move the mic to answer so first of all let me start by saying that I am terribly excited to be here and thank you all for coming I'm an attacker so I'm going to talk to you how do I attack a problem to actually build an attack the first thing that we need to know is some stuff that we're going to cover

later and who else we will cover what are the areas that we should target and the third point that we're going to cover is what do we need access to be able to perform this kind of attacks clicker please again okay so there is an interesting point here as we have been talking about machine learning before there are two different ways that we can go when we talk about attacking a system like this we can either go against the infrastructure or we can go against the algorithms or in the math and just to recap them and now I'm going to to tell the difference between algorithms and infrastructure if you remember when when guy was explaining what we are looking

for when those black magic voodoo thing became something real we were talking about an internal representation and we're talking matrix multiplication output and mapping internal representation is directly a system operation and infrastructure operation and mapping is also an infrastructure operation and we are going to see that afterwards well networks a multiplication output are fully an algorithmic point of view and this is very very important so why is this important because the first part whenever were talking about machine learning is that machine needs to take an input any input that you can imagine and it needs to convert into an internal representation so if we return here we have our input the nets to became an

internal representation that any framework can understand and you know making any input into internal representations always has been hard parsing stuff is really really not easy if somebody here has something written a parser to understand me I hate parsers I will not pursue good sir and the second important thing is that whoever knows how to develop stuff for machine learning or artificial intelligence or whatever are not file format developers they done they know how to do matrix multiplication they know how to understand that kind of stuff but they are not five formats or parsers people I am NOT a file format or parser person

and the solution is something that we all have ever done in the past whenever we need something that we need don't know how to do which is let's ring an external dependency Pippin stroll parse this bitmap file because I say it's just wait to do that we are not 5'4 right and not it's not our expertise so let me just repeat not our expertise let's ring gun dependency let's finger outside libraries into our stack and not only let's bring out suppliers let's bring out levers that we don't fully understand what they do and bringing a very very very common problem in the industry which is supplying chain and patch management when will I know that

certain library that it's the most important part the first building block is being touch further I'm going to ensure that whenever a modification one of those core libraries for me to perform my work is changed and account framers must support multiple file formats so if we have multiple file formats and we want to exploit the hell out of these platforms we need to start with fuzzing because it's the cheapest and easier way to to do it to understand parsing file form and simpler relatives and the first question that we have is where to start we can start add okay I'm going to speak all the time in this side Oh that also works so we could we could

just go for the deep learning framework which is called kefir which has full coverage which whatever input I will send and if I find a crash it will read directly related to my objective or to my attack target so it's great what's the big issue that whenever I'm fuzzing affair and fastened end-to-end system and end-to-end framework so it means that it's every one of my inputs is going to the full matrix multiplication process and it's going to be take a lot of time because now we are doing machine learning when the objective is not to do machine learning the objective is not to find the labels the objective is to find the crashes but as we were telling

before it uses dependencies so we can go to the open computer vision framework which is what they do to parse the images but the issue here is we will have a limited coverage not probably not all the functions that exist in the open computer vision framework exist in the cafe framework the advantage is that the speed is way faster because we know we are not running there full end-to-end matrix multiplication and stuff like that or we could go to the library itself which is extremely fast but we have an own cost pets or go to upstream and just twenty five one of the open abilities that exists there are also relevant to our system so now that we

have to another to understand how is a fuzzing and supposing theoretically that we have a crash and exploit we have an open question is remote code execution the King that can lead us to the biggest fears on this industry so let's try to demonstrate the first one we're going to take a look at is a denial of service and we are going to be abusing a memory lake that exists when parsing certain file formats in images so on our before you start the demo just to explain the system that we're doing here is just taking an image as an input and produces a classification or a label as an output so it will take an image of

a dog and we'll execute it the machine learning and will output a string sayings dog as an output so this is like the overall image and we're going to abuse that so on this side is the package perspective on this side is going to be the server perspective we are now going to run h stop and now we see beautiful graphs then we go to the library wait we're running the model then we start running the model and at the moment that we start running the model if you take a look at the memory utilization it starts to go higher and higher and it will continue for a lot of time actually let me fast forward to a

few actually three minutes later so if you can take a look now the memory is now at 6 or 3 gigabytes D to us this was a 10k input file and it's practically using all the resources available in the system until it crashes now we just use 6 dot a gigabytes of memory for a 10k file the business impact is amazing I mean imagine just seeing that the costs of your cloud platform just go up and up and up and your following services and downtime and whatever just with 10k images but who likes in a love service when we can get remote good execution and in this scenario of using a memory leak we are going to abuse hip-hip

exploit GUI we found a memory corruption bug in the ER that could be used for the same purposes and again left side attacker right side in this scenario is going to be also attacker as you can see we have a host name that it's called WM we go to the library we run our exploit against this input file what we're basically doing is that we are uploading an image to the service this web cloud service whatever it is which you can see on the right hand side this is the server side of what's receiving our inputs so we have an image file as an input and something happening on the server so now at this moment we're

asking the system please tell me the output the classification of this specific image and guess what and guess what at this moment it binded me a shell I have now a show where I can connect an idea that typo here and instead of typing hostname hi type host I'm at WN and I can see all the files in the system and I have a full shell and I can do whatever I want at this moment I have full remote code execution where the input was a single image and this is a very important part and I'm going to return in a few seconds to this but remember the input that I provided to the system was a picture the output was

a shell and now we start with the part that we were saying okay what can we do with up with our remote code execution vulnerability and actually we can change the entire weight the model was created to return outputs because we have file access because we have filesystem action because we can do whatever we want and let's take again a look at at at the demo where we just connect to the server and now we are going to go to the library we would live to the directory where we store the model and afterwards we are going now to take a look at what are the classification that this particular model knows how to do and if

you see we have a lot of image classification stuff like gall fishes and tigers and cuts and Hanson whatever and then the next thing is again we are going to run an image to this system through try to understand what happens which is again an overflow we run it we have a segmentation fault and that's the only command that will happen and afterwards all the labels all the identifications become hacked so imagine this at this moment every image you sent to the system is going to output hacked disturber this is like that Armageddon scenario for any machine learning system and the last one would be the IP theft I'm not going to show the demo because

we saw it after before but if you remember we we were able to see every file in the file system including the model including the label files so we have full access to steal the model to steal the intellectual property and again we're doing gives us posts RC so the thing that we found now is that maybe remote code execution is the king however what happens whether you will you don't have a remote code execution let me give you the answer we talk to guy okay so what we've covered up till now was if we attack the server we upload an image which is maliciously crafted image that we've created we can get a remote code execution on the

server we can cause a denial of service we can cause a lot of different things but if you don't have access to that kind of attack we don't have an exploitation ready or payload ready there are a couple of other things that we can still do on the algorithmic mathematical level and I want to walk you through a couple of those examples so the first and maybe the most interesting attack is called cloning attack or IP cloning and what I want to show you here is that if we have an a cloud-based service or a server that is doing machine learning as a service it has some sort of public facing API if I can use that API to query that system

I'll send it images and it will output labels I can use that to train my own model using the IP that's already inherent in that Oracle in that server that's already up there in the cloud and that means that if the company invested I don't know a hundred million dollars of engineering time buying the data and ten million dollars on engineering to actually have data scientists building the model and I will invest five thousand queries to retrieve the model practically I'm the winner I had didn't have to invest any NRE on developing that model I just used the Oracle to build my own so there are a couple of different attacks from that perspective there's a white box attacks

where it means I have access to everything I know everything and this is you would wouldn't expect it to be a reality scenario a realistic scenario but actually it is because many machine learning companies are pushing out open source projects and papers and going to conferences and discussing their algorithms and I have a lot of inner access to how they're building their machine learning infrastructure how they're building their topology how they selected their functions so if I have all of that knowledge that means it's much easier for me to clone it and if I don't have access to that knowledge even if I only know the domain or the topic that the machine learning system is

designed to do it's pretty easy for me to clone it so I'll give an example let's say that I have the traffic recognition a traffic sign recognition system for the United States and somebody trained that system and now it knows how to recognize traffic signs very very well I want to clone that system and to steal it I don't have access to the data set so I don't have access to the US traffic science database but I have access to the Brazilian one so Brazilian traffic signs and US traffic signs are practically the same I don't have to have the exact same data set I just have to have something similar enough for me to build those

queries and get that information out and that is a gray box attacks meaning that I have a domain knowledge but I don't have specific knowledge and we've done that and with very high success rate the thing we're working on right now is black box attacks black box attacks are very difficult in the sense that we have no idea what the domain is we have no idea what the machine learning system is doing we don't even know what kind of labels it knows how to put out so we need to do an exploration stage we have to build our own data set we have to introduce a lot of effort into it because practically we're going to

reverse engineer a system that you cannot reverse engineer you have to really have to explore those code paths by yourself but if I had access to the training data I could do a lot of more interesting stuff for example if I had a control of the in the training dataset I could introduce my own data into that training data set and control the output and I'll give an example let's say that we have our classification system that knows how to differentiate between cats and dogs so the system was introduced to lots of inputs of cats with labels of cats lots of dogs labels of dogs and they knows how to function correctly and to identify cats and dogs correctly very

boring very very well maintained system but now I take images of bananas and put labels of dogs on them and put them into the training data set the system was still trained you would still put images of dogs and get labels of them you'll still put images of cats and get labels of cats but nobody knows that you've inputted an image of a banana to get a response of a dog but you know that's a back door realistic we speaking let's say that you have an intrusion detection system that is looking through network traffic trying to understand if some packets are malicious or containing malicious payloads and some packets are benign if I control the training data

said a nanny I introduced a magic header or some prefix that's known to me but not known to anybody else I can build the system or trick the system to ignore or to always consider whatever comes after that prefix up to that magic that header is always being benign if it even if it is malicious and that is one hell of a backdoor but a lot of the interesting stuff that's happening in the AI world today are called adversarial attacks and that is taking a set of inputs and modifying them in order for the machine learning system to mistake and take out the wrong prediction out of that system so in our cats and dogs example it will

mean that I will take an image of a cat modify a couple of pixels send it through the system and now it be identified as an airplane and that's pretty easy to do and pretty scary at that and the reason that that works is because the system is designed to fixate on the strongest signal that it can identify in the problem space so it's not looking at the images of persons like I can detect two eyes and a nose and a mouth and whatever it's actually looking at relationship between pixels color gradients histogram of the image a different stuff of what a human is looking at and that means I can exploit that in order to introduce maliciously

chosen chain artifacts into those inputs to control those outputs and I want to show you a couple of examples so I discussed a lot about an images being manipulated in the talk so far but there was a lot of heated discussion in the AI community does this even apply to a real-world scenario so in a real-world scenario you would have other problems such as objects of not really flat there are 3d objects have lighting problems they have linear perspective problems not everything is looking the same thing from every angle so the challenge of doing these kinds of attacks on a 3d object in the real world is much harder and a couple of researchers proved that

this is actually doable and the way that they did is is this is not an actual turtle this is a 3d printed turtle just to remove concerns and if you will look closely at the top shell of the turtle you'll kind of see like red spots so this is currently running through Google's inception v3 Network and it classifies it on the left if you will lo closely as a rifle in almost every angle and that means that those dots that they've carefully selected on the shell cause the system to mistakenly think that this is not a turtle this is actually a rifle and that is pretty easy to do a different example from a different world and I want you to listen

closely right now is from an audio perspective

without the data said the article is useless okay so we have a voice sample and this is the attack sample that's useless so I hope you were all able to hear the attack it was very clear should I repeat it okay I'll repeat it so what happens here is that what we're hearing is without the data set the article is useless what the machine learning system would hear is okay google browse to evil calm and for us there's no difference Willy can't hear the difference or if you listen very closely you would hear like a small noise in the background maybe you'll be able to pick it up probably not and also from a real-world

perspective so this is from where last year's taught by a higher Anderson at DEFCON 25 and what he did he'd build a machine learning system to full virus total and what he did is a system that we compile this code sent it to virustotal got the responses back and pushed the system into classifying his virus as a benign one by keep mutating and modifying the code until he got the right response back from very soul so he built a system to full virus total hence all of the different AVS around very interesting stuff and I want to give you a short example about privacy because I don't have a lot of time and that is

that our machine learning systems the machine learning systems are pretty leaky so if we have a machine learning diabetes detection system and we have Joe come in and Joe has the lowest score of 7.4 or I don't know maybe Joe have the high score of 35.3 and the insurance company who bought the system will sell him insurance and won't send him insurance that is their privacy concern but if James came in and James was part of the data set that this system was trained on we can pretty convincingly say that he was part of that training data set data set just based on his score and that means that the privately identifying information of him of being

part of that system was leaking just by the score that he receives fooling facial recognition system is also pretty simple and the reason it's simple is again because facial recognition system will fixate on the strongest signal so here are a pair of printed glasses that if you put it on you will be identified as Miller Hawk of itch or perhaps like Carson Daly and I've seen a different video where someone put this on and it was identified as Vladimir Putin always a very smart thing to do at least in foreign airports and I want to wrap it up because we're really out of time and to give you some information what you can take out of this presentation so the

first one is that we need a better trust model we need to have better understanding of what the attack surfaces are what the attacks are going to look like and how we're going to improve that so Richard researchers should focus on that area and specifically be more about the infrastructure so a lot of heated debate is going on about can we build better image attacking attack image classification attacks but actually everything else is kind of broken so we need more researchers looking into that our tips for attackers for building those kinds of attacks is that there's a lot of valuable targets out there there's a lot of interesting stuff that you can do it's not a very difficult

area to go into it's you don't need to learn a lot of math there's a lot of things that you can do there's a huge dependency stack you can go and do lots of stuff without even touching the eye and finally for defenders we believe that you need to put more controls in more sanitation's more understanding of where your inputs are coming from and how to make sure that they're trusted and you need to validate those data because if you're not going to validate someone is going to give you a malicious payload instead the bottom line is we have to do this machine learning is here machine learning here is not going to go away so

we have to do as a security community to find what the weaknesses are and how we are going to overcome them and provide solution and not just attack surfaces we would like to acknowledge the members of our team honor daddy then is lazy Adel superior knowledge who are part of this research and we will release the slides later along with all of those references I don't expect you to take their picture and we invite you to come to talk to us after the talk after the Q&A catch us later and please let's have that conversation thank you very much do we have time for questions yeah we have a couple of minutes I will remind you if

you have a question to raise your hand so I can come over and bring the mic to you so also if you want to reach out to us that's our Twitter the best channel and we have a question in the front please keep them high so we can see it over everything

yes sir is there a methodology to objectively minimize confirmation bias when in your training sets you mentioned earlier that the the deep learning you can't really say how it concluded this from that you the the middle is a black box in which case if bogus data is put in there it seems impossible to pull it back out or even know why it's there so there's an entire field right now growing in the area of artificial intelligence research called explained ability studies and what it means is that are trying to design new machine learning schemes or new under planning schemes that will also be able to explain how the system reached that conclusion or from an example

perspective you will have an image and you will have the object classified in that image but you will also have like I classified it as a dog so we'll know which pixels correlate it to the answer of a dog so we have that very relationship to look at your data what caused that response from your data so it's an open field of study there's up like a math very strong methodologies yet yes it looks like a lot of what needs to be done as inputs annotation what would an input sanitation solution look like so inputs annotations can take a couple of different levels here one would be don't use untrusted data so if you're going to input data from some sort of

set of sensors encrypt them authenticate them sign them whatever you need to do to make sure they're getting data from your own sensors and not from something else but if you have a generally available API you would probably want to look at the way that those inputs are feeding your system are introduced into your system and what kind of effects they have so an attackers perspective or an attackers built input will have different characteristics than user defined inputs so that's one way to look at them the other way to look at them is the end result so if you're looking at the outputs the way that outputs are built based on those inputs will look

differently if it's like a user generated expected inputs versus an adversarial inputs and how they generate the same outputs anyone else has a question just raise your hand can I go girl don't be shy so in many cryptosystem there is this approach that you have known good data so the crypto system can do a self health help test health test what's your thoughts about adding such tests to machine learning to detect changes to the model itself and is it possible that attackers would then just over fit your model essentially to that one known good example to bypass this mechanism ok so I'll break it down into two different aspects the first one is kind of how to use topics from

cryptography in order to enhance security around machine learning so one of the interesting things that we can leverage crypto for is like in self integrity and self-healing things the machine learning world is very firm from it at the moment there are a couple of problems how do you would deploy that we've been working on that for the past couple of months we think it's a viable way to go forward but there are limitations and drawbacks the other aspect is that more often than not you don't have controls a cryptographic controller in your inputs so what you can do a full cryptographic lockdown for your system the inputs still need to move through your system and your system

still needs to react to those inputs so if you're going to tie your system to strictly down it will use its effective 'ti as a machine learning system so it not won't be able to learn in depth so you need to have a certain measure of flexibility in the system so you can't really lock down the inputs and it ya can't really lock down the system yet the basic problem here is that the problem stays the way that the inputs pastes sorry the input space is much larger than the inputs that you're thinking of because the mapping from an input to an output is much larger than your selected inputs that specific output so I can always go and select a

different input for that same output sort of like finding different collisions but in a much easier computationally speaking space okay anyone else have a question all right thank you guy in Ezra thank you very much come talk to us [Applause]