AI Security Challenges in 2020

hi um so the topic of this presentation is called ai security changes and ai security is one of the most underestimated cyber security areas right now so my name is alex i spent last 15 years in cyber security from penetration tester and researcher to application security manager and cto and now a founder and i also tried all the areas like from network to endpoint to application and now to ai and we found just a year ago with my colleagues and a new startup called adversary uh and our mission isn't to increase trust in ai systems because um what why it's so important last year i spent a lot of time like traveling uh all around the world

and asking absolutely different people from like amazon jungle to the markets in oman uh like would they sit in the self-driving car and most of the people say no like i don't trust this uh and i i also don't trust the autonomous car until it will fail to recognize a kid instead of con or vice versa so i i don't trust it as well and so we understood the trust to new technology is very important and one of the uh key trust areas is actually security so our mission is to to bring the most the smartest researchers and cyber security experts generic experts and neuroscientists uh to work together and to create some innovative solutions

to to to protect uh ai systems and the agenda for this presentation is quite broad because what i'm trying to do is to give you an idea of ai security in just 40 minutes and to do that i want to answer a number of questions so the questions you see in the slide those are the most important important questions to cover to understand each area t1 so why it's important like what it is uh who uses when it started where it's uh located and how actually does it work so let's start with why why it's the most important question why ai security is different and why should we care about it yeah because first of all

that traditional software made by like a program logic powered by program logic while ai systems are powered by uh machine learning techniques so and those approaches are absolutely different and when we talk about the interaction with traditional software we mostly talk about the like graphical user interfaces with menus and buttons and commands that you enter so most of the typical attacks on traditional software are actually uh improper validation of some kind of structured inputs like sql injection for example while in ai the interaction uh mostly the cognitive and solutions parts like a vision or audio or natural language and attacks here are basically have the same approach trying to poison some data but here we have like unstructured data

so when we talk about the traditional software we have like traditional injections like sql injection and we kind of know how to uh understand if it's input or valid or malicious because we have some kind of understanding of this data but in ai systems uh for example in visual data we don't really know how to you know separate the malicious uh data and non-malicious data so the trend the the threat landscape is really changing and we need to have uh a new solutions here why it's important uh now mostly first of all because uh because of the research progress uh and currently we have more than two thousand different research papers about different practical aspects of ai

security we also have a number of different public initiatives uh like for example darpa is a created grant for ai security program and we also have a number of market signs uh a number of companies started to provide ai skills to services and new startups uh like ours and also the gardner also highlighted the ei securities topic as one of the strategic trends for 2020 and why it's important in general because in ai as in any traditional like old-school solutions we also have confidentiality integrity and availability and for all those risks we have examples we have examples of confidentiality uh with netflix um data set uh incident uh we have integrity issues like uh facial recognition

solutions were bypassed during a hong kong protest and for example the malware detection system from silence uh the machine learning based milford detection system was also bypassed because of the vulnerabilities in machine learning algorithms and like self-driving cars they also uh you know causes different cars to actually to crash because of the uh potential attacks and the vulnerabilities in schindler's algorithms so we have examples we already have examples of uh different uh real issues in app so what is uh ai it all starts with data and different ai solutions actually deal with different data most of the attacks on ai are actually against the image based machine learning applications more than 60 prostate attacks then we have algorithms

and there are different algorithms to deal with the data like classification regression clustering dimensionality reduction such association rule learning and etc currently we see more attack examples against classification uh probably around 80 to 90 percent of all attacks but it doesn't mean that other algorithms are not vulnerable they also vulnerable and there are examples about attacks on all types of algorithms but just because the classification is more usable now and when we combine the data and the algorithms we actually have the application so there are different applications as of now and those are uh the most popular uh in terms of the number of research papers i've focused on those the security of those applications so

you see image classification of course is most common but we also have the face recognition malware detection and speech recognition those are the most common applications then uh the next question is who actually use those ai solutions and i would say that ai uh it's is everywhere now or uh even if not now it will be in the near future and here in the slides you can see like top 10 ai powered industries uh those are the the most common right now like automotive and health care uh and even cyber security the those all those industries are ai powered now and among those industries we can tell that some of them are more or less analyzed

by researchers so here you see the top five uh industries that have the biggest number of uh research papers focused particularly on this this industry so you see like internet and cyber security and biometrics like face recognition are the most common but it doesn't mean that uh only those industries are affected actually if we calculate not the number of particular papers focused on political industry are but the papers which can be uh applicable to each industry we can see a much bigger figures and we could can actually say that you know most of the research papers uh even if they about like image classification they really applicable to most of the industries because you can you can find like classification

tasks everywhere pretty much everywhere so i just want to say that all industries are kind of equally affected in general so the next question is when uh it was started and where are we going and uh where the most interesting uh things happening first of all the number of research papers about ai security as i said before is is growing is growing really fast and i would say like in 2016 when i uh first started uh working this area i was able to analyze each new research paper now i can't do that because like you have at least like three to five research papers about this topic like every day i would say that uh there were no

uh papers about ai security like before 2010 uh of course they were and probably the first examples of really like ai security papers were in 2004 but the most um intriguing point i think was in 2015 with publication of first uh paper about adversarial attacks against neural networks and this is where when actually everything started we can also look at the different countries and of course we can see that the u.s is actually leading here and then we have china and then we have the rest of the boston european countries uh but what i would say is that the the number of research papers from china is actually growing much faster and probably we can see in in a few years

like u.s and china uh again sharing the first and the second place but i won't be really sure that u.s will be on the first place in few years but let's see and also what we have here is uh different initiatives that actually those countries are making based on their research and probably the first uh initiative uh where security of ai was mentioned [Music] was published in u.s in 2016 16. and then a number of other countries uh joined this trend and published uh other ai security or ai initiatives uh with different security topics covered and now and then we see a number of uh initiatives uh which were particularly focused on security uh they started to uh appear

in 2019 and we predict that in 2020 and 2021 all the rest of the countries will join the same trend and they will publish actually ai security initiatives and probably different laws or regulation rules and etc okay now i think the most interesting part is how are actually ai securities is working so what are the attacks what are the approaches and defenses and so on so let's start with attacks first of all there are like three big categories of ai attacks the first one is manipulation when we try to somehow manipulate with the input so that [Music] the system will wrongly recognize our inputs then we have an extraction uh which is quite the opposite

uh our goal is to extract some data from machine learning model uh and there are different types of data extraction some attacks can extract the data from the module some attacks can extract uh particular parameters of the model so we can actually steal someone's model remotely and the third area is injections here we have different types of injecting data into a training set so that the system will be trained with some kind of malicious data and the top three attacks from this list are actually uh evasion yeah uh or it's also called adversarial examples when we throw the models detection or prediction now the second place by the number of research papers is actually poisoning

when we retrain the model with kind of malicious data and in the third place we have membership inference this attack allowed us to guess if a particular example was in the training data set so let's quickly uh look at the attack examples so evasion attack is actually if you what like you see in the picture when we have some kind of uh picture of a peak and we want uh the system to recognize as big as airliner so how we can do that there are a lot of different approaches like mathematical approaches probably more than 100 different types of attacks uh but simply saying we um we discover first of all we're trying to discover

the most important pixels that affect output and then we actually change those pixels a little bit and craft maliciously put a malicious image that can uh fool the model so after that the model makes wrong prediction um so the method of finding the particular pixels and finding the values of those particular pixels the methods are very different and but this is the topic for like a more detailed uh discussion because there are hundreds of different uh types of attacks like how to do that um then we have a posting attack in poisoning attack uh our goal is actually to somehow retrain the system so that the system will change decision boundary and and then you know think that the target

example uh will actually change the class so to do that uh we need to obtain some training data uh we need to choose some target instance from the trained data then we we make changes with this example to produce a malicious example uh and then so we we retrain the data and this kind of poison poisoning actually shift the decision boundary in such way so that our our target now will be in a different class so this is how we can for example bypass the spam spam detection system yeah i would just change the the normal spam and do kind of poisoning attack

the the next example is a membership inference attack uh our goal here is to understand if the particular example was in the training set like um for example there is a website which can collect um your data like uh your picture of your face and then train their face recognition system based on your data and you want to be sure that they don't use your face during the training so what you can do uh you can create your neural network that produce uh two different types of uh probability vectors uh by testing the the target network so you what you can do you you take the training data and uh and then you put this data in the in the your

target neural network and you collect all the probability vectors then you took some non-training data and then you collect this probability vectors uh from non-training data then you create some kind of uh your own attack network and this attack network you train it to classify uh the data from training data from training set and from non-training set by uh training this model uh by showing them probability vectors from training and non-training data and then when you have this uh attack network now you can take your own picture and send it to your attack network and most probably this attack network will tell you if this uh the picture of your face is actually came from training or

non-training data this is how you can understand if the target network actually used your face and there are other approaches to do that okay well um we have a [Music] a little bit of understanding of attacks now uh let's look at the assessments so how we can assess systems and this is was the example of just the case study that uh the company one of the smart solution providers asked us to tell them which camera and algorithm is the most secure for implementing the fascial recognition system and the problem is the problem was that there were over 2 000 different research papers about ai security and all of those papers have different [Music] attacks and different models

different data sets in different environments and and there are no clear understanding of like a real world if the real attack is really possible because some of the attacks were just based on the uh images of people not the real you know facial recognition systems uh and so on so in order to to test the solution first of all uh we should understand how to attack it and to to create some kind to attack um for example facial recognition system we should answer a number of questions the first one like what is the goal if the goal is misclassification so just to hide your face from facial ignition system or your goal to bypass biometric

security so you want to make your face look like uh someone else face so it's the misclassification or target misclassification then we have different constraints because if we talk about the digital world like the website uh checking the image of your face uh this is one class of so here one class of attacks is possible for example you have you can change each pixel like a little bit but if we talk about the physical world we cannot you know change pixels there uh because uh in the physical world you need to to have like a real patches on your face uh because if you change a few pixels then you have some camera distortion and

so on so your attack will not work okay we know some kind of constraints but then we need to create some form of uh for the our attack uh so if it's a again fast recognition bypass we need to either create glasses so lens or mask or something yeah then we need to think about the algorithm what kind of algorithm we use to produce these uh those glasses adversarial glasses and then we need to think about the robustness of of those glasses because if you conducted the attack you probably know like like uh you need like those pixels in red color but when you print it the color is quite different and you need to deal with all those robustness

issues like inconsistency of colors then you have different positional glasses and so on and so forth and then uh we have model and everything like a lot of things really depend on the model because uh like what kind of data set you use what is the model architecture what is the model input or output and like how how do you test this model is if it's a white box testing or if it's a black box testing uh and so on so a lot of questions actually uh you need to answer to understand uh to to check different models security and then of course we have environment because even if you can break any model in the lab when you transfer to the real

environment you have issues like different lights and brightness and distance to objects we have some device features like different cameras have different color rendering and resolution quality and different preprocessor features like codecs and data transfer compression and so on so we need to take all this into account when we do the actual ai security testing so i can say this is this is not uh simple uh this is a really like complex uh task uh but this is the only way how you can really um check if your uh ai solution is first of all vulnerable and like this is the real threat and once you assess your system your next question is how to defend it

and here we have uh four approaches the first approach is a kind of security assessment it's a prediction stage here we have like testing against different types of attacks or uh different verification methods um this approach is quite easy to apply because some of the attacks are available and you can test against those attacks this approach is more or less transferable because you can apply like some of the attacks for um different types of machine ai systems but for example the attack testing it's really limited so if you test for one attack it doesn't mean that it's not vulnerable for other attack and if we talk about the verification methods they are more precise but they are very slow

so you can look at this area as a penetration testing like application security for ai then we have prevention approaches like modification of input for example jpeg compression for all images or a new type of deep learning model or a modified way how to train the system for example adversarial training when you add adversarial examples to your training set those approaches are in general are very good but the issue that they are task specific so if you modify input with jpeg compression you know it only uh can be applicable to images and particular types of images and so on and also those approaches are quite complex so if you want to modify your module yeah we you have some advantages but you

also have some disadvantages this is like uh different types of hardware hardening and vulnerability management kind of approach for ai if you compare it with traditional security solutions then we have a detection approach we can detect by supervised learning attacks that we know or by unsupervised learning attacks that we don't know um a number of methods are really good i say a number because there are a lot of different detect approaches but some of them really good and they are more or less easy to implement you don't need to you know change your model on something unfortunately it can be too late to detect attack because it's already exploited so sometimes it can be too late to detect attacks

and some of the detection methods can be applicable to your solution so you can look at this as a security monitoring or thread detection type of solutions for ai and respond uh those are also different approaches uh currently there are very few papers in this area but they exist are they mostly about like counter attacking or online retraining or some ways to um kind of hide both approaches to fool the attacker those approaches are really unusual so they're quite hard to bypass because attackers don't even expect it but unfortunately they much harder to implement uh comparing to other approaches and in some cases they're really unpredictable so you can look at those approaches like incident response

or like real-time application security protection and kind of valve for ai solutions so uh as a result uh for air security life cycle i would say that all of those approaches are important just in the particular direction like in direction which is presented here so you should start with kind of predicting uh like conduct some kind of security assessment and understand what are the risks the next step is to apply some kind of fixes to prevent those risks uh but also next step is to apply some kind of detection mechanisms to monitor and threat monitor stress and anomalies and finally uh some kind of responsive measures to react and mitigate and retrain and so on

so uh thanks for listening and i hope you now have at least a little bit more understanding on ai security and if you have any questions i'm here uh to answer them and you can always write me you can write on this address or uh to alex at verso i and i love to talk about the ai security so thanks for listening uh thanks for joining for joining um if you want to you know be part of our team you also can write me and we're hiring so thank you again and have a nice day