BSidesSF 2018 - KubeScope for the Extraordinary World of Containers (Tongbo Luo • Zhaoyan Xu)

[Music]

okay hi everyone our session my name is Tom ball and this is my colleague Acharya we are security researchers at the stock rocks and for our company we we build the security tools to protect the container world and we deliver container native a security profile that prevented stressed by reducing the attacker surface and today definitely I will talk about container security so it's called cool scope for the extraordinary world of containers and in this presentation I will start with some introductions about the container security octet readers and also especially for the kubernetes and then I'm trying to explain the what is the current problem that is facing by the container world and then is our solution

to solve this problem and at the last there will be a quick demo there so for the for the introduction I will start with two numbers here and the which is revealed in a reasonable way so it shows that enterprises of all sizes are increasingly using containers by 2020 more than half of the global organizations will be running continuous application in their production and more importantly the majority of them are you are deploying and managing their continual Iser production using oxidation software and among all of the oxidation softwares communities definitely the dominated one which has been adopted by 71 percentage of all of them and they they can be used to managing their container infrastructure

so so the kubernetes which as a it's just the open source the tools developed by google and it provides us container centric management environment and it's very portable and a flexible to use I think most of people if you if you work in the same area you should be familiar with with the kubernetes and however the problem is that when kubernetes and those of the traders become the center of of the new universe it's also become the major target by attackers and here's a figure here shows why the security of octa traitors become so vital in this new environment in the most of the time the attack happened from the from the pod where you are

running your micro services for example if a tiger can take advantage of command injection vulnerabilities in your in your web application we can you can inject up recoded in your pod and I secure those commander in your pod but at this time the damage is limited because the container sandbox which isolated a compromised content in your pod from the rest of your cloud and the next move of attackers should try to break out of this sandbox and basically there are two ways in front of them we can either compromised the underlying operating system or they can compromise the traitors here that that is a two major target here like let's take a look at the first away so if they try to if

we can manage into is fraud a kernel one ability we can take over control the whole the host so they can take away what they can do whatever they want like compromise other containers for running in the same host and we do need to - we do need to worry about those those things because the operating system running in the cloud native environment has no big difference from the operating system in our daily life and there's a there there there are many tools out there can help people to to protect and a detector those attack to the operating system people can easily deploy them and the provided those attack so we are more interesting on the second of the second

way the attackers can compromise the orchestrators here because of the readers like a kubernetes they provide a reachable communication between the border and some whole part of the combination a kubernetes call component and if attacker can managing to compromise the orchestrator they can not only control the powder continuous running at the same at the same machine but also the one said that in the other machines run deployed by the same oxy traitor and that is the real reason why the security of kubernetes security of those oximeters is so important in this newly emerged the container based environment and what makes the things worse is that there's a no such a - subsidiary protect aquatic critters and another reason why we

contacted our our research and try to solve this problem so we basically we implement we design an incremental cook scope and we leverage a combination of machine learning and the route based detection techniques and basically we we do a we do a profile that not be based on the normal access to the kubernetes and then for each new access to the community we would try to use our machine learning model and a roo based model to detect the abnormal access and for our prototype we only protect the api server which is a very centralized place in the community architecture and syncs all of the interaction to the AP to the api server is using the restful

api it's basically the HTTP wasted traffic and we the access here we mean is the network traffic so here's the figure to show that aware our cooks cover component located in the whole communities community architecture and basically it on located between the API server and the rest of the Kubik system so I read traffic to the API server mustard goes through our cooks cover component other is the ictv or as vs so paid basically our cooks Capcom hundred will inspect all of the all of the traffic to the API server and I try to mark and the abnormal ones so if we double click it you need to the course cover architecture it highly containerized so

each component here is a container running in the system so we have a mole entire container which will intercept all other traffic to the API server and and pass the request and they put it into a queue and later on the the machine learning module here the coop inspector will will get a feeder from the queue and I try to identify the other normal ones so they were all this module will also put the result back to the queue and it also to a backpack and to the backend storage and we also provide a user interface for users to monitor what it actually happened in their community system who is intercept who is interacting with the API server

and what that indicates traffic is and also they can clearly see where is the of a normal access here and an axis is the cooling inspector a cooling inspector module my colleague Acharya we will take over and explain in more detail about it coop inspector it's actually a components which exist the inside of the coop scope and this this component is merely take care of for detecting logic of the Cooper scope so the import of the Cooper inspector is the decrypted in the normal HTTP request from the clients so the normal here is a basically simplify that the Kouba inspectors measure is a profile page semesters which basically means is to profile all the normal behaviors of

API servers so the normal behavior here we refer is the normal assess from the each different parts to the API servers and the output of the coop inspector will be the models that the models including post machine learning models and also the rule based models to do profile the normal kubernetes @ss so here a list example here is how we actually extract the information from the normal kubernetes request to the API servers so here we extracted a three part of three parts actually four parts of the informations first it's the URL and its parameters the second is the URL it's self and the third part is HTTP headers the HTV headers including all the keys of the HTTP headers and also

it's a corresponding values and the last part is a request to parties for all experience we see a lot of requests party including JSON objects which is issued from the party to the API server to ask you the new functionalities so this is an example I want to show how extract informations so after we extract all these kind of informations we'll analyze each request in three levels first is the URL and its parameters the second is HTTP headers and the third is the payload measures we want to propose different kind of models for each different API course which basically means for example if kubernetes offers different kind of service API it's a different kind of service for the API

course we'll build a one model for one API course so for models we actually build the three types of the models the first type of the models we call it unknown access models that's including something like an unknown URL and no parameters for URLs and no user agents for the HTTP headers and also unknown content type so we list the some explanation here the basic logic is if we find any kind of a URL or parameters which is not been seen before we think it's a possible of a normal assess the second type of the model we call the invalid SS models so in value basically means the contents of the request may contain some abnormal on possible

malicious payloads so here we list the the parameters in validating the cookie values and also evaluate the payloads we were detect whether they have some like a share code on malicious file or malicious script inside of these kind of things and then the last part the last part is based on the machine learning it's called we called up anonymous S models so here we at least the two examples here is the statistical models which we were profile order like a normal packet delenn's parameter values lens and cookie values lens and also the number of parameters so if we find any kind of a pecan the requests that have abnormal behaviors like that they have a

super long packet and also like the value is a super not we were abnormal statistical model were actually trigger and fire up anomaly signals and that the next part is assess frequency models that actually we were recorded the normal frequency of the incoming packets so if any kind of four packets coming up normal frequency we were also trigger alert so here we list some one like the rules here to illustrate how we actually determine whether this request that we the API servers is up a normal or not in the following demo so we were we were demo some like reasons listed here in the product in the body letters and here is information detailed illustration of

how we attack or whether is of a normal things for a non ho if it's a URL parameters which at whether the URLs keys is no or not whether the value it's a valid title for lamp or it should be a string into our IP addresses and also we were seeing whether this value is so it's true or not whether they have some possible CQ injection or not exist in these parameters and also similar scene so we were to fold a request party to see whether this request party is open or not whether they have some known keys or values and whether they contain some shell code and also whether they have some malicious file

including in the request or not so next combo will show all you guys some demos we have present here okay so so Joey and just discussed several reasons we made detector the abnormal request and so there will be several demos the first demo is to detect the compromise the secret unipod and so the scenario is that say you have a pod running your kubernetes and by default the community communities will mount the server service account a secret to your to your pod if we get a compromised that the attacker may trying to leverage you this will try to steal these tokens from your pod and use this token to access your API server and achieve a further damage

and and because our cool scope we were located in front of the API server and so it will intercept the such a request from the generated by the attacker and the things this pod itself it rarely make such a such a request to the API server we will identify this as a malicious one so so here because yeah so here is the video to show so our tag with launcher communities you know in the local and I see you you can see there there are some traffic already been intercepted and and I will try to try to launch it to launch the application which is stéphanie of 101 is a while it's a web application okay it's being

being launched and also the service account token is automatically mounted it to that pod and here is the dummy I mean the one about page and from these figures here's this and whatever you poster from this form will get executed so so I put a several code here at the idea Serta yet so added this this input field is basically it just still the service token in your pod and trying to trying to send the requester using using this toe-to-toe key so so see see the the requester had has been successfully successfully be launched and we we get detect one of the abnormal requests and it so and we list the requester here so so here so here is the route so which is

a if we double click into this of a normal request it's generated by the attacker and trying to use the stolen token to access an API server and because it used valid the wily the body the talk to token and but but it shouldn't be used by by this pod and then we just mark this as a normal one and the the second demo is quite simple it's just the device attack so the scenario is like the attacker may generate a burst of a request to the API server because because the API server use the right restful api for for each request it might just process processes processes money terms and not the requests them may be lost

from the pod or formula Tiger attacker itself and and then our cook cook cook stove component will get this rate will intercept all those requests and one of the feature we extracted from the request is the time it's a temp use a time step so usually the communication between the compete between the worker the worker Noda to the master node is quite a periodic early including the house they check the the left the the problem management request is periodically but such a such a DDoS attack requester they will be just be generated in a short period time it will trigger the oven normal it will be marked as a normal request to do to the temp due to the time step feature

and the demo is also where is also very simple look so I was just using a Python script to generate a bunch of requests in a shorter period of time and the die it will be marked so like I just mentioned earlier and it will be detect and uh providing in our cook our coop scope architecture so as I mentioned earlier because one of our feature you know more machine learning module is those time step and and and the way we also show show show the reason why we mark is abnormal it's a frequency of package coming is abnormal so that is the detection reason here and yeah I think probably that's the of all of my

presentation if you have any questions please feel free to ask and the some sense for you thanks for you coming to our session [Applause]