Rooting out security risks lurking in your kubernetes ecosystem

so we will start with uh setting up the playground and i have shared a pdf document in the slack channel maybe if you click on that link it will give you all the instructions like what we should be doing to set up the playground

so the first thing is like you will have to start by clicking this link and it will let you download the vm which we will be using throughout the workshop

and these are the steps that you should be following to install everything you'll have to follow the instructions as is please don't miss any steps they're like very clear instructions like you have to be very careful especially with the network settings because your the vms needs network access or internet access to download the virtual to download the docker images on the go when we spin up some kubernetes clusters or some containers and it also needs a local network connection in order to ssh the host so please follow the steps as is i'll also do it on my screen and also have a recording i'll be sharing it on my screen if you have any issues everything

you can always message me on the slack channel or in the q a channel i'll be shortly sharing my screen setting up the machine

if you feel that the pace is a bit fast or is bit slow always feel free to tell me and i will adjust the pace accordingly so this these are the steps that you should be following while doing the setup after you download the virtual machine from the link just follow these steps as is import the vm that we have downloaded choose that virtual machine click on continue and here we will set up these always ensure you give some better ram i'm giving somewhere close to like six gigs but it's always good to give anything about four should be fine then just modify this mac address policy setting it all looks good then just click on

import and once if you click on import it may take anywhere close to a minute or two minutes

click on import

the vm has been imported and now the first thing we will be doing is like doing some network settings which is very important so we click on network and ensure just as for the documentation make sure like the first one is host only adapter and the next one is netting so this is very important guys so first adapter is like host only and the next adapter is nat just make the settings as it is shown on the screen and we won't be touching anything else just click ok then we are all set just click on start

so the username to login is the username bassand and the password is ubuntu everything is provided in the document in the setup document and then switch user as root and the password is ubuntu only this particular screen for the next one looks small and then going forward it will be like pretty big so and then issue the command bh client and then try to grab the ip address of this machine because we will be using a terminal to ssh into the box and do it continue with the rest of the workshop so here the ip address for this machine is 192 168 56.104 for my machine similarly you should have an ip address so grab the ip address and then just uh

test it from your shell or terminal or whatever you use just ssh the username is bassand at the ip address yes and the password is ubuntu so the switch user password is ubuntu and then there is one document that you all should be testing it it is like the number step 4 in the documentation dot slash requirements and once it is done you should be able to see the screen if you could see the screen we are all good to go ahead with the rest of the workshop

can you all just raise your hands are you able to follow till this step or do you want me to wait for a while or do you need any assistance just leave a message in the slack or somewhere

yeah so we just got a message saying that someone needs a minute yep i have answered it no rush guys we will just wait for a couple of minutes to ensure everyone is on the same page we are on time so yeah no raj

and now we will be setting up the infrastructure for the workshop so i'm pretty sure once if you give the command door requirements and it should say everything is available and the next probably the first thing i'll be doing is i will be sharing with you all the documentation which we will be following for the rest of the workshop it is like a huge documentation somewhere close to 200 pages so could you all please check the slack channel i'll be sharing it over the smart channel

so i have just shared a document on the slack channel which says like attacking defending and monitoring your kubernetes ecosystem we will be following this documentation and really following the steps as is throughout the workshop it will cover all the modules like attacking defending monitoring and runtime protection

could you all please check the slack channel and download this documentation even if you happen to miss any steps during the workshop this is your go-to place anything and everything that we do during the workshop will exist in this documentation

so the next step that we will be doing is like everyone has to issue the command as is as you see on my screen create iphone n workshop iphone w2 this will set up the rest of the playground for us it will bring up the whole environment different humanities clusters and all the required containers for the workshop so i am sharing my screen and i'll show you how to do it

and depending on your internet connection this step might take a little longer some machines may take like five minutes or some people might even take 10 minutes but don't rush while this is happening i'll be covering some uh theory so don't stress yourself at this stage but this is very important stage if anybody has any difficulty or any errors during this stage just reissue the same command again the command is create iphone n workshop iphone w2 if you have any difficulty during the workshop please just copy and paste the commands through the workshop at times you may have to change the ip addresses and the port numbers here and there

you should see something like this is this happening are we good to go with the theory while this is happening in the background because like i mentioned this step may take anywhere close to five to ten minutes so i will start with some theory so once it is done it will give you something like this and make sure you copy this all ip addresses and port numbers into a text pad if not just leave this screen as is and open a new terminal at least open two or three terminals from which we will be doing the stuff so while this is happening i will cover some theory part so first under someone mentioning an error

on the start code code based server uh right like i mentioned just uh make sure you have like proper internet connection so just give some command like ping iphone c3 some google.com and make sure you have proper internet connection and then give the command create iphone in workshop and it should work if you have any issues please read on the same command again because i have been using the same vm for all the workshops it's working absolutely fine if it is not working in your machine there is something wrong with your internet connection

christian can you still see my screen yes all right it says containers and dockers right yeah contains okay on a diagram perfect cool so let the setup do it work in the background and let's complete some theory part so let's start with the very basics of microservices docker and then the kubernetes so the microservices architecture splits your application into multiple services that perform fine-grained functions and are still part of your application as a whole the advantages associated with microservices such as like their elements for agile development and artifacts and then architecture that is highly maintainable and testable which will enable businesses to develop and roll out new digital offerings faster and also evolve with technology stacks makes it the

obvious choice so where do you put your micro services in containers so containers are the packages of your software that include everything that it needs to run including the code the dependencies libraries binaries and more this gives developers the ability to create predictable environments that can be run anywhere and allows container based applications to be deployed easily and consistently regardless of whether the target environment is a private data center or the public cloud or even a developer's personal laptop so this containerization is this trend that's taking over the world to allow people to run all kinds of different applications in a variety of different environments and when they do that they need an orchestration solution in order to keep

track of all those containers and schedule them and orchestrate them that's where kubernetes comes into the action

kubernetes uh is an open source orchestration platform that automates many of the manual processes involved in deploying and managing and scaling the containerized applications you can think of uh kubernetes as you would a conductor of an orchestra in the same way a conductor would say how many trumpets are needed which one plays the first trumpet and how loud each should play a container orchestrator would say how many web server front-end containers are needed what this and how many resources are to be dedicated to each one and kubernetes does not include functionality for creating or managing container images and it does not by itself run containers it needs to work with an external container source and

runtime as kubernetes is a container orchestration it needs a container runtime in order to orchestrate well kubernetes is most commonly used with docker but can also be used with any other container runtime like cruncy crio and container the another container contents that you can deploy with kubernetes and kubernetes is hot in the devops space and is now the most uh wanted platform among developers well getting started with kubernetes is easy takes like matter of minutes to set up a new cluster and run applications however the real concern or the challenge is what follows this the pivotal question of how to make sure your cluster is secure for any organization security should be like the primary

concern and not an afterthought because we all know well like especially in the current kovit environment preventation prevention is always like better than cure in fact let me tell you what happens in the real world when the kubernetes clusters aren't secured properly

on uh july 19 2019 like the second largest order finance company in the united states operates like a responsible disclosure program where security researchers can disclose potential vulnerabilities received an email the email was not a voluntary disclosure email but an email notifying them that they have been hacked and the company was asked to pay like 80 million civil penalty for its role in the cyber security breach and this is like capital one and uh there is another one like with the values of cryptocurrency skyrocketing and the limitless compute resources located in the cloud hijacking resources has become a lot more lucrative than stealing info and world's famous automaker was one of the earlier victims of crypto jacking when a

kubernetes cluster was compromised due to an administrative console not being password protected it's tesla these are like a few few examples and what was interesting about this attack was the number of in genuine precautionary measures taken to avoid detection in this attack the attackers made sure like the mining script didn't use enough cpu resources to cause an alarm or to get detected and they also used a non-standard port making detection based on port traffic virtually impossible and we'll be looking at it in today's workshop as well so these are just two examples of many incidents but there are many other incidents like this well now that we have understood the danger space in our industry and given

the knowledge gap among teams and the lack of solid security measures to protect humanities you might be wondering how in the world are we going to secure all these moving pieces and stop these attacks and that's exactly what we are going to learn today

could anyone any issues with the setup or is it all like going fine

so is everything fine can you all see the screen where it says this one and make sure you copy this output into a text pad or some document or just leave the screen as is because we'll be using these urls going forward so what i want you all to do is just copy this ip address

put it in your browser

and i have already provided the username and the password in the document however the username is admin the password is busan you should be able to log in and once and this ip address will change for each and every one you will have your own ip address in your local box so just click on the infra and you will have something happening like this which is like which will be setting up our infrastructure and once if you reach to a stage where it says proceed and abort just leave this one as is don't neither click on proceed nor abort at this stage i'll tell you when we have to click the next uh coming so

are you all good till here can you all see the screen

so if you are till here you are like pretty good to go ahead with the rest of the workshop

can you cue uh somebody sent me a message saying that uh stuck on installing build server like i mentioned that step will take some time because it will pull down like so many docker images and containers and it needs to have some good internet connection but don't worry let it happen in the background but still we can go ahead with the rest of the workshop

can you all raise your hand or give some thumbs up to go ahead with the rest of the workshop all good

so some people are still set up is happening their machines however i would like to cover the very basics and very high level uh commands of docker just to ensure we are all on the same page and then we will see some very high level commands of the kubernetes as well so if you open the documentation there is a section it says like uh common docker commands so i'll be showing the same stuff one more time but if you have any difficulty typing them you can just copy and paste them in your screen

right so the first command that i'll be using is like to find out the docker version in my machine it's a docker iphone iphone version and for example if you want to pull some image we use the command called docker full and the name of the image for example in this scenario i want to pull something called httpd or maybe if i want to pull something called alpine so this is how you use the command docker pull and the name of the image that you would like to pull or for example if you want to search for any image we use the command called docker search and the name of the image for example if i want to search httpd

we use the command called docker search and the name of the image

and for example if you want to run any container we use the command called docker run and we provide a name for that container for example http

and we will be providing like some port numbers and then we give provide the image

so this is how we are running an http image which is like a web server i'm just canceling it and if you want to see like all the images that are running inside your machine the command that you use is like drps and if you want to see all the containers that ran and that are shut down we use the command or we pass the flight icon a so this is the one which we ran a few minutes ago and for example if i want to remove that one ask the command docker container rm and that's how we delete our container and i in the pdf i have also given some additional commands like for example if you want to

exec into one of the containers of youtube or if you want to stop a container or how to kill a container everything is provided in documentation so just keep or just practice and try those commands after the workshop and another command that we would like to show you is if you want to see all the images that are running inside the machine we provide like the docker images it will list all the images that exist inside that machine

and another important command is like if you want to see the logs of any container we pass the command called docker logs and you provide like the name of the container it will provide logs of all the containers you can try with any of the containers that are running in your machine

and there is a section called our docker kung fu which are like some good useful commands which we'll be using in the real time for example if you want to run or get the names of the containers that are running we use the logger ps and we are formatting it using go templates and if you want it in the table format so container id and the name of the container

cool so that pretty much covers like the very basics of docker i don't want to dive deep inside the docker stuff we will be covering the contain we will be covering most of the stuff in the communities and we will do some basic commands of the kubernetes as well

right well there might be some kubernetes experts here this content may sound a bit redundant to you but i want to make sure we are all on the same page so i want to start by explaining some important relevant components of kubernetes which we will be attacking in today's workshop so like you see on the screen these are like the various important parts of the kubernetes well kubernetes follows a client server architecture model and the working kubernetes deployment is called a cluster you can visualize the kubernetes cluster as two parts the control plane and the compute machine or the nodes it is possible to have a multi-master setup but by default there is a single

master server which acts as a controlling node and it is like the point of contact i'll explain you what that means so what happens in the kubernetes control plane so control plane if the control plane is the brains of the operation the worker nodes are the muscles so let's begin in the control plane in the nerve center of the kubernetes cluster so the control plane here we find the kubernetes components that control the cluster along with the data about the cluster state and the configuration these core kubernetes components handle the important work of making sure your containers are running in sufficient numbers and with the necessary resources the control plane is in constant contact with your nodes

if you have configured your cluster to run in a certain way the control plane make sure it does and the next important component is kubernetes api server cube api server so if you need to interact with your kubernetes cluster talk to the api the kubernetes api is the front end of the kubernetes control plane handling internal and external requests the api server determines if a request is valid and if it is processes it you can access the api through the rest pulse or through the cube cdl command line interface or through other command line tools that are like q am and many other tools and what is cube scheduler does is your cluster healthy if new containers are needed where will

they fit these are the common concerns of the kubernetes scheduler and the cube controller manager the controller takes care of actually running the cluster and the kubernetes controller manager contains several controller functions in one one controller consults the scheduler and make sure the correct number of pods are running and if a port goes down another controller notices and responds so another important component is hcd so configuration data and information about the state of the cluster lives in that city it's a key value store database it's fault tolerant and distributed and hcd is designed to be like the ultimate source of truth about the kubernetes cluster now let's see what happens in the kubernetes node the worker nodes the

worker nodes are the muzzles they run and control all the pods and containers from your cluster you can have zero or more worker nodes on your cluster although it is not recommended to run your parts on the same node as the control plane the main components of our workers are the cubelet the container runtime interface and the q proxy as shown in the image let's discuss what the pods are a pod is the smallest and simplest unit in the kubernetes object model it represents a single instance of an application each pod is made up of container or a series of tightly coupled containers along with options that govern how the containers are run and next important piece is cubelet

each compute node each worker node contains a cubelet it's a tiny application that communicates with the control plane the cubelet makes sure the containers are running in the pod when the control plane needs something to happen in a node the cubelet executes the action and the last one is the cube proxy each compute node also like contains a cube proxy it's a network proxy for facilitating kubernetes networking services the cube proxy handles the network communications inside or outside of your cluster relaying either on your operating system's packet filtering layer or forwarding the traffic itself

so we will also look into one quick stuff now let us visualize how an attacker will look at like different components of the kubernetes architecture so when an attacker has to target a map master like in the previous diagram we have shown like the diagram shows all the communications go through like the kubernetes api server this is what defines and controls all of the kubernetes management and operational functions it is generally exposed on every deployment since it is needed for management purposes you know what but exposing your aps server to the public is the most common entry point for attackers it is actually a really juicy target malicious attackers or actors will always try to get access to the

kubernetes api server and the control plane and once it is compromised they can then proceed to compromising the whole cluster at times it may not be a bad day for just your cluster but also your underlying cloud account or the underlying infrastructure where the cluster is running and the next important component that will be attacked is like hcd it's a key value store and the core component of the kubernetes cluster and it's the main data storage of your location of the cluster this means that all of your cluster objects are saved here fcd is considered like the source of truth for kubernetes it is used to store highly sensitive configuration data but it is also easily left unprotected

anyone who gains access to xcd targets to retrieve service account tokens and secrets and once if an attacker has access to privileged secret or a token then it's game over a very quick win for the attacker we'll be looking into that stuff during the workshop and likewise pretty much the attackers will be targeting the cubelet like we have seen on the node like cubelet exposes to uh endpoints read and read write and point and read only endpoint and we'll be looking into attacking that stuff and the next important thing is like container runtime this component enables the functionality required to start run and manage containers on a given node i can't stress enough how important it is to

secure your workloads to the containers a privileged container when gained access gives an attacker the privilege to run the command in the context of the container and the option to escape and access the host resources this is an end game too an attacker who gains access to the container has gained access to your cluster and even eventually the underlying account in today's workshop we'll be looking into all the stuff stuff like the initial access like gaining access to the kubernetes cluster by using like anonymous access or an insecure port or cubelet read write ports or read only ports and we'll be looking into abusing a web application and gaining access to the underlying kubernetes cluster via a

pod and we'll be looking into like executing into multiple different containers or doing some remote execution inside the stuff and we will also do some uh persistent techniques such as like back door containers or malicious ignition controllers and we will be doing some privilege escalation techniques and uh we'll be looking into defensive and techniques and credential access and we'll be performing some lateral movements and collection and then we'll be covering most of the stuff in today's workshop so now let's quickly start with some very basics of the kubernetes commands

so once i want you to all to start by just running this command and it will ensure like it will give you the information it will say that your cluster is up and running if it is up and running it will give you the information about like your control plane the ip address on the port number and also like the cube dns are you all still here do you have any issues with the page am i going too fast or too slow just raise your hand if it is everything is fine and we are good to go ahead with the rest of the workshop and we are pretty good with the timing we are on pace

so once this section is done we will have a quick five minutes break every one hour we i will give you like a quick five minutes break to revive and then we'll be continuing with the rest of the stuff just raise your hand are we good to go

perfect good to go well the documentation we have all these commands guys so in order to identify like your kubernetes clustering for that is the command we use it's pretty much the same in the dot for docker we used the command called uh docker client docker and then we pass the verb like command run get something pretty much we'll be doing the same thing for kubernetes we'll be passing like the cube cdl client and everything by default is installed on your machines on this workstation you don't need to do anything i have set it up everything for you so we use like the cube ctl and we pass the work get to get some information and

then we pass the resource name it could be like the pods or the nodes or the name spaces whatsoever for example if i want to get the information of the pod cube ctl get bored is the command if i want to see all the pods i will pass the flag iphone uppercase a it will give me all the ports that are running inside the cluster at this stage you may not see all these containers but don't stress you will be looking these many containers going forward so likewise instead of board i can give something like node which will give me like the information of the nodes that are running inside the cluster which means i

have one control plane the master node and i have like two worker nodes for the workshop pretty much you can get any information like for example deployments so use the command called cube ctl get and one last one if you want to see the name spaces we pass the flag get namespace iphone 8 will list all the namespaces inside the cluster and for example if you want to exclusively get the resources in particular name space we pass the flag called hyphen n and the names for example this command says hey please get all the parts that are running in the name space called hube system so we pass the flag guy to n and cube system this

way i will be getting all the pots or i am filtering the ports that are running in the cube system namespace

and another important command is like if you want to see the configuration of any of the file

we pass the flag called uh hyphen o ammo or iphone or json for example cube cdl get port in the cube system name space i'm giving this one in the format i want the output by amble format so i'm passing the flag iphone or ammo which will give me like the configuration of that pod in this gamble format so we'll be looking into this stuff going forward but that is the command that we use if you want the manifestation file in the json format we've passed the flag iphone json which will give everything in the json format we'll be using these commands in the workshop and for example creating anything we pass or we use the work called cube ctl

create for example in the scenario i want to create called namespace barcelona it says name says created your cdl get namespace cool the name space barcelona has been created six seconds ago so that's how we create something and for example if you want to create some deployment we use a command called cubectl in the namespace called besides or maybe i will use the barcelona which i have created deployment functions it says deployment created we will see the deployment using the command ctl get deploy fnn barcelona it says it is coming up so that way we create the deployments and we can also scale the deployments and we scale in like scale out by passing the command called iphone hyphen

replicas you have that command in the pdf document you can just try it and for example if you want to edit any of the deployments use the command called cubectl iphone barcelona edit deployment for example this one will give you an option to edit for example if you want to change the name of the image or if you want to change some specs or the flags or if you want to change anything else you can change here

and another important command is like a qcbl auth can i this will tell you what kind of privileges you have inside the cluster for example cube cpl or can i and then i pass the verb called get the resource name secrets it will we are asking whether do i have the privileges to get the secrets or not it says yes i have privileges to get the secrets likewise sports yes so this is how you check for the privileges cube cp a lot can i or if you want to see the completely stock privileges that you have got use the command called cube city log can i iphone list so this way here it says you have like

star on the non resource urls and the verb star which means you can perform almost everything because you are an admin on this cluster at this stage

and the next command is like we'll be looking into some secrets cube cdl get ns and for example if you want to get secrets in the name space called uh cube system you'll see they'll get secret iphone and cube system it will list all the secrets that are running or that exist in the name space and if you want to describe or if you want to get the token of any particular secret use command called troop cdl describe secret and any random one for example name of the token and then we pass iphone n the name space it will help us to describe the secret which will eventually give us the token and using this token you can pass it to

the rest api to authenticate and then for authentication authorization and you can start interacting with the cluster so you'll also see how attackers will abuse these things and at times during the deployment or something we may have some issues and that's not if you want to identify what's going wrong with any deployment or any payment set or any form use the command call describe cdl describe for example i want to describe this for something called and i will provide the name it is in the name space called monitoring so that way it will describe in case of any issues it will tell you what's going on what's going wrong with this particular part or the deployment that way this particular

command will help us to troubleshoot any stuff cube cpl described and needless to say another important command is like cube cdl logs this will give us the logs of any particular power cube ctl so that is the command that we use ctl logs in the name of the particular pod it will give us all the logs of the container and another important command is like cube ctl get events which will help us identify all the events that are happening inside the cluster it says unhealthy something is unhealthy so let us see what's in healthy cp's here hey says pod falco exporter it says something is healthy but it is up and running arenas probe failed some

issues so that way you identify the events that are happening inside the cluster and i have like few other commands listed in the documentation guys if you want to exit into one of the running containers how to do it and another important piece that i would like you to know is like the cube ctl config so every time you issue a command using huge ctl and the way it is interacting with the kubernetes cluster there is something happening in the background so in order to understand what that is it says you are interacting your current context is this the current context and there is something defined in this current context and we will see what that is

cube ctl config view it will give you the information of your config file so here it says this is the cluster that you're interacting with and this is the user that you are interacting with and this is the name and this is where it has like the client certificate and client key data so every time you want to interact with the community's cluster the tube ctl client will ensure there is a acute config exist and it has like defined certificates to ensure it has proper authentication authorization in place

pretty much and there are like so many other important commands for you to try just uh test them and play with them and if you want to identify what are like the total resources that exist inside the communities we use the command called huge cpl apif and resources so these are the different types of resources that exist inside the cluster cube ctl api and resources it says like the bindings the config maps endpoints events image ranges namespaces nodes pods secrets service account services these are the different types of resources that exist inside the kubernetes cluster and these are like the short forms instead of getting something like namespace instead of typing the full word namespace we can just type ns

and if you want to get something a good appropriate output we use the command called cubectl api resources iphone or wide it will give a proper and more information like what verbs applicable to that particular resource iphone white is always useful in some other scenarios as well for example if i want to get cube cdl get nodes i have no y will tell us like the name of the node the status of the node the version and the ip address and where it is deployed on which it is actually running will give all this information so it's always good to use the command iphone y

cool that pretty much covers all the basic stuff guys uh we have we are almost close to like one hour we have set up the workstation we have seen like the basics of docker and the kubernetes and the next one over we will be completely working on like the attacking stuff and then we'll be looking into like the defending and then we will have different kinds of monitoring and then runtime monitoring and container scanning all these kinds of stuff so we will take a break for five minutes and we will catch up in the next five minutes again is everything good anybody any questions feel free to share them on the slack channel or leave a message here

if you feel that you are missing something anything and everything that we did so far and whatever we will be doing for the rest of the three hours is everything is mentioned in the documentation if you have missed the recommendation it is available in the slack channel please download it from there

i'll catch up in the next five minutes thanks

all right guys i'm back so let's start with our attacking and defending modules so we have like close to 13 scenarios in this one so we have like scenario one two three and four so you also have uh the solutions for the scenarios you can just uh try them if you feel that i'm going slower going fast you can just try it yourself uh you also have the solution so let us start with our scenario one two three and four first and then we'll go ahead with the rest of the ones the first our three four scenarios are pretty straightforward so we will see what that means

so in the scenario one we'll be looking at the stuff called anonymous access

so this anonymous access is not enabled by default however sometimes some engineers like me in order to do some research or to make their lives easier they can create an enormous user or give some anonymous privileges and they eventually forget closing the stuff so if there is an anonymous access enabled in any cluster we will see how to identify it and what we can do in that scenario so like the first thing uh to realize when thinking about securing cloud humanities based clusters is that there are attackers and bots constantly searching the internet for exposed api servers and posts of ports that are related to kubernetes it is critical that the cube api server is not left publicly exposed

although the default setting is secure but exposed api servers are still the main entry point for attackers to compromise the kubernetes cluster so we will see here so the first thing i want you to try is issue the command call called cpl cluster iphone info the command which we have seen before it will give you some information about your cluster and the next command that i would want you to try is like cubes if you will get voip and white and just make a note of some important containers for example these are the three we have like work control plane so let us cube cd we will get voip and white iphone n q iphone

system so i want you all to identify the control plane just make a note of this ip address 172 180.5 that is the ip address of your control plane so in order to go ahead with the rest of the workshop i want you all to quickly install nmap so just copy and paste the commands that i have given in here in the pdf and map is already installed so we will quickly scan

so this is the ipad res of the control plane and and if you google or just do a bit of research like what are the important kubernetes ports it will give you the list of posts so to make your lives easier i have already given those ports for you let me [Music] just run the nmap command on these ports for now

so you will see an output something like discover open port on four double six three two three eight zero one zero two five zero we will be trying and playing with all those boats so it says like two three seven nine fcd client two three eight zero city server and four double six three six four four three one zero two five zero one two five five we'll be exploiting all the spots so usually the port the kubernetes secure port runs on either port 443 or port 8443 or port 6443 here i'm giving you like the direct solution however after enumerating a bit on playing with it you will get to know okay this is the kubernetes and this is

the port that is running so what i would do is like i will do curl to the particular port on that ip address 172.180.5 on 4643 and i also have jq installed on your machine to get a good output there you go so you are able to interact with the kubernetes cluster on the port 6443 this is ip address of the control plane and this is the port where the api is accessible and because there is anonymous access enabled which is letting you to access with different apis on this port number and what we do is like i further go and explore other stuff such as like slash api will give me like the server address and

the information the server address the stuff api or slash v1 it will give me different end points that are available on v1 like service accounts or forwards or services secrets etc if i want to further explore api v1 for example i would start with the fonts

uh can you all still see my screen just a hands up

cool perfect so and if you want to see some other stuff we will use something like secrets so that way you can use this anonymous port just one sec

likewise for example if you want to gather the information of the nodes

you will pass the flag called nodes that way it will give you all the information so when you have like an enormous port or anonymous access enabled on the kubernetes cluster it's almost like game over an attacker who has anonymous access to the cluster he can do anything and everything and we will also see the reason why or the reason why this anonymous axis is enabled so we have probably i'll show it to you after a few sessions when we discuss about the roles the role bindings inside of the kubernetes cluster why or which role has actually enabled this anonymous access and i'll also show you how to fix it so we have seen the kubernetes

anonymous axis on a secure port now we'll be moving on to like the scenario number two

so from the previous and map results

so we came across at this port we have identified this is like a secure port with anonymous access now let's try to play with port 4663 and see what's what we have there so what i would be doing is like and because this is an insecure port what i would do is call http 0.5 and i'll pass the port number 4 663 and the port number may vary for different clusters this is for this scenario i made your lives easier but any attacker who gains like initial access or has like a map access they would probe and enumerate the stuff that's what we are doing here so i'm trying to it looks like there is something i could

still access the same stuff but this is like in secure port the major difference between secure port and insecure port is like on the secure port until unless there is an anonymous access the users or the attackers won't be able to play or interact with the api whereas if the insecure port is enabled they don't need authentication they can straight away access and start playing with the cluster so what let us see what else we can do when an insecure code is there so what i would do is

463 slash api there you go the same stuff api slash v1 slash boards likewise you can get the secrets everything

even this insecure port is not enabled by default like i said there is always a possibility that somebody could enable and leave it by default it's always good to ensure you check that during your risk assessments or during a penetration testing so this covers yep just going to interrupt people are saying that they cannot some of them can see your screen can you see my i i do see your screen but there are at least three people that said that they cannot see it so if you can try to stop sharing and share again to see if that fixes the issue

can you all see my screen now

christian can you see the screen yes i can say it um let's see if people can confirm can you can you confirm slack or raise your hand if you can see this screen again

yeah that's strange uh chris you can see it right and others can't see it yeah i can see your screen yeah they say like this is like now which yeah i'm seeing just like screen now you yeah i'm saying terminal yeah it's all good i don't know why some people can't see it

okay they say visible now good now cool uh no worries guys i'll quickly cover like the scenario number two again so during the nmap scan we have identified some quotes and we have in the first scenario we have seen like how to uh attack or access like a kubernetes cluster when anonymous access is enabled using a secure port and this scenario we will be trying to access and see what happens when there is an insecure port of the kubernetes cluster is accessible via some other port calling secure port so what i will be doing i'll be doing pretty much the same stuff but i'll be accessing it via port other port 463 where the insecure

is running and uh just instead of the flag https i'll be pointing the flag http and you will be able to do the rest of the stuff pretty much like api v1 pods or nodes etc so the only difference between uh the anonymous access and secure code is like if there is no anonymous access enabled you won't be able to interact with the kubernetes cluster via secure port whereas with insecure port you don't need any access or authorization at all if there isn't secure port left by default the attacker can start playing with the cluster straight away so i'll be moving on to like the next scenario which is like uh abusing the cubelet read write port

so kubernetes if you just google it says like kubernetes read write port is available on one zero two five zero so what i will be doing is this time after running the nmap scan i'll be pretty much doing the same stuff but this time i'll try to proport them one zero two five zeros

[Music] there you go likewise if you the other is like if you want to see the only running parts in the cluster we should apply like running pods will give you all the information of the pods that are running inside the cluster so this is what you can do using a cubelet when a cubelet rewrite port is accessible inside the network so the major difference between cubelet read write port and read only port as the name implies using the read write you can read at the same time you can write by which means interact with the cluster if there is by default if this cubelet rewrite port is accessible it's almost like game over the attacker can spawn

different containers they can spawn some shells in that and they can gain access to the underlying cluster so there's a section called extra mile in the documentation so for example i want you to all to just try a few commands

your search

this way you can get all the running parts inside the cluster

we give you the containers that are running so what i would like you all to do is there is a container called xcd client and let's execute some commands using a cubelet read write port

it's inclined

so we're executing a command using the cube let's read write port on the container called lcd client there you go the command has been executed on one of the containers that are running inside the cluster and once i needless to say like once if you know when you can actually run some commands on the containers you know what to take it ahead like you can grab like dc to instance metadata or you can even like grab a river shell and you can do heaps so under the extra mile section in the documentation you have what you can try and test just follow those commands and you can even get a reverse shell and gain access to the

underlying cluster and the next one we see we will see is like the cubelet this time it's only read only port so the read-only port runs on port 10255 [Music]

25 file [Music] it's actually

dashboards so using the read write code you can always gather some information so it's pretty much when there are like other options other things that you can always try and everything is provided in the documentation and you can also explore different endpoints that can be accessible on the cube like read write port and read only for just google a bit and you can always play with different things well that covers the first four scenarios and now the game will be up we will be like chaining multiple attacks from uh scenario four we will initially try to gain access to one of the pod or the container that is running uh using a vulnerable web application and using

that web application we perform some remote code execution and then we perform some lateral movements until we gain a full complete access to the cluster and the underlying cloud account

so on your screens you will have something like this so if you go back there is something called dev environment and we also have called web application just copy and paste this url inside your applications

there you go there is some kind of web application running and let's issue some commands test ls it looks like we are getting some output test id however that output seems to be a bit encoded so we will see what is the output and how to decode it example i'll copy this one

echo 264 md so the output of this one when we decode it's like it says uid 0 0 so when we pass command call test iphone id it says the container is running with the root privileges so and we also identified that there is some kind of command injection or remote execution bug in this application so what we are going to do is we try to perform some remote execution and try to gain access to the underlying shell so for that what i'll be doing is you can either use your vm or you can also use your local host for example this is my localhost

i'll try to identify the ip address of my local host

192 168 0.3

1916 56.1 and i just try to bring it and see whether the machines are commentable

all good so like uh provided in the machine what i'll be doing is like i'll be using a python reverse shell to gain access to the container using like the vulnerable web application so for that what i'll be doing is i'll be running a netcat listener on my machine netcat iphone 444 and in the web application what i'll be doing is so there is a command that's been provided to you in the documentation just copy and paste it and make sure you make some changes like the ip address and the port number

so the port number that i used is like port four four four four

just one second

i'm just replacing the port number where i'm running the netcat and also this is the ip address of the host where netcat is running so i'll just click on encode something is happening here and eventually it should get me a shell but there we go we got the shell and being an attacker what i would do is like i will do some uh initial assessments like checking for some environment variables and other stuff so pretty much i do the same stuff i'll check for environment variables and after grabbing some things identified there is a kubernetes something which reveals us that this is a pod or this is a container or this is a web application

running on a container that is hosted on the kubernetes so from here what i would do is like pretty much like an attacker i will do some enumeration try to gather some information such as like humanity service sport and other stuff and with the basic stuff with the basic knowledge that we have learned during the basic commands in order to interact with the kubernetes cluster we need something called cubectl or cubecrate kind so what we do is we quickly download that one so i'll go to cd slash temp and if you miss some commands every command each and every command is available in the documentation you can simply copy and paste it

there you go

so let's match

hey hassan there's someone that says that one of the commands is not correct on the documentation that if you can send it i'm not sure which one it is though you can tell us like which one it was uh is it in the slack yeah he's saying on page 49.

oh sure uh this one

just one sec

and send the command you just need to replace uh the ip address on the port number and you don't need to use this one as is there are like if you search for something called reverse shell there are like so many commands or so many reversal snippets that are available on the google this is just one example that i picked up so you can use this one so what i did is like i have downloaded like the cube cuddle there you go and now i'll quickly give the right permissions to this one and i'm going to move this one to the bin folder perfect so i will quickly check the same stuff that we have tried before

cube cpl get cluster iphone info successful we are able to interact with the kubernetes cluster using the port that we gained access to so from here we try to do some privilege discussion some lateral movements and gain access all the way to communities cluster and the whole account and we will be going to use some other command that we have seen learned before cube cpu lot can iphone iphone list it will tell the privileges that we have gained on the kubernetes cluster pretty much these are the commands that we have for the parts for the secrets and for the service accounts we can get list create these kinds of stuff uh cube city a lot leo can i

get reward yes cube cp a lot can i get here yes

cubect a lot can i get secrets now it says you can get the information of the pods but you cannot get the information of the secrets it's like there are pretty limited privileges for you so let us see how we can bypass certain controls and let us see how we can gain uh complete access to the cluster at this stage we have very limited access because we are not able to access the secrets across the cluster likewise we may not be able to access many other stuff across the cluster so what i will do is

we will create a file called config i have already created that for you and guys if anybody has any difficulty copying the commands instead of copying them as is just make sure you put them into a text pad format them and then paste them sometimes copying or pasting directly from pdf may not work

right tool i have created something called config.sh so we will see what this config.sh is so it is basically trying to grab the config server ip address by issuing the commands called this is like a basic script for example if i run this same command on my terminal we'll get the ip address of the cluster likewise it will get the token name the token config and it is creating a config file for us whatever we have seen before it will create some set secrets and everything and using this file we are going to interact with the kubernetes cluster and if you observe clearly we are also trying to get the target config token call from the developer name space we

are getting a secret likewise the reason why here i'm using a secret from the developer name space is like here i'm giving you like the straight and pretty straightforward answer because after enumerating a bit inside the cluster after going through different secrets inside the cluster as an attacker and identified that this developer someone like in the development namespace or in the developer namespace there is a secret which has got a little bit high privileges so i am saying that going forward if i want to interact with the cluster please use this secret use this secret for authentication and authorization and talk to like the kubernetes cluster using this secret and try to get some more info so what i will do is i'll just

quickly run this config file so i'm in the temp folder sh config.search looks like it has done its job now let us see what it has generated for us cool there we go it has created a config file for us this is the token that we'll be using to interact with the cluster this is the certificate and this is ip and we are using name called research which i have created for the workshop so in order to use a cube config instead of like the default cube config we pass the file called flag cube config equal to config and then we use the flag called we pass like the rest of the commands cube ct will get secrets iphone a

so for example it says using this uh thing they're using this developer uh secret it says secret is forbidden user developers for the service account developer service account cannot list secrets the user does not have the privileges or the permissions tracks as a secret so but being an attacker the attacker won't stop there and after enumerating a bet he came across something called uh privilege escalation using misconfigured role or the role binding so in kubernetes if you want to impersonate a user or if you want to run commands as a different user we use the flag called iphone iphone yes so after enumerating the cluster for a while after doing a bit of research the

attacker came across a different kind of technique called impersonation so from that he identified that this service account called developer writing essay has some privileges to impersonate as a group it means sometimes this is what we do in the real time we may not give like the privileges to the users directly but we give them or we add them to a different group and we let them hey probably like a normal user you won't be able to do it but at times if you pass different flags or if you impersonate a different user you will have some high privileges so after a while the attacker has identified that this developer or someone the user called developer

iphone sa has some privileges to impersonate the group called system masters and then he tried to enumerate the stuff earlier we didn't have the privileges to get the secrets but when he is impersonating like the admin the group system masters he was able to get all the secrets i'll also teach you like the technique uh during like the role bindings how to stop these kind of impersonation attacks what will let these impersonation of attacks happen so here the attacker is successful he was able to impersonate a group called system masters using a normal user he could not get the secrets forbidden but after impersonating iphone iphone as nil as the group system masters they were able to access all the

secrets so this is one technique where the attacker has initially gained access to the underlying cloud account or the underlying container of the pod but has limited privileges but after enumerating a bit he uh identified there isn't like an impersonation access to a particular service account and then he is impersonating the rest of the commands using that service account and this is something which i came across all which we still come across in many of the penetration testings on the real-time assessments so we were able to successfully access the secrets and now we try to identify the list of pods that are running inside the cluster and these are the parts that are running inside the cluster

and after enumerating a bed the attacker came across if you identify we are still running inside of web application container that we gained access using remote execution and from this we are actually interacting to the underlying cluster using an impersonation technique so after going through all the commands and after identifying the pods he came across something called jenkins which is like a build server usually in the recent times in the organizations these build tools are like the juicy targets they're like the crown jewels for it for the attackers because they will have very high level of access to like your code base or it could be to your uh to the infrastructure where they actually deploy uh the infrastructure or

the code usually these build servers are given high privileges if you come across 10 machines and if you see like a build server the attacker will always target the build server first because if he happens to gain access to the build server it's a game over so pretty much in the same scenario the attacker still tries to attack this build server and we see how we can attack jenkins and gain access so that is like the next scenario so we identified all the servers that are running inside the cluster there is a build server called jenkins so what i will try to do is i will execute some commands into that server so you can

copy and paste the command call i so what i'm doing is it's pretty much the same thing which we have seen so far i'm impersonating as a group system masters so in the name space called jenkins i'm executing some commands or executing into the container which one it is it is like the jenkins pod knife and slash pin there we go so we are inside the jenkins server so so far using a web application bug using of application bug we gained access to one of the container or the underlying container and from the underlying container we try to identify some privilege escalation using impersonation technique and using that impersonation technique we identified a privileged border a bill server called

jenkins and we executed inside the jenkins and we gained access to the jenkins server and from here we see what else we can do so basically it's as an attacker i will pretty much do the same stuff i will check for all the environment variables there seems to be some juicy information such as my secret it says the password and the host name the jenkins port the username just make a note of them admin and the password percent and we also have something called gogs token so we have some critical information like there's some token it could be like the build server and the username and the password and if you ask me hey wasan will this

really happen in the real time yes these things still happen in the real time most of the companies they still pass secrets as environment variables or even even if they inject during the runtime you still be able to access this information and this is something the scenario which i have created based on my real experience so when an attacker gains access to a build server there are like different critical folders the attacker can always play with so one such important folder is like the jobs folder where different kinds of build jobs or unit testings any kind of jobs that might happen on the builder build server so i'm getting inside the jobs folder for the jenkins

so exploring jenkins is also off of the scope of this workshop so i have given like the straight commands so i am inside the jobs folder there seems to be one job created now what i'll do is i'll get inside that folder city docker file [Music] the job could be like some the company or the organization of the devops team is building some docker files using the job of the docker images so here there seems to be config.xml looks like some juicy information i will try and explore that file config.xml so it says it reveals me like the internal git repository this is like the ip address or the path of the internal gate repository likewise

you can gather some other critical information from here let's make a note of it so this is the ip address of the internal git repository and from here we also have something called a token so what i will do is using this token and using that ip address i'll try to explore like the internal bit or internal code base repository so that the command is provided in the document and these are very real-time scenarios that we still use while pen testing and infrastructure and this is how we change different attacks so i'm passing like the token that is said as an environment variable and this is the ip address that i'm trying to explore and what i'm doing is i'm

trying to get different reports uh that exists inside that environment inside that internal repository there we go i got some information so inside that repository there seems to be having some repo which is creating docker files or docker images some application repository some developer some environment repository from uh for infra there is some backend repos and there are some front-end reports so this is how we can explore different reports and you can always clone the repos and see some for some use information from here what i will do is i will just try to target the repo called docker file which the devops they might be using to build docker images inside the infra i'll get

inside the temp folder usually in the organizations most of the time if you target like a master repo there could be some restrictions in place like not everybody can access those reports or not everybody can push directly to the master you will have some workflows or you need to trigger some things before you push something to the master branch however many companies still don't have enough uh restrictions for feature branches or the branches anybody can try test different things on the feature branches so as an attacker instead of trying or attacking the master branch i'm trying to gain access to the developer branch so i'm cloning this one git clone ifnb dev alright i have successfully cloned it

so i'm getting inside that wrapper docker file and inside the docker file now let us see what the docker file has so here it says from httpd and when we so did the team or the dog the devops team they are building some images from based on this image so as an attacker what i will do is i'll make some changes and i'll push these changes to the master or to the dev branch and the way the workflows are designed these days in the organizations is every time you make some change to uh your account you make some push some changes to the branch it will immediately trigger the build servers and it will do some

assessments and it will run all the things all the steps or all the build steps that we have configured in the build server so we will see how we can take advantage of it how we can attack the jenkins server in this scenario so here we'll have to play some bits and pieces of some git so what i'll do is we have seen what the docker file has so i'm going to remove this one i don't miss an rf docker file i'm going to replace this docker file with a malicious docker file that i have created

let's go once again [Music] so i have given the file in the documentation you can just copy and paste it

i have pulled the docker file now let us see the contents of this docker file which we have created which we have pulled cool so i'm saying from so and so i'm using i'm saying like use this evil image being an attacker i know what the evil image does i have created this evil image in such a way that every time somebody spins up a container or uses this docker image or this image inside their infrared will immediately give some kind of a shell to my infrastructure so this is like the real time there are like so many malicious containers or crypto miners that are posted in the docker hub and other places which have been recently

deleted and it's still happening most of the times so i am i have created replace the old docker file with some malicious docker file so some basic git commands like git status it says modify docker file so git add perfect hit comment iphone test uh it says please tell me who you are okay let's it is advising to run some commands copy this one full it's done get status hit add hit comment it's like the file changes are happening now let us get push so we have successfully pushed the changes to like the dev branch and like we discussed before every time you make some changes to a dev brand there should be a pipeline which will be triggered

and will run all the steps that has been designed so if you go back the very first page where we have did the setup there is a build and deploy server for the dev environment so just go and access this url

right perfect my username is admin and the password is basan

the username and the password which we got it from here in the form of environment variables

so looks like a job has been run 30 seconds ago when we pushed it if you click on docker file let us see what has happened cool perfect it has pulled as per the docker file it has pulled the evil image from here so and these are the build steps that are by default defined in that organization so we have just made some changes to the docker file and it has pulled the docker image it has successfully built the image and it has successfully tagged as per that organization's uh naming schema and it has successfully pushed it so if you understand correctly what we did is we gain act we using like the web application we identified a remote

execution using that we gained access to the underlying container and in the underlying container we gained access with limited privileges but then we enumerated a bit and identified there is some impersonation techniques impersonation uh impersonation vulnerability taking advantage of that vulnerability we try to get access to secrets and some pods and from there we identified a privileged board which is called jenkins we gained access to the jenkins and we targeted one of the bills inside the jenkins we deleted and we created some malicious docker file and then we pushed the changes to the dev repo and then it has immediately triggered the bill and it has built an image based on the vulnerable image that we have defined in

the docker file and then it pushed it to the company's internal repository so now what happens is every time a devops or a developer whenever they use this image inside their infrastructure when they spin up application or container based on this image it will immediately give access to to the attacker that's what the attacker has defined in that uh evil docker file so let us see like how that can actually be exploited

so from here i'll just quickly exit from the docker machine and i'll draw everything from the vm the workstation to make the lights easier because here at times we are not use some commands like control c control v or other step so i'll be using like the vm but pretty much once if the attacker gains access to like one of the containers let us assume we are doing the same stuff so what i'll do is i'll get back to the vm so i can see all the pods that are running inside the cluster so these are the ports that are running so what i'll do is i will target this one called web app so

cube ctl get po and this is running in the namespace called developers cubectl iphone developers i would like to get the manifestation file of this one so that is the command which we have learned at the very early stages we used we passed the flag iphone or ammo

so i will place this one into web app dot ammo cool now let us see cat by bob dot camel we has been generated i will edit this file

so

what i'll be doing is i will replace this image with the malicious image which we have created a few seconds ago so and everything is dynamic guys this is like happening in the real time so this is the image this is the thing which we have created and did it like just a few seconds ago it says like 30th september 2021 944 my time so it's just happened like a few minutes ago four minutes 36 minutes ago so we are going to use that image here and i'll also pass some commands

being an attacker i know these are the flags that i have passed through that malicious image so what i'll be doing is i'll be providing the attacker id [Music] i will just get a netcat listener in that cat iphone 7777 and my ip address is 190 to 168.50 6.1 [Music] just quickly check it

192.166.56.1

only to 168 56.1 everything is good install is fine so i'm going to save this one so before deploying i'll just check that one again add to about.yaml so we have replaced the image with the malicious image that we have created we are passing some additional facts so let us apply this one for applying this deployment the command is cube cpl apply iphone f the pop dot ml so the moment it is deployed we should get a shell on this one so this is what happens so this is one example but usually in the organization the devops team will build but there is something wrong

[Music]

i mean one second guys i'm just checking it

where is the ever a pod someone

i'm checking for the error just give me one second

well looks like i missed something i don't know about ammo

command

okay

looks like i missed something let us try this one otherwise we will move on to the next scenario we will tackle this in the last cdl apply and i've adopted a camel or still there is something wrong

so this except one set at 777 [Music]

or i'll try to do this one last technique if this doesn't work we will move on to the next one we'll get po n

all that we are targeting this this one let's see here

uh sorry guys just a small mistake so here i i generated a manifestation file for the pause what i should be doing for the deployment because i'm making a change for the deployment so cool so cubectl getdeploy fna so this is the deployment so cube cdl get deploy and then [Music] developers get deploy iphone perfect so what i will do is some evil dot yaml at evil.ml perfect now i will edit this file perfect so i am going to change this image

image that we created which is like tag assassin [Music] perfect

and we pass some odds

should work perfect control x yes cpl apply for nf dot ammo oh there you go this time it looks like it is successful it has applied qpl get voi looks like it is terminating the previous container and we have like the new container coming up so once the new container is up and running hopefully we should have the shell on the netcat listener fingers crossed

here it may [Music]

so yeah the container is up and running did we get the shell let us check for it id there you go we got the shell on the netcat listener so this is what happens so this is one example but let us assume this is like a golden image or this is like the base image that the teams build in their infrastructure so every time any developer or any different teams if they consume this base image this vulnerable base image inside the infrastructure and if they happen to deploy it it will give like the remote access to the attacker anytime or this is one example or sometimes instead of building this kind of reverse shell that the attackers may

also build some kind of crypto mining images which will do crypto mining inside the infrastructure it's not always about gaining access to the infrastructure or gaining access to the data but given like the huge compute resources located in the cloud crypto mining is one form of stuff that is happening these days so well that successfully concludes this scenario i would like to just reiterate what we have done so far if you're using a web application remote execution vulnerability we gained access to one of the container and then we identified that it is running on the kubernetes cluster using the environment variables and then the attacker had very limited privileges to that container and then after enumeration he identified

that the service account has impersonation privileges and the attacker could impersonate as the group system admins and using these privileges he identified one uh crown jewel which is like the build server called jenkins and then he executed inside the jenkins he gained access to the jenkins and then he targeted one of the builds that are running inside the jenkins server he made some uh changes to one of the build file he removed a docker file and attack and created a malicious docker image and it immediately triggered the pipeline the pipeline has built the image and pushed the image the vulnerable docker image to the docker hub and what the attacker did is like just to uh show you an example we

changed one of the deployment and used the vulnerable image that we have created and it immediately gave reversal or the access to the attacker so it can also happen in different ways like i mentioned the attacker can also create some crypto mining and a crypto mining image and anytime the devs or the devops use different images it will lose it will start cryptomining in favor of the attacker well that concludes this uh one now let's move on to the next scenario this is again uh some interesting stuff like if using like the docker socket mount like i said uh the kubernetes always needs some kind of a container runtime that is up and running so

securing kubernetes cluster or securing docker runtime the container runtime is as important as securing the whole cluster so let us see an example how an attacker uh if the gains access to one of the pod and if there is like a docker socket mounted and how the attacker will take advantage of it so if you ask me hey bassan will this really happen in the real time will that docker containers have like docker socket mounted yes because any of the monitoring tools for example it could be even like security monitoring tools or log monitoring tools will always have this docker socket mounted onto the containers in case they need access to the underlying post or to

the underlying locks let's quickly look into that scenario how we can abuse docker socket mount so i'll quickly get into the temp folder we are good guys we are still on time we are on track nothing to rush so that should be i will be creating and downloading a file which i have created for you so just copy and paste that if that doesn't work just let me know

so just quickly apply this one

it says there is a pod called monitoring created we try to attack uh this monitoring port and see so before that i would like you to understand what says so what we are actually doing for the scenario is like we ourselves creating uh one test environment we are deploying a pod called container called monitoring and the base image is alpine however what we are doing is we are mounting the docker socket where slash run folder of the host onto the container so what this means we will check it so cube ctl get po yep the pod monitoring is up and running now let's quickly exit into this pod cube cpl exit iphone id monitoring bash awesome perfect we are inside the

board so after enumerating the folders for a while the attacker gets into the web slash run folder and then he identifies something called docker.socket so after uh doing a bit of research the attacker gets to know that when there is a docker socket mounted onto a container he identifies how to play with it and how to gain access to actually the underlying post so let us see what happens so you can use different ways to interact with this uh docker dot socket you can use curl and then you can use different other clients but based on my experience what i would simply do is i would add the docker binary to make my life easier it's just one simple command

apk add docker just remember that you are running inside a pod you're not on the host so we already have a docker running on the host however now inside another container we have installed the docker binary again so this is called docker and docker attack sometimes so now using the docker client that i have installed i'll be showing the all the basic commands like docker images there you go we are running inside we are running inside a container however we are able to access with the underlying host and the underlying images so from here needless to say what you can do it's it's completely up to you you can run different containers you can grab reverse shells you can

delete images and also one thing i would always do is like exfiltrate the images for that what i would do is you can just uh simply docker use the command called docker login using your personal credentials and once your login you can just steal the images that in this corporate world you can just steal them into your private account upload them into your private account and then pull them onto your local machine sometime later and do all the acceleration stuff so how to do it everything is given in the document so you can go through it and you can try the stuff these are the basic commands that we have learned before like docker pull docker inspect you can play with

all the stuff and you have the commands exactly to do it in the documentation so i will move on to like the next scenario which is like now we have so far we have seen attacking the master nodes now we will try to attack the worker nodes so we will see how we can actually gain access to the worker nodes i'll quickly exit out of this i have created a file for you [Music] just download that one from the github you can use this url

i will show you how to gain access to one of the worker nodes that are running inside the cluster and these attacks may not be just be done by an attacker is always attacking from an external party or it could always be some pentesters coming inside your intro or even the developers who actually work with you like day in and day out if by mistake if you give some additional privileges developer can always abuse them developers are like techie savvys they are way more uh technical and knowledgeable than the pentesters or the security professionals because they are the one who actually built the stuff so let us see what we have on the shell on node.sh

so basically i'm just not doing nothing but creating a power or a container i'm using the command called an image called nsender and i'm using i'm leaving it to you like what ns enter means what innercenter does just do a bit of research but also i want you to understand that i'm passing a flag called privileged equal to true

so what i will do is first let me identify the nodes that are running inside the nodes okay for example let us say i want to gain access to like the master node so what i will do is dot sh and just pass the node and hopefully after some time in a few seconds we should get the shell on that one oh cool there we go it says now we when we run the command we are running from the host but now we are inside the control plane we got the shell on one of the worker notes so this way the depths the devs who have like limited axes or even a pentesters who had like limited axes or

even an attacker can always gain access to like the master nor or any other nodes so being an attacker once again access to the master my target would be to exfiltrate and explore different sections different folders first name

id there will be some juicy folders on the master node this is very good to keep and note series slashes slash kubernetes and well there you go these are some juicy files for you to keep an out of it just keep exploring the stuff admin.com will give you huge information about you you can have literally like you can take a copy of this config file and like we have seen before you can pass this config file and keep interacting with the cluster or there are some other stuff like cubelet.config manifestpki i'm leaving it to you to explore what this does to what extent you can go using this information i'll just quickly exit so likewise

if you want to attack some other thing for example let us quickly see how we can gain access to this node bash and spinning up same command in the next few seconds i should have a shell on the other worker node there we go and this time we have like the shell on the worker node so i'll leave it to you to export the rest of the stuff so that's how you can actually gain access to like the worker nodes directly in sometimes when you don't have access to like the master node or the api server you can directly gain access to this one so now the other important stuff is like the xcd we will see how to attack the

xcd client this is like another uh critical uh thing for the kubernetes cluster so the setup has already been done for you so if you give the command call cube cpl get po there is something called lcd client let us see i have given some more commands how to interact with that cd client we use a binary call at cd ctl like cube ctl we use that cdcdl to interact with that cd client so we passed the commands there we go flying and this is the server and now let us try to extract some keys from the xcd client there you go it is listing all the secrets that are available inside the xcd so again pretty much what i do is

like as an attacker i try to enumerate each and every ck secret and find out which secret has got like highest privileges uh how i would do is you can just write a small bash script or a python script to automate the stuff so cube system for example i want to explore one of the secrets in the cube system for example this one pod garbage collector that way you can you will get in the namespace this is the token so pretty much you can keep exploring like different uh secrets and enumerate and keep hunting for the token which has got like high privileges so this is how you can attack or exploit the fcd cluster and there are different

other things that you can still do with the lcd stuff but um i'm just leaving it to you to explore the rest of the stuff and there are like few other techniques that i have listed in the documentation please try those steps as well

and we have one important thing called how like the crypto mining attacks happen but i would like to show like the crypto mining along with the monitoring stuff that way we can also attack and also monitor at the same time so i will leave that scenario to the last one so well that pretty much covers the attacking scenario and after this we will be looking into like the defending stuff we'll be looking into like the role-based access controls we'll be looking into network policies we will be looking into security context admission controllers and especially like the open policy agents how do we use them for defending these kind of attacks inside the cluster so i just need to have a quick five

minute break and i'll catch up with you in exactly five minutes and we will go ahead with the rest of the stuff

uh i'm back guys let's start with the role based called our back

so well uh our bike is like a key security feature that protects your cluster by allowing you to control who can access specific api resources like we have seen before uh the attacker was able to impersonate as a different user and we also able to access the cluster anonymously without any authentication authorization we will see how we can actually restrict these things using our back so in cuban means everything is a resource the pods nodes services service accounts and all the rest but these resources don't have ownership of permission attributes instead there are like additional levels of abstraction called a role which will define the rules that specify a set of resources and also set of words

like the actions that one can take on those objects and the role binding links a role to our identity this might be a user a group or a service this will actually complete the whole part so the roles and the role mindings apply to the name space level there are also cluster wide equivalence called cluster roles and the cluster role bindings as the name implies cluster role in the cluster rule mining supply and the clusterable so let us play with some basic stuff the roles and role bindings so

cool so i want you to all to get into this folder which says like root code base and then the files and then we have a folder called powerpack we'll be looking into some different stuff and so let us start by we have already seen this before however i would like to give you an example let us start by creating a name space called our bike example in the namespace in which we will be trying all the stuff and i will create a service account called a dev user cube ctl create service account in the name space our bank example called dev user likewise i'll also create another service account which is called admin user cool now i have a name space called

our bike example and two users a dev user and an admin user and also we have seen in the document i have provided if you want to impersonate we use the command called iphone hyphen as so i'm going to set some alias so going forward if i issue the command called cube ctrl iphone admin i am giving this command as a normal user as an admin user i'm setting up another alias are back i think example which is like see if i run the command as uh cube ct life and user so i'm running the command knife and as system service account as a dev user so going forward if i issue something called cube ctl

life and user i'm running in the context of dev user if i give cube city life admin i'm running as an admin user so just to make the lives easier to make the command short i'm just setting up values so let us try some quick examples as an admin cube sql admin get po yes because an admin have all the privileges and they can get everything now let us try the same stuff using a cube ctl as a normal user not because the user cannot list their user does not have enough privileges or let us see remove iphone a and just see like the bots no still the the user doesn't have enough privileges so what do we do how do we provide

privileges to your user or how do we delete some privileges to you so we will be looking into this stuff so i have already created certain examples for you let us start with one called role let us start let us look at an role so let us see what this role has so in the kind role like i mentioned you provide the resources to what resources and what kind of actions a user can do so this role says that anyone who has this role attached will be able to list and get secrets so next one is role binding so what this role binding says is i have created a user called dev user in the namespace called arbitration

example and attach the role that we are creating to create role get secrets above to this particular user so when this is implemented the dev user will be able to access secrets list and get let us just quickly apply them and see so cube cdl and admin apply secrets i initially applied the role now i'm applying the role binding now the user should be able to get or the normal user should be able to access tickets there we go cube ctl user get secrets earlier he was not able to access anything but now he is able to access the secrets because of the words list and get secrets and if you observe one thing we have restricted the privileges

only to that namespace the attacker can access the secrets in the name space which we have created our backlight for example he won't be able to access secrets in any other name space or for example if you just pass the flag iphone a there we go still the attacker cannot commit the user the dev user cannot access the secrets because it does not have privileges so this is how and this is where we actually use the role based access controls to limit the access this is pretty much like the users and the groups that we have in the active directory this is at the namespace level now let's look at a quick example at the cluster

level so let us i have a cluster role called get ports which says we we didn't define the name space name space is only since cluster roles are not name space so to anyone to whom this role is attached they will be able to get to watch and list at the ports across the cluster level and we also see called cluster role mining so i'm saying hey attach this role whatever i'm going to create about the pod reader to the user called their wife and user who exists inside the name system called l back as an example so let me quickly apply these things

i have created the role now i'm also creating the role binding and now let us see as a normal user will he be able to access the pods or not there we go initially the normal user the cube ct like a normal user or the dev user couldn't access the ports but now because of this role binding he was able to access all the ports so that's how we actually use the roles and role bindings to restrict the access so previously there was a scenario where there was like anonymous access to the cluster so how did that happen i have created a file called anonymous access so in this scenario if you look at there

is a role called cluster admin role which comes with the kubernetes cluster by default every time you spin up a kubernetes cluster you will have this role called cluster admin and what i'm doing is i'm attaching this cluster admin role to a group called system anonymously a user called system anonymous that is the reason why the attacker initially we were able to hack or play with the kubernetes cluster using anonymous access so in order to see that stuff cube cdl get roll bindings we are exploring the rule binding the iphone a so these are the role bindings

uh it should be cluster roll bindings

there is something called anonymous review access which we have created let us see qcdl get the role binding there we go so this there is something that exists inside the cluster it says like hey we are providing like cluster admin access to like the user system anonymous that's the reason why it was able to how we how we actually prevent from this one just keep enumerating and testing for privileged roles and eliminate these kind of holes if you relate this if you delete this role binding you won't be able to anonymously access the cluster i will leave it to you guys to test it and i'll also tell you how to actually implement these changes and test it

going forward please hold on so that's pretty much here we are covering like the roles and role bindings at the name space level at the cluster uh level and we also have seen anonymous level and i will also leave it to you guys how to avoid the impersonation stuff [Music]

there's something called developer role mining let us see something else i mean

oh yeah pretty much so that is how you actually exploit or uh take advantage of the additional privileges and these are the different ways you can actually restrict using like the role based access control to the cluster so uh let's quickly move on to

now let's move on to like the network policy stuff well basically the kubernetes network policy lets uh administrators and the developers enforce which network is allowed uh using the rules so kubernetes by default allows power to board uh pods to communicate amongst uh themselves for simplicity however you can always use network security policies and some interest rules and force a tiered architecture within the kubernetes well in simple terms network security policy and kubernetes are like a simple example or simple replica of like the firewalls to let or block the traffic so we will get into some examples of network policies this is going to be an interesting example so let us quickly look into one file and

see what we actually defined there and before that i will quickly delete cubectl delete namespace our hyphen example which i have created and that will delete all the even the resources that we have created there

less

so let us see explore this manifestation file network demo app this seems to be pretty big file but it's very easy if you understand what i'm doing is i'm creating one deployment which we have seen before i'm creating a deployment and the name of the deployment is like api deployment and this is the image that i'm using so this is one deployment and i'm creating a service for this deployment services something which will let us access the deployment that we are creating so there is like another deployment called nginx deployment and i'm creating another service for it so only two things that i would like you to note in this point is like the api deployment is

accessible on the port 3333 and the nginx deployment is accessible on the port 8808 80. so i will quickly apply both this file

so it says api deployment created api service created nginx deployment created engineering service creator it may take just a minute or two to bring up this parts cube ctl says container creating nginx container creating it shouldn't take too long the api api deployment is running and nginx still creating [Music]

so in order to see the some flags we can use pass the flags knife and selector app equal to api so api is running and i'll pass one more thing engine x okay we have like both the deployments that are up and running nginx and the api that we are doing so we will just check for a connection from the ancient export to the outer world cube cdl exec and i'll choose one of the nginx pod

perfect from nginx we were able to reach the external world now let's see if we can reach out to the api from the nginx remember we have created like the api service on port 3333 there we go we are able to reach both api and the external from nginx likewise let us check also the connections from uh the api deployment good cto api deployment and this is where we go and i will try to see if the api deployment has access to the external world yep even the api has access to the external world also let us see if api can interact with the api itself

perfect there we go so from nginx deployment we have access to the api on the output like even from the api we have access to uh nginx and api and the outer world now let us play with some rules and see how we can actually restrict the traffic inside the kubernetes cluster so the first one i would slay or play with is like the address rule i have created a file manifestation file for you let us see what this means so here we are saying we apply this one address to the match to the api port 3333 only to port 53 to the outer world let us see what this means qcdl apply fnf address

network policy has been created now let's quickly

click on mention export let's try to exit the outer world

we can't access google now let's try and access the nginx service on the port 80. now click on access however let's try and access the api service on port 3333 there we go we were able to access so what we defined the rule is like hey if somebody is trying to access to any of the applications with the label called api and port 3333 let it access and we are applying it to the engine so that is the reason why from nginx we were able to access port or the api service on port three three three three likewise to make your likes more easier let us see another quick example cat so i'm saying that hey

from the api or nginx like happy block to this one we are trying to block some access so this is like the ip address of the google let me quickly check must have changed

oh yeah it has changed so i'll quickly change this one and

so i will apply this one cdl apply iph

[Music] so now from the api pod i'll try to accept and see if i have access to

no i can't access on port 80 however let me see an access on port 443 there we go i have access the reason is that's what we have defined in the file let it access only on port 443 but not any other port that's so we can restrict the traffic inside the kubernetes cluster based on the ip address range and the port number or even or on the labels i have like few more examples provided to you in the pdf sheet just go and try them and play with how different network pulses can be used to segregate or network the traffic for example in one of the scenarios we gained access to the web application and

from the web application we gained access to the jenkins server and that place really the web application doesn't need to have access to doesn't need an access to like the build server or any other server so that way we can use these network policies to segregate the traffic and stop certain attacks pretty much like fireballs

now let us look at uh something called resource constraints which is again a very important thing which will help us to uh stop some kind of dos tags or the resource consumption attacks so when kubernetes schedules a pod it is important that the containers have enough resources to actually run if you schedule a large application on a node with limited resources it is possible for the road to run out of memory or cpu resources or for things to stop working it is also possible for applications to take up more resources than they should this could be caused by a team uh spinning up more replicas than they actually need to artificially decrease the latency or to a bad conservation chain that

causes a program to go out of the control and use hundred percent of the available cpm so regardless of whether the issue is caused by a bad developer a bad code or some bad luck what's important is that it has to be in the control so that's where we actually use this resource constraint and for that we use something called request and limit so request and limit are the mechanisms the kubernetes uses to control resources such as like the cpu and the memory we will look into that example so so let us look into cv memory limits

so let us start with a file called set limit range i want you to explore and see what this is strange so we are creating a tile called limit range and we are setting some limits like the maximum cpu is like 800 and the minimum is like 200 n so let us see what this means cube cdl apply iphone f set limit range.yaml

the set limit range is created and uh i have created another file called pod with cpu within the range which means i'm creating a pod with the limits like 800 m and 500 m which is as per the specs that we have defined however instead of testing it let us test something which is a power cpu like less than or more than that gives us like a better example so by default these are the limits that i have said but i'm trying to deploy a pod with the cpu limits which is higher than what i have defined so what i'll do is cube cpl apply it it says forbidden error when creating pod with cpu more than limit.angle because

maximum cpu usage per container is only 800 but the limit is 1500 this way if there is some crypto mining attack happening inside the cluster or if some of the containers are utilizing like higher resources than what they actually require this way we can actually stop likewise if you want a container to be running always with some more than what is defined for example a pod with cpu more than limit so that we have seen with less than limit now let's see here that's right and f less than limit because this time we are trying to spin up a container with 100 m but the minimum limit that we have defined is like 200 m that's how we

actually define like the memory limits like higher limit and the lower limit to ensure everything is under our control the utilization and the consumption of the cpu or the memory is always under control so likewise i have a few more files and examples for you like the playing with the same stuff like we have seen like the cpu elements likewise you can also try for the memory limits and then spin up some containers with higher memory limits and lower memory limits now what i will do is i will quickly delete whatever i have created so that it won't stop the rest of the workshop but it might likewise whatever stuff i have created in the network policies

i will try to create them so that it won't impact the rest of the workshop

we have seen like role-based access controls and we have seen like uh resource constraints now we'll be looking into the stuff called uh security context this is uh another important piece like the security context defines like the privileges and the access control settings for a powerdirector container and they are actually defined as a part of the pod and container specification in the pod manifest and represent parameters of the container online what it means is like there are different flags like privileged flag privilege escalation flag or run as a group run as a non-proof run as a user the different flags that you can actually pass to the manifestation file or to the deployment of the daemon set

or the pod when you deploy it and you will see like different flags what it can actually be run so for that i would like to use uh an example called popa so for this one you need to spin up like the oppa stuff in your machine so what i want you all to do is uh just please run this command in your machines launch oppai fn stack workshop iphone dev if you go to the section called uh admission controllers and we have something called open or admission controllers and there is this command launch or python stack workshop iphone dev and it will bring up like the opa stack and we'll be able to do the rest of this

thing so when it actually asks for any like a slack or a web url you don't need to provide anything just press enter and it will spin up some containers i'll just show you

so when you run this command launch or python stack

it will bring up some containers cubes and you'll get voip iphone a it will bring up these four containers called gatekeeper system and it may take like a couple of minutes for you to see these things so just issue this command launch for python stack workshop iphone dev and once it is done [Music] we can go ahead and practice with the stuff so

well basically like what is this uh what is an automation controller basically it is a mechanism by which the request coming to the cuban base api server can be intercepted prior to getting stored in xcd they are like part of the kubernetes api server and using this admission controller like the word implies controller we can control the request coming to the kubernetes cluster admission controller limit like request to create delete modify or connect they do not support like the read request through so open policy oppa agent is like offers it's like an open source service that can evaluate the inputs against the user defined policies so how this works is like we use the language called rego the

users can write some policies using the oppa custom language record it's very simple syntax and small set of functions and operators so once you write something in the ego it will be checked against the rule set so here i want you all to remember two important terminology one is constraint template and the constraint the constraint template consists of both the logic that enforces the constraint and the schema for the constraint which includes like the parameters that can be passed into a constraint so what is a constraint it is an object that says on which resources are these policies applicable i'll show that to you in action

so this is uh the architecture or how how like the oppa was it's like a policy or a decision based some request comes in we have like the policies that we have written oppa validates the income incoming service or the request against the policies everything is fine it will let it to happen otherwise it will detect it straight away for example these are the examples like which subnets the traffic is allowed to or which cluster the workload must be deployed to or from which registry we can actually download the images or the capabilities of the flags that a container can run with so all these can be defined in the oppa like i said this is the command that you

should be our implement running it to have the hope up and running so let us see a demo

all right uh quickly can you all still see my screen or any issues just a quick hands up or some answer any issues or you can still see my screen uh christian can you still see my screen ah yes so you can see this screen still okay perfect so there is a folder called oppa just let's get into like the oppa like different uh files that i have created for you to try test and play so let us start with something called privileged containers cd privileged containers samples

so here let us look at like the constant template it's it may look like greek and latin but it's very straightforward here what what i'm trying to do is like we are trying to identify a flag called privileged under the security context so if this flag is identified we are saying like privileged container is not allowed we want this to be blocked we don't want the users or the devs or anyone to uh spin up like the privileged flag containers with the privileged flag inside of the kubernetes cluster so let us apply this one cube ctl apply iphone f that has been applied

now we need to apply the constraint so how does the constant look like so we have already created a constant template now we are saying hey please apply that template to pods apply sms

it has been applied let us see some quick examples example underscore allowed we are trying to run a container with a flag privileged equal to false which will actually let us deploy because we have written a rule to stop any containers that are being deployed with the flag privileged equal to true so let us apply and see

says uh privileged allowed created now likewise let us apply let us see what is cat example disallowed so we are trying to spin up a container with a flag called privileged equal to true

there we go so it says privileged container is not allowed because privileged flagpole true exist this is the reason why previous content is not allowed nginx security context privileged equally true that's the reason why it could not deploy because we have created a rule saying that don't let anyone to spin up a container or a board or a demon set for example a pod if the flag privileged equal to true enabled this way we can stop most of the stuff this is also called as compliance uh policy as a code or compliance as a code in your organization you might have sort of compliance or the policies saying that hey don't let users to spin up

privileged containers or some flags with host path mounts or maximize that rules so we define those policies or compliance rules in the form of a code so this is one example so i will quickly delete whatever i'm creating so it won't impact the rest of the workshop ucl delete iphone f example underscore dot ammo

also leading constraint

now let us look at a similar example where we have a low privileged escalation flag so we don't want the containers of the pods to be spinning up or to be running with a lot of privilege especially in flag what i will do is template.animal i'm simply saying that where we have like a low privileged installation just say or block it saying that privileged escalation container is not allowed so let us apply this one

now let us apply let's look at the constraints so we have created a constant template and now i'm applying it to the kind called pods ctl apply actions

so that has been applied let's look at a quick example of example this allowed what they are trying to do and we are trying to spin up a container with the flag called hello privilege escalation true and see what happens

this checking against uh the open policies that we have created and ideally should not let us to spin up this container or this pod because we have the security context hello privileges solution flag enable true there we go it says denied the request privileges station container is not allowed nginx so this is how we basically use oppa [Music] in fact the oppa can be applied uh anywhere to any action you can use the oppa even to your cloud infrastructure to the cloud uh to restrict certain things even in the cloud as well let's see we'll give you guys an app

so i have like created many other files for you like testing like letting or pulling the images only for the allowed reports or the capabilities or the host file system https only you you can just try them you have all the commands and everything is just in the pdf just give a go so

no worries so well that concludes the over section so we have seen uh different things under like the defending section we have seen like covered like the role based access controls like role bindings uh roles cluster rule bindings cluster rules and we have seen like the network policies ingress and aggress and then we have seen uh uh constraints such as like memory limits and cpu limits and we have seen uh like the opa stack and in the documentation i have also provided like the port security policy a power security policy is a cluster level resource that basically controls uh security sensitive aspects of the port specification it also defines a set of conditions that a pod must run within

order to be accepted into system it's pretty much like oppa however it is going to be deprecated sometime soon and i'm going to leave it to you guys to test it

so now let's move on to interesting stuff like the monitoring i have like two or three minutes of monitoring the theory part so before we do monitoring i want to run this command in your machines so this is the command that you should be running launch monitoring iphone stack workshop iphone dev it will spin up a few containers and it will bring up whatever we need for this workshop to go ahead and please make a note of this ip address this is like dynamic it will depends on your machine and it also the username and password is dynamic for everyone so please make a note of this username and password which we will be using to log

into the screen how this works is

just from the command launch monitoring iphone stack the workshop in dell just copy and paste this command from the documentation and if you have any issues any errors with this step just read on the command it should work and please make a note of this ip address and the username and the password

and if you look at the parts cube ctl get poi iphone a you should see some new ports coming up it says like container creating it will create around seven or eight containers which we'll be using for monitoring the stuff so while this is happening i would like to go ahead and uh discuss about some theory part shouldn't take more than one or two minutes so you might be wondering like why are we even doing this after all the monitoring stuff because a kubernetes cluster has multiple components and layers and across each of them we will find different failure points that we need to monitor these are we'll be looking at some uh typical use

cases of monitoring today uh especially by monitoring the cluster you can get an across the board view of overall platform health and capacity specific use cases could be like cluster resource usage project and team chargeback or node availability and health likewise we will also look into like missing and the failed ports the number of running pods or the secrets and all kinds of stuff like everything like application availability application health and performance and i needless to say like the monitoring is place a very vital role for any infrastructure and kubernetes is no exemption so let's start so i will try to access my monitoring stack on this url

for 2000 it will ask you for the username and password i have just logged in so in the document i have given to json files that needs to be downloaded please make sure you download them and we'll be using them in the workshop just copy and paste that url in the browser and if you have any issues with that url just copy them in a text pad format it and then use it in the command prompt so what i will do is once after you download those two json files just click on this plus sign and you need to click on import so click on upload json file and you should be downloading those two json files and

load them humanities so i'm just renaming it to prometheus dashboard make sure you always search for this prometheus data source like prometheus this is very important and then click on import and once the dashboard is imported you should see some data being populated on the screen it will give you heaps of information such as like the nodes this is like a demo cluster that's the reason why you see a limited information but in the real time you have like the whole screen like blowing up like christmas tree so here it will starting word that is giving the information about like the nodes or shop network one developer two it is giving like the up time since when

it is up and running and the memory that it has and the cpu that is being utilized the memory that is being utilized the screens and rides and heaps of information now let us import the other one json file

this is another kind of dashboard which will give us like all the network related activities the cluster memory usage if you don't see any information immediately just give it some time because still the containers are coming up because the vm is a small vm it might take some time everything however it will slowly show up all the information the next few minutes so this way this is how we can use the mounting stuff now let us dive into some more interesting stuff let us see how we can actually identify different kinds of attacks that are happening inside the cluster so let us start with uh let's click on this button called explore explore i want to use just choose loki

loki will help us to get all the events or the audit logs that are happening audit events that are happening inside the cluster and prometheus will give us all the resource usage so let us start with loki and i know this is all together a different uh thing like writing these queries is not the scope of this workshop but i have already written some queries for you and i have put them in the document you can just try them but just to test like click on the log browser and then let us choose the pod here and then always choose make sure you always choose the control plane pod because control pane will have all the juicy information

where is the control okay this is the kubernetes api server workshop control plane and for the last one hour show logs there we go or maybe if you want to see for more just click on like last three hours here we go it will give all the audit logs that are happening inside the cluster so for example let us start and say if you want to identify any requests that are coming from the anonymous group or the anonymous users of the company's resources let us see how we can actually identify uh these things

one second bringing up my notes

yep there we go and let us give something some power supply called you

so if you look at i'm passing the flag called anonymous to identify all the anonymous requests that are happening inside the cluster this has brought up some hey these are the anonymous requests that are happening so probably if you click it it will show you like oh the time where the anonymous request has come from which end point it is actually hitting along with it or these are the anonymous requests that are happening inside the cluster out of which if you want to extract only the logs which are like successful so you can pass like 200 it will further uh drill down everything it will give you all the results which are like successful also it says if you

look at this one it says user system anonymous this is the source ip and the user agent response status and the code 200 and the ip and the end point that they are so this is how you can actually for example let us do a quick stuff let us make some uh curl requests as an anonymous user

so curlifun k we are trying to access nodes and we are trying to access pods or we are trying to access secrets the address one more time secrets as i'm creating some junk data for the example

now let me reduce the time to the last five minutes and if you look at these are the logs that have been successful in the last five minutes so if you look at this is the stuff that we did just a few seconds ago we are trying to hit the pods or we are trying to hit the other stuff likewise if you want to further drill down anonymous if you want to extract the data where you know most anonymously is trying to access the secrets so this is how we filter the data so press class finance there we go it says an anonymous user is trying to access the verb secrets so this way you

can actually identify any anonymous and any anomalies behavior that is happening inside the cluster or for example if you want to identify the total number of requests that are actually happening [Music] it gives you like hey these many requests that have happened i don't like six requests uh somebody's trying to access in the last 60 minutes this is how you can actually define or write some additional stuff so there is like heaps of other information that we can actually see now let's look at the prometheus stuff if you choose like the prometheus here and if you click on metrics there are like heaps of metrics that comes by default with prometheus if you go to api server it will give you all

the information such as like the total number of api audits even total for example if you look at these things or audit level total inflate request encryption or init events total can get lots of information likewise for authenticated if you want to see authenticated user requests these are the number of requests that have authenticated users done they're like heaps of information that basically comes with this thing likewise i have given i have written some additional things in the document example cube or container info it will give all the pods on the containers info running inside the cluster it is the name and this is the container where it is running so for example if you want to see the

information of the nodes cube mode created so it gives all the information about the nodes and the other stuff if you want to see like the list of the secrets hcl secret you a list of all the secrets that are inside the cluster so this is how you can actually leverage the monitoring stuff monitoring is all together different animal to what extent you can take it it's up to you there like many other tutorials that are available to do like the monitoring stuff we are actually running out of time we have like one hour we have a few more things to cover but in the documentation there are like heaps of other stuff for you to

try and see for example if you want to see the containers that have been uh deleted or which has been seen in the last 60 seconds so it will give you all the containers that have been deleted likewise we have we want to identify the total number of namespaces there are different operations that you can do using like prometheus and grafana this is just for example likewise you can also use yellowstack like many other things so i leave it to you guys to try different other metrics inside the kubernetes and everything is defined in the documentation with different queries you can try that stuff so now let us look at like the build phase security so build phase security is

where we actually define or look into the security one during the build stage for example we have used oppa that is considered as a build state security one building or deploying a cluster it is being validated against a set of rules likewise when building or deploying the containers we can also perform like document scanning or container scanning i'm sure most of you must have already done this but still let us see a quick example for that i would like to use a tool called uh i'm core drive

this is one of my favorite tools which i use mostly for scanning like many other tools that are available but this is my favorite

so it has been installed right

i think it's populating the database for the very first time that's why it's taking uh some time let's give a few minutes

there you go it is up and running now let us list some docker images that are running inside the machine yep we have a few images now use scribe and let us test some image for example maybe this one or i provide the image id describe the image id

it is like setting up the tv should be done in a few seconds

uh

it is loading the image

so it's scanning the image

there you go the scan has been completed and it says like hey these many vulnerabilities exist inside this docker image so this is one example like you can even imagine like the number of bugs that it has just one image has like almost in hundreds so this is one of the reason why like scanning a docker image is always like plays a very vital role in securing it says like 1004 vulnerabilities in just one image so i want you to try scanning some other images as well like many other open source or scanning tools but this is one of my favorites now we will look at some uh other interesting tools like trivia and play

they have been provided in the documentation we can scan them now we will look at uh the interesting aspect which is like runtime security so what is runtime security it means like betting all activities within the container application environment from analysis of container and host activity to monitoring the protocols and the payloads of all the network connections while creating an environment prevents the security of risks from becoming the realities paramount monitoring containerized applications that run time is still essential runtime protection involves monitoring every cluster for all containerized applications running in each node so for this uh i will be using cystic falco so hope you set it up it's like pretty straightforward so this is the only one command that you

should be running in your machines launch runtime security stack workshop python there you can just copy the command as is from the documentation and uh run it and when it actually asks for the slack webhook url just press enter if you have one you may give it otherwise just press enter the reason why i have set it up slack is like very straightforward if something happens if there is some normally it will immediately alert you on slack and once you have everything up and running like once you shoot this command it will bring up some containers in the background which may take around two to three minutes based on your speed so it is already set up in my machine i

will quickly walk you through this runtime stuff cool so this is how it looks like runtime security so we have so far we have done so much stuff and everything has been recorded and these are the events that have happened for example we try to do some remote code execution on the port 7777 and netcat remote for execution by container launch privileged container and these other stuff for example let's do some hands-on demo and see what happens

and there is a file called event generator which will basically generate some test data for us to inspect so i will just apply only before applying lettuce to it one hands-on i will execute into one of the shell called exiga alpine bin dash cool i'm inside the container ls and just do some stuff like netcat or echo txt so whatever we have done so far should be reflecting in the run security stuff so there we go this is the stuff which i have done terminal shell in a container and it will also give us the information about the name of the container every stuff this has happened just like few seconds ago and we launched a command called netcat

launched suspicious and put two in the container and then we tried to create a file called test.txt echo so this is like the runtime production everything every anytime there is an anomaly it will immediately alert us so the reason why i have used slack is that because every time there is an anomaly it will alert me on the slack channel so runtime security monitoring so many alerts since the start of the workshop but these are the recent ones recently i have created a file for test.txt in the container like alpine so this is one and the next one like we try to exit and gain access to one of the containers so this is like the runtime protection

every time there is an anomaly it immediately alerts us since setting up this lab hasn't been easy like monitoring and everything like the runtime protection i have made your lives easier you just need to like run one command to set up this infrastructure and then just try and practice it so likewise you can try different stuff and i also have for something called event generator which will generate some test events cube cdl apply icon f even generated dot ammo it has been created qcdr will get void event generator

that should be job even general job has been completed it must have generated some colors

so these are the some alerts that has uh created dashboard so that's how you can actually use like the runtime protection runtime production plays a very vital role

uh well one thing i would like to show which i didn't show in the beginning of the workshop is like the crypto mining attack i'll just uh do it you can do it in your machine or there's a possibility that it might crash however let me try in my machine or you can do it along with me so

[Music]

in the attack scenarios we have something called crypto mining attack just follow those steps we will be deploying one container

message

let's get into temp folder and download this file

at the same time we will be monitoring this wasn't like the oppa monitoring stack

what the file has been downloaded let us see what this does i'm running like an image called cryptominer and it is doing something else it is doing some kind of mining and it is also has like the memory limits and cpu limits set appropriately so that it won't alert anything also the memory request has been set so that it won't consume too much so cute ctl apply iphone f ammo cool the pod miner is created so while this is happening what i would do is i will get into my opa stack i'm sorry i get into my monitoring stack which is running on port 2000 which we have seen before

so let us get into kubernetes dashboard just keep observing this stuff we have just launched the miner and let us see how the behavior changes here let us observe like the used section and also while this is happening let's see some interesting stuff like the matrix example let us choose uh loki and let us see what this pod miner is actually doing like log under the pods

um

oh the pod miner is still being created that's why it is not showing up let's give us a few more seconds

now the pod miner is up and running

loki log browser pods or let us search for the miner pod

cool show logs so this is how also you can use the monitoring source to see actually what's happening inside a pod or a container probably i would not want to discuss this stuff this is out of the scope like what's happening as a part of the crypto mining but in the monitoring using the monitoring stuff you can use you can monitor like what's happening with the containers and also if you go back uh to like the dashboards using one of the dashboards which we have seen before initially everything was like green and now seems to be if you look at the cpu utilization has increased to 96 98 and the partition memory has utilization has

been increased i'll quickly delete this one otherwise it might crash my machine cpl apply ctl delete so this is how you can leverage monitoring stuff to identify any cryptomining attacks that are happening inside the cluster you can directly see what's happening with the pods using the logs and also you can use like utilization resources so within just fraction of seconds utilization has picked up has spiked up to like 96 if you leave it for a while the machine might crash or the utilization might be like very high not letting the other resources to have enough resources machine has become already [Music] slow but in the next one or two minutes again everything should be like back to

green and back to normal so this is how like the crypto mining attacks happen and this is how we can actually raise the monitoring stuff to identify and defend against the victimizing attacks we can also write some kind of slack channels and alerting possibly we can also use like the runtime protection to identify these kind of attacks

if you go through these logs you will find somewhere about the cryptomining stuff it's been already one minute since i deployed now if you look at the utilization is coming down slowly back to orange and then it will be like back to normal because i have deleted my crypto mining uh stuff

well that pretty much concludes uh like the runtime security i just want to cover like the last piece which are like some useful tools which we use in the real time and uh i would like to start with canines basically canines provides a terminal or the ui to interact with your kubernetes cluster it's a very handy tool and the aim of this project is to make it easy to navigate observe and manage your applications in the wild and canines continually watches communities for changes and offers like subsequent commands to interact with your observed resources so it has already been set up in your vms you can just follow the commands that i have provided in your machine in

your in the pdf so in order to bring up canines the command is like just k9s that's it and it will bring a beautiful cli this is a cli though it looks like a ui so this is a my pod my cluster information if i choose it will provide me all the information such as like different name spaces powers and everything so you just need to press escape and shift semicolon to run the commands for example if i want to see all the namespaces just namespace it will list all the namespaces that are running inside the cluster for example if you want to see all the information of the pods cards likewise secrets secrets or

deployments you give me all the information about the deployments now for example if you want to get a shell access into one of the ports choose the port that you want and here it defines if you want like the shell just press s so i'm pressing yes and it will give me a shell on inside that port that's so handy this so what else you can do it will if you want to see like the deployment just press y it will give you like the complete manifestation file of those deployments and there are like many other things that you can try and play with this stuff so let me quickly go to the next one

i would like to show you the tool or the tool called cube striker which i have been working and building

last few months you can spin up the cube striker with just one command well i would like to show you the documentation side of cube striker it is cubestriker.io there is the documentation click on box at a glance it will give you like the different editions it has like two types of versions one is command line interface the other one is the application it will have like the current capabilities it can do and how you can actually spin up a container or you can perform different kind of scans everything is listed in the documentation just give a go give it a try and it also exists on github this is the page just type githubcubestriker

and this is the link for the tool just started because i'm going to uh release a good version with some major updates for the black hat europe come which is happening next month just keep an eye on this one and share the word with your community and with everyone and always looking for contributors if you're lucky to contribute like part of this project just please let me know and i'm more than happy to welcome so this is the link github and cubestriker and this is the documentation cube striker dot io and let us see like the cube striker in action at least the command line form so we are inside the container python iphone m

striker will care of the cube striker interesting

[Music] just a small spilling mistake it's nice and cute striker there you go the cube track is up and running so it will accept three types of inputs the url or the ip address or the config file or like the range of ip addresses which can have like the list of master ids and the worker nodes so in this scenario let me scan using a url with an ip so my

my ip address this is the ip address or the url i just provided so once if you provide the url in the ip it will start uh enumerating like for different services it will start for secure board and secure all the things that we have covered during the workshop and once it is identified it says hey these are the end points that have been identified in this cluster we have cube server secure endpoint identified read write service endpoint identified at city client that city server queue proxy and there is like open 42 which is an ssh that has been identified and it will give you two options to perform the scans whether it is like an

authenticated scan or an authenticated scan which is like a black box testing versus like a white box testing so in this scenario let us because i have an anonymous access enabled i'll go and scan like an authenticated scan and it supports like different clusters running on-prem or it could be like a cluster running in azure it is or it could be a cluster running in eks or it could be a cluster running in google it will support all kinds of clusters and in order to provide a token or an authenticated scan these are the commands that you should be using to grab the token so in this scenario because there is an anonymous access enabled i'll just press enter without

providing any token and it says authentication successful because anonymous access is enabled and will give us an information option like perform all checks or perform individual checks i'll go with perform all checks it will start our scanning for iam miss configurations like admin roles relationally admin roles destructive roles whatever we have seen like impersonation all these things are identified using this tool then it will scan for misconfigured containers such as like privileged containers liveness pro readiness group these things and then we'll check for port security policies network policies cube like reader service if you run the scanner in a real time in a on a running cluster it will give like heaps of vulnerabilities or misconfiguration inside the cluster

and once the scan is done it will also give you an option called uh execute commands on the containers you can play or interact with the containers during the printers if you come across with any of uh like anonymous access or any other stuff you can actually execute the commands so these are these are the ports that are available i choose one of the ports it will list all the containers that are running inside the pod and we choose one of the container it will ask me to enter the command i press ls so command execution successful so that way you can do it and once it is done click on exit it says scan completed and the results

generated with the target file name so the target file name this is the target what is the target that we have input 192 168.99 101 and we have 5.99 let us see the output it gives like a very elaborate output this is the beauty of this tool it gives like for example read-only admin roles it says like who has what kind of role it is a cluster admin role and this particular service account called generic garbage collector has created privileges in the name subscript system will give very useful information like what who have access what can do and we'll check for uh premature roles for security policies it will check for containers with various misconfigurations like cpu limits or

events pro memory limits priority class name secrets mounted on socket etc just give a go and try it on different clusters and you will make the best views of it well that pretty much covers the cube striker and there are a few other tools that i have listed inside the documentation like popoy and other stuff

so popoy is pretty straightforward to install it is again uh built by the same team who has built the canines so just run powerpoint probably it's not set up in my machine but yeah you can try it and give it a go

and needless to say keyboard it there's another tool that you can try i have given the commands and everything in the documentation

bench is another tool that we can try and test cool we are on time exactly and uh i think that we have covered pretty much everything all the fortunately all the attack scenarios have worked we have seen different kinds of defending scenarios and we have covered monitoring stuff and if you feel that this is not enough there is like heaps of stuff in the documentation just give a go and practice the stuff and try like the runtime security and all other things well that pretty much concludes the session for today we still have like 20 minutes left if you have any questions i'm like more than happy to take them and you can always reach out to me on

linkedin if you need more information and one thing i would like to share with you all is at the very beginning we have set up this stuff so just go to like the build server 8080

okay [Music] now initially at the very beginning i told you like whether you want to click on proceed and abort so now if you click on proceed it will spin up another kubernetes cluster along with the cluster that we have tried so if you want to fix the bugs and test it you will have like two more clusters so i will leave it to you how to actually interact with the second cluster now we will have two clusters running in the same vm but so far we have been interacting with one cluster and i will leave it to you to figure out how to interact with the second cluster you need to play with the

q config files that we have discussed in the document the solution is inside the document i'll leave it to you to test it and go ahead and it is like an inception everything is built using the containers for example docker pierce i'm using something called kind so on a host everything is spin up using containers and inside the containers we have like the kubernetes containers humanities clusters running and inside the kubernetes clusters again we have like multiple containers and the build tools build servers running and along with web applications some decent amount in time has been spent on setting up this lab to teach people and everything hope you liked it and appreciate it if you need like more

information if you would like to learn more stuff like i said i had to uh make some changes to the content to fit in the like the three and a half to four hours but if you like to learn more i have like more modules just reach out to me and i can help you with some more labs to just practice and test well that pretty much concludes the session if you have any questions i'm like more than happy to take them thank you very much that was uh very good um very professional and i think quite clear and a lot of thoughts for everyone um so there is a question on the q a and

i also saw that adrian rico you had your hand raised but you lowered that do you want to ask a question live i can give you you know i i can allow you to talk if you want to so you raise your hand again i can pick and load uh does it works in a bare metal air gap cluster uh adrian i didn't do much research on the adriat cluster but i can definitely get back to you in some time oh yeah oh yeah absolutely cube striker works on a cube striker just uh needs access to your url or the ip address and it does all the scanning on it like i said you can perform two types of

scanning like authenticated scan and authenticated scan uh and you can also use it in the ci cd pipeline even before you deploy the cluster with the protection if you have it up and running in the dev you can invoke this container in the form of ci cd and it's it's very user friendly csc differently you don't need to pass any commands in between just one single command will scan your cluster you can build a different step in your pipeline and you can scan the whole cluster and just refer to the documentation like cube cubestriker.io it will do it has like heaps of information and just keep an eye on it i have like new version releasing in the

next month which will have like a better ui with more features with continuous runtime monitoring and protection with container scanning everything and it's going to be like an open source

cool thank you so i don't if you like to follow up or something you can now uh talk on the insole if you want to well i don't have anything much to discuss but if you are if anybody has any questions i'm more than happy to take them or else i'm available on slack just ping me there all feel free to reach me on my linkedin i'm available at uh so this is my linkedin name passanchinibili you can just uh send me a request over here i can always anytime reach out to me all right thank you very much uh and i guess you also can drop questions later on slag or something or contact us on

twitter or whatever about the cubist striker or the workshop in general remember also that he uh put all the information on the pdf so you can like complete all the steps again um so yeah you have a ton of information to play with thank you very much for your time i know it's very late now in australia thanks so much for organizing this christian it's very good and thank you very much for giving me the opportunity looking forward to work with you again soon