2018 BSides Toronto: Haydn Johnson

funny thing microphones work better when I turn them on alright folks if we get out of your attention for our next speakers coming up we are going to be talking about kubernetes and the security associated with it please welcome to the stage Hayden Johnson all right thank you very much for having me so this headset is like the matrix that comes out the back here and hangs out so we'll see how I go and can focus good talk by Fernandez or finish always get it wrong on explaining containers and everything to you so I don't have to do that so recorder for my talk I can just scratch I how you guys doing enjoying b-sides yep cool good to hear I'm gonna

try and stay enthusiastic because I've had some coffee and it's all jittering and everything I will see what happens so I know we make a joke that if you're not here for kubernetes you're in the wrong tool can you jump to the other track but the good thing today is there's no other track so if you're not here for my talk too bad sort of thing so let's see it's not gonna work just before and I didn't even know how to get to that screen yeah that'll do my notes aren't there but she'll be right so before we get to the good stuff obviously you want to know who I am so that I have some credibility in your

life yes it's guys sort of knows what he's talking about my name is Hayden Johnson you can contact me on twitter at my original twitter name at hayden Johnson my name I pull that joke at different conferences get different reactions total dead silence is a new one like really raring to go here guys I've got my OSC P and G X P n so I've got pen testing history five years in the industry one year in Australia at Deloitte the big for a couple years here in Calgary and now I'm in Toronto I saw on Twitter there's this discussion about presenting and not referencing people and taking credit for other people work so I just had this big slide to

answer that that they're really good talks like if you want to learn kubernetes in a proper way and not rushed in 25 minutes you should go and see those as some motivation on why you want to see this talk is because with kubernetes it's becoming the future red teamers and attackers are all over it because as we know with technology it's secured at the box and yeah I've been learning it from my org you're all maybe going to it and let's have some fun now I didn't want to say it's growing huge and have you just trust me I did some googling like two seconds and found a couple of medium posts on kubernetes containers things like that on the

popularity and with github as you can see this nice blue line on this faded screen oh my laser light does work awesome it jumps up quite high so from 2014 to 2017 kubernetes is quite popular on github words so here we are - the talk kubernetes security you need to know about it I should probably slow down here I'm rushing so what we're gonna cover today is not an in-depth look at kubernetes and containers cuz it is just 25 minutes we're going to go through what happens at a company before kubernetes or docker things like that what is kubernetes here I am forgetting my own talk the benefits and importance of kubernetes and containers but

primarily kubernetes and the security principles to help but before that I'd like to show you some basic hacks that have happened that may be or may not know of things like that so who he works at a tech company with coding apps and things like that yep Wow actually what are you that's awesome everyone's got jobs yay so as you probably know or you might know these terms a company that creates applications or code has to test it has to make it sure it's bug free things of that nature so these terms such as continuous integration continuous deployment and development or these different methodologies were created so when you type into Google continuous they generally come up

straight away so this is that term you hear with opps DevOps devstack ops and everything just how they develop code the tech apps and keep the infrastructure up so this little one I wanted to show because there's continuous delivery and continuous deployment now continuous delivery is where you write the code and then you think it's running so you hand it off to a QA or the production to manually test like Fernandez were saying but they test it manually the continuous deployment is that it's all done automatically so when you write your code it runs it goes through the tests yes security passed on automation good job yes it passed it the functionality yes processing worked its

gonna it's not gonna break anything then it goes into production which is great but you can imagine how many security holes that are tested by scanners right so companies in the tech industry have many moving parts as I said and they need to be out of produce code applications so then DevOps sec ops agile etc all these things were created now for those visual learners I wanted to create some pictures because it's easier this way to explain so developers and coders have lots to do lots of milestones scrum Kanban board sprints man being at my new place is like open my eyes to hell hell why security isn't in the state it is anyhow they will create automation in

continuous integration to help and that's fantastic you get your products to market sooner you make more money you make your investors happy however as we all know with technology that is not secured very well now the idea of continuous integration the cloud virtual machines is to get everything created faster so what if developers as Fernando said could code without worrying about the underlying infrastructure or how the environment variables are set up now this is where containers come into play so this is when Katinas were born containers are abstracted from the operating system so it can be a web server or anything else in like you have your little little box and you say from my web server I want the libraries the

environments the variables if I can probably skip this because it was done in the previous talk but the idea is that it's a nice little box and it's not a full virtual machine so for the visual learners again you have your host or your ESXi server and virtual machines are another complete operating system so if Windows Windows ABC sitting on a Windows server or something like that now containers here you have your host operating system but the containers all use the same plumbing and infrastructure of the host so the libraries for the web app are contained together so you have web app a and C which are separate but not complete virtual machines now it's

holding to a mentor of mine on Twitter and she's like have you heard this analogy I was like what analogy of houses and apartments as I thought must be pretty good so the idea is that a house has its own plumbing its own bathtub things like that or infrastructure has its own concrete foundation an apartment or a condo which I live in it's like small contained environments where you live or sharing the same infrastructure that makes sense like house containers are abstracted and then kubernetes is abstraction even more so what the containers help with they just help the developers code without having to worry about the baseline environment so it helps speed up development speeds up troubleshooting

things like that and it all gets automated in today's day so now we've had a little bit of a history lesson on DevOps IT ops and things like that the real talk is on kubernetes so what is kubernetes i've only got like 10 20 minutes if that really to go through it or five minutes I guess with other stuff I want to show so how does simplify this at work I was going through because we're deploying kubernetes and there are key security managers security dude what should we be concerned about I was like WTF bro I don't even know kubernetes yeah like so I had to run through build it on my home server to get an idea

do the googling and then this talk came out of it so if you go to the kubernetes website they'll say kubernetes is a portable extensible open-source platform for managing ba-ba-ba-ba-ba it's really professional but you don't understand that so Cuba net is is basically a container platform like a management console the way I think about is like active directory for containers that you can centrally manage everything so the way kubernetes is broken up so that you don't have to think low level is that there's a deployment aspect to kubernetes and docker so that the deployment is setting up an image downloading and installing it that's good now services allow you on your local machine to access it internally so

that's all local so you have to and then ingress is your external traffic so how does your customers or had as a cloud get to your container you add ingress so you would say with ignoring syntax and code deployed this image let me access it internally and make it accessible on port 8080 which is also known as a node port for external traffic so this just helps you to sort of visualize why how kubernetes then abstracts everything but what is actually what does kubernetes actually look like so as I said with the service you've got the internal traffic so you have your laptop or your server it's called a node this is how because you guys don't have their underlying

infrastructure so we're gonna jump back and forwards but each node your laptop can host a pod what's a pot you may ask hidden a pod content can contain one or more containers so I put the little image of docker to remind you that docker is a container and then a pod controls everything that happens in that container again kubernetes is an abstraction on an abstraction so kubernetes helps control helps you control this pod while not worrying about the intricate details and like I said a pod can have one two three or multiple containers so it's it's quite simple not really but the main idea is when I try to say in 25 minutes how do I teach people

kubernetes and get you excited to learn about it I have to shorten and simplify so to simplify it there's a master node which I think of as the brain and the control panel or active directory like your domain controller you have these systems there's configurations you can push out great so you have your brain and it can control different pods that's fantastic so with kubernetes you can either have containers run on like your local system kubernetes can also as a brain stuff up or without swearing it can also be a virtual machine so on my ace XY server actually experimented with creating a cube master that original name but the master controls the two slaves and then

it pushes the app maintains it all that way but it can also be just on your local machine your different pods so you have your front-end you have the bootcamp which is a tutorial black virtual machines but a little glider and abstract it again even more your containers so it's really good that way so we understand kubernetes controls containers and allows you to do all these cool things without having to look at the actual lower level networking and actually coding the ports open example so what are the benefits of kubernetes so when you go to Google it benefits a cuban essage it came with infrastructure lock locking and things like that but what that actually

means again is your developers don't have to worry about the operating system so it's all fancy wording but the great thing with kubernetes is a scalability there health checks and what's the total on less downtime now what that looks like is that the health checks automatic you you know IT ops if you've ever worked in there a system goes down you a crap I've got this alarm going off got a reboot it tell everyone pull backup takes forever or maintenance what happens is say you have three pods a web application a and one of them goes down this little red circle it has a health check with kubernetes that it'll say okay one of you put these down so there's two it's

meant to be three so we're going to bring it back up online which is really good it helps with the reliability of applications were bouncing things like that now upgrades are great as well because at work with the operations team we have maintenance windows for windows or production on Linux servers upgrading patches with puppet and things like that and we have to tell our partners our maintenance window is going to be six hours please we're not going to be making money for that time let's just hold on so here you have your like version a I call it version a if your pod or your server you want upgrade it to version B so instead of having to

take them all down all at once kubernetes says oh you want to upgrade from v1 to v2 let's do that logically or systematically so it'll take one down it'll put it back up upgrade it to version B and then all three running then it will take another one down and pop the second version up so it systematically helps you upgrade that way which i think is really cool and with businesses and agile development these days it's quite necessary and critical so we've gone through wow I got through this fast so we've gone through containers kubernetes and why cuban ease is really important okay so my thing was security we really need to know about this

because the things I'm learning a pre scary and then I was like how do i express to you guys why it's necessary but you already know technology when we are when it's new goes so well like how you just googled the amazon hacks the ec2 incest is Google Cloud Internet of Things how many of those have default passwords and have been hacked and Mirai botnet and everything so I just throw up some screenshots it's quite quite amazing that every time we get new technology and it's going to save money everyone's on board but for securing it No so this is where we come into play but the thing with new technology I didn't want to just say we all know it gets

hacked because you don't know me you don't trust me so I do go to B and I found kernel ownage Chris gates he's done purple teaming and this develop talks around the country in America I think I don't know if he's done it here yet but the idea is that all the technology we use such as puppet excuse me Amazon Cloud and Amazon credentials things like that to help speed up development there's obviously gaps in what we do so as hackers or attackers we will use what's native like Windows PowerShell and figure out way out in through the cloud or excuse me the idea is that it's not tuned properly and it'll get hacked I did make my font bigger that's why

it's hanging off the butt of the the bottom of the slide so that does get much worse I'll try to fix it so before containers we sort of went to virtual machines and like a virtual machines is totally secure because it's a whole operating system on another operating system I gave away my secret so you've got your virtual machine nobody can get to the hypervisor but yet with the technology and the attackers these days or over the years they there are hypervisor escapes there's all these exploits to get to them so the reason why security needs to know about kubernetes because of the excitement and the growth is because kubja days is no no different

there's default configurations default passwords there's functionality within Cuba nas is quite powerful for the low-level users so the first topic I wanted to cover was default configuration so I'm just going to take a minute if you guys want to read this so the key line if you are wondering is that they treated as anonymous requests and give a username of system Anonymous so if this HTTP endpoint is not blocked or controlled by the firewall or a firewall rule to stop traffic or it's not flagged as do not allow you can execute commands as system pretty cool huh so on this medium blog post they're like my kubernetes machine was running crypto mining software and but Mike admin

console has a password there's been no password hacks there's no like what's going on here and then because of the previous post they were able to find out that there was a curl command that actually executed because the HTTP endpoint was not flagged as do not allow now with new technology it's great to get it out there as quick as possible but there's no one securing it and even myself if I deployed kubernetes I wouldn't be like oh I don't need to deny anonymous access wait there should be one on one it's like a ftp server with an anonymous access like why so this other one again it's not the HTTP endpoint but there was an administrative

console open on AWS so from that they just logged into the the web admin console the Cuba News version and found a pod and then started crypto morning like nice and simple it's just it's a rerun of the databases that are up on the cloud it's a rerun of everything that goes up in the cloud and just is exposed everywhere so Tesla Ellen must shoots cars into space creates flamethrowers but doesn't hire people to add passwords to their consoles so this one was just an insecure Cuban Eddie's console so I didn't have a login name or password Oh probably default but look someone on the cloud went hey cool I found this one but the thing was one of their kubernetes

pods had access to the Amazon s3 credentials and telemetry that way so not only could they access kubernetes pod they could then get to Amazon so again with all this technology its exploiting and creating interconnections that we just have no idea how to control this is not from Tesla but it's it's an example of Kuban at ease with some s3 credentials in it like it's that that's how easy it was or so apt some simple more advanced kubernetes attacks is that there's this great YouTube video on hacking and hiding kubernetes but it's all about using the environment so you can install custom tools if you execute a command you can reach out to the API

and say hi Cuba Nettie's API or master please give me a list of each pod it's name and everything like that it's just like here you go and you don't need much access so it's just like Active Directory like it's so hard to configure and the new versions are getting much better yeah you can read metrics there's might better install other services inside the cluster so cuban eddies really needs your help or at least needs security security help so some security principles again for everything is basically be aware that security matters the Internet is a dangerous place because every time you put something up it's gonna be attacked with default credentials or a script that brute forces it so don't use

password password again this Center in for Internet security benchmarks is really good but you got to make a contextual to business and deployment and everything so with containers there's image scanning so like Active Directory or your work laptops you just have a gold image you know it's secure it's hardened you use the same image again you don't just go up download or get clean someone's image from like random hacker not hacker and I deployed in your production so it's all about sanitization and being safe about it and then runtime security so each container is should be like specific to one application or one one reason for it existing and so you should be running it

monitoring logging what it does when it's up so if it's a web app server is it meant to LS is it meant to say Who am I isn't meant to create its own get request to some random mining company or something no so there's this tool called system Falco which has great tutorials to bring you awareness on what you should be logging and monitoring that's really good so I suggest doing that and just walking it down hopefully you've enjoyed my 20 23 minute 25 minutes run of Cuban Eddie's thank you very much having me

and any questions yep sorry they were not google kubernetes engine instances know they were self yeah however the tehsil one and that are actually hosted in amazon but not gke because the Google communities engine obstructs it again and adds a lot safety and control over it so you really have to be a idea 1ot whatever this thing is idiot to really stuff it up that way that's gonna say the the medium one this one was actually a handy employee but it wasn't at handy it was like local oh yeah anything else oh thank you thank you alright before anybody goes away but once I be sure to sign that Aidan sure design that alright before anybody goes

away we have an opportunity to raise money for cam H so the idea here is something that we do every year and thankfully we have been very lucky to have the folks at sector share that share this with us we have two passes to give away for sector one for this calendar year one for next calendar year and the whole idea is it's a bidding process and it's an open forum type of deal last year we went up to what almost a thousand dollars so that is a really good cause especially when I mean I've been we've all been touched by folks that have been affected by mental health issues so I think this is actually a

really good cause so how should we start the bill that bidding next 250 all right so for the pass for this calendar year for sector which it starts on Monday we're gonna start the bidding at 250 do I hear 250 Wow last year we had to fight them off with a stick here anybody Bueller anybody want to go to say okay let's put it this way who's going to sector okay so you guys don't need tickets who needs a ticket all right we got one guy there Oh terminator nice alright so Kevin what do you want to bid alright anyone have a bid for 250 do I hit the head ah we got one for 250 excellent alright so we've

got the ball rolling that's right 300 anybody for 300 we've got a 300 over here 350 350 right there this is more like it 350 400 400 anyone Bueller come on this is for a good cause yeah there we go 400 400 do I hear 450 450 in the back row alright 500 do I hear 500 we're at 450 right now do I have a 500 for this year sector conference all for a good cause 500 there we go alright 500 do I hear 550 we're at 500 good cause great conference I might be biased I've been working on for 11 years but what's the list price actually you know what that is an incredibly excellent question I

don't know is that 14 oh my god okay we're up to $1,400 now as list price thank you I you know heaven forbid I would actually mention that part alright so at last cause we were at was it 500 yeah we're at 500 right now for a $1400 ticket so do I hear 550 but ah there we go alright we got a 550 550 to a or 600 600 Wow awesome okay now we're getting someone that Amir 600 do I hear 650 650 Bueller is that a hand up in the back row there yes you with the lanyard no okay he looks incredibly alarmed yes alright so 650 anyone for 650 oh we got it 650 alright 650 do I hear 700

yes I'm doing $50 increments because I don't math all right so this is where my brain locks up and my jetlag kicks in where are we at right now say fifty two hundred all right six fit all right just just for that we added two hundred no six fifty do I alright so we are at six fifty and we have any takers at six fifty or we're gonna go at six hundred going once at 600 going twice at 600 sold to the gentleman right there well done great cause so if you could make sure afterwards you give your information this gentleman here in the front row you'll be able to sort you out all right now we get to do it all again

all right this is for next year sector not this year this is for next year's iteration all right so let's try this remember $1400 which I should have mentioned in the first place thanks again $1400 ticket probably gonna be more next year because Brian's funny that way so we're gonna start things off at 250 do I have a bid of 250 we got 250 all right 300 300 do I have math apparently 350 350 oh we got a whole bunch of 350 400 450 500 damn 550 600 600 oh we got a 600 650 there we go 700 750 the plot thickens 800 all right 850 all for a good cause 850 do I hear an 850 you want to go to

lunch I want to hear an 850 all right we got an 850 900 Oh 900 awesome 950 950 going once going twice sold to the gentleman in the back row thank you very much give her a round of applause

don't worry I'll let you go in just a sec do we have a representative from Proofpoint in the house awesome proof point is our sponsor for lunch today and give him a chance to say echo Lord thank you my name is Stefan shahram nod I'm a principal sales engineer but a proof point we do have a table outside there a lecture hall I do encourage you to come say hi I'm very proud that our company would associate self with this wonderful event and I look forward to coming back make sure thank you thank you very much all right all right lunch is downstairs one key point at 120 is our next speaker you do not want to

miss that one 120 see after lunch