Contain your Enthusiasm - A 101 guide to containers

[Music] good morning besides uh before i get started with the talk i just wanted to uh send out a a quick thank you to everybody who's who's helped put this together um 2020 has been a bit of a tough year uh for everybody and getting all this together was uh a major effort so uh janice and josh uh it's been amazing being part of the team um and looking forward to doing it in the future and for the rest of the team um you guys are are absolutely the best and uh you know can't be doing this without you okay so on to the talk um so we're gonna talk a little bit about containers um so this is more of a

kind of a 101 on containers we're not going to go too too deep um but hopefully by the end of this you'll understand what containers are and how they all work uh this is the obligatory me slide you can read i'm not going to go through all this but if you want to contact me there's my info so what is containerization um so back uh sort of dawn of computers in the beginning etc you know your basic computer was uh some infrastructure put an operating system on it and then you ran apps and apps can see each other um you know you can get onto the machine you can run multiple apps next to each other and you

know right at the beginning uh you know part of the fun of of breaking into things and and sort of going down the security path was being able to jump from app to app and and interfere with other apps reading memory changing things on disk etc one of the problems with normal computing like that is that we have these super powerful machines that can do all this stuff and we put an app on it and we realize oh wait we want to secure this and we don't want other things to use this so now we have this super super expensive super powerful machine running this small little app that doesn't really utilize much of the

power of the machine so we started looking into things uh and on how to use how to better use the resources we had and they came up with virtualization so the the center of this slide here is sort of a look at what virtualization is so instead of your host operating system you have instead something called a hypervisor and what a hypervisor does is uh it creates a virtual environment um kind of emulates the cpu um you know memory is still there uh devices are still there but what the what you see inside of a virtual machine isn't necessarily what the actual hardware is for that machine um so inside of a virtual machine you

can install the full guest os um you know whether it's windows or linux or whatever and then you get your your application on top of that so now we have the ability to build these virtual machines that we can run multiple on the same hardware and they're all sort of isolated away from each other and that's great and wonderful um but now we run into an issue where um you know these these guest os is it's it's the whole operating system every time so it's all you know several gigs of linux or windows or you know whatever you're running um and these images that you're creating for the virtual machines are pretty large um and and they're not the

most portable things in the world now companies like vmware have done you know a lot of work to to make these uh be able to transport these between machines but a lot of that system depends on super high-speed connections between the machines so that they can transmit all that data back and forth so over the years um and and this technology isn't exactly new but over the years we've had a lot of of uh different advances in sections of uh operating systems with being able to isolate processes or memory etc and in the past couple of years um they've sort of put all this together and come up with this uh this basic like premises of of

containerization uh and what a container is well we'll talk about a container a minute but what containerization is is similar to what a virtual machine is um but also trying to keep the the speed and the uh the the the small footprint of a you know a base system you know running things on bare metal so in a container you still have that host operating system like you do with a regular machine you have a thin layer on top of it called the container engine and then you have these these objects that are the containers themselves and the the container itself has everything it needs in it to run um but instead of putting a

full operating system inside of there uh you're only putting in exactly what you need so let's look at what a container is by itself um so and we'll go through each of these and identify you know sort of what all these different terms mean um but at its base a container is a layered binary image it's just a blob object of data containers are isolated they're kind of sandboxed away from other containers and from the host there's exceptions to this sometimes you want the container to be able to talk to the host sometimes you want containers to be able to talk to each other and there's different techniques for that containers are portable and by portable

we mean uh being able to move uh to transmit them to multiple machines um you know across the same os you know but being able to move them at a moment's notice um so one of the one of the big things about containerization and running things like kubernetes and docker and all these different uh orchestrators is you can run it across a whole bunch of different machines and if one goes down it just pops up somewhere else uh you can do this with virtual machines as well um but it's there's a lot more heavy lifting there and a lot more resources needed to do that uh containers across platform sort of um and by cross-platform i mean

you can take a lin a container that is uh built for linux and you can run it on a windows box or or a mac there's uh definite exceptions here there's bits and problems etc uh and containers are secure um and you know it wouldn't be a security conference if we didn't say that that's not entirely true so layers um when i say containers are built in layers a layer is basically a uh a thin slice of uh an image um so we start with a base image at the bottom which is you know kind of a complete running container and then the layers that lay that are on top of it are differences between

the layer before it and the current layer so your base image may be something like uh alpine which is a common distribution that's used for for containers and layer one might be installing ssh and layer two might be putting a config file in for ssh and layers can include things like metadata labels and descriptions etc layers are chained together so each layer depends on the layer before it and the whole thing is put together using what's called a union file system so all the layers um uh the specifics of this are a bit technical but basically it kind of smashes all the layers together uh flattens them and it looks to the end user like you're on a single system with your

normal mount points and binaries and libraries and everything else in actuality you're sort of jumping between layers on disk uh layers are cached meaning that when you update a container you may have the first five or six layers of that container already uh and the only thing that's changed is is the last couple so you're only going to download those last couple layers the first layers just stay in place and those first layers may be used for you know a dozen different containers across your system so one of the one of the optimizations that you can make when you're running a container system is you know all the containers that you build and you use

try to use the same base image try to use you know make sure that the first couple of layers which are usually the thickest layers you know they're usually the biggest because they have the the most the you know the the operating system or the the bits of the operating system you need are there um but if you have all of your if you layer it the right way you have all of your common components at the bottom you don't ever really have to update that long term you know you update it for security obviously but you know every time you update your application you don't necessarily have to update those those base layers um and of course you won't download the

same layer twice so if we look at a traditional vm and they've done deduping and some other stuff with this but traditionally when you have a vm every single vm it's the whole big huge blop you know so if you have a 50 gig vm you've got 10 of them you know even if there's slightly only very slight differences between them you know you're using up 500 gigs of data just for that if you have you know and containers are much smaller but if you have you know 50 different containers and they're all using the same base layers you may only have you know a total of you know 10 or 15 gig of data um and

doing the same thing that all those vms are um layers are immutable and this is sort of a weird um uh a weird thing to say because once you get once you start running a container you'll see that you can change things when you're inside the container you can you can erase things and you can create new things what's happening there is the only layer that you're changing is that very last layer that writeable container so if you delete that container just shut down the instance of it and remove it from running and restart it again you'll have the original container with all the stuff in there that you deleted it also means that any data that you

saved inside of the container is now gone and we'll talk about that as well um and that's kind of what the layers look like so we'll take a quick look at an example um on the left hand side we have what's uh commonly called a dockerfile um docker was one of the first uh container systems or first commercial container systems um and so a lot of the terminology has sort of included the word docker in it um i don't know if they've come up with a better naming at this point but this is this is what a container file looks like um so you've got uh the very first line tells you what my base image is

so in this example we're saying we're taking the alpine image which is just our sort of our os layer uh we're taking the latest version of that so that's the the tag is on the end of that and then we're going through and we're modifying that base image to be whatever we want to we want to run so the first four or so lines are arguments um there's a maintainer tag so this is just metadata that's inside the system uh and then we get to a run line so that run line and to be clear the the arg line the maintainer and label lines also create layers but they're they're zero byte layers so that's just a

metadata change so there's really nothing there to to show in the in the image the first real change change layer that you see is that that run line um apk is the package management tool that you use in alpine and all we're doing here is adding two packages open ssh and shadow so we sort of chained together a couple things this is one of the tricks that you can use with containers to to sort of keep the number of layers down is to chain things together um in in you know it's just a bash command um so we're installing the operating system installing open sh and shadow uh we're creating a directory and we're removing

the default keys uh then we copy our config in uh we copy what's called an entry point so entry point uh is commonly uh it's the common name that's used for the script that runs when the container starts so we copy in the entry point uh we're exposing port 22. again that's a that's kind of a metadata thing so i'm not really showing it an entry point which is the tells it what uh script to run or what command to run when the container starts uh is also in there um again that's another metadata command so now we have um you know what looks like a five layer container in actuality it's uh what about ten

layers um all of that together is is a uh is the container and what you can do is look at the history of a container which is what we see here and you kind of read this in reverse so the very um the very bottom of this is that first layer that we have so it doesn't say you won't see it in there that it says alpine latest it's actually showing you the uh the sha-256 hash of the layer and it goes through and shows you all of the different layers that it's created you can see the zero byte layers and then you can see the difference in size for the layers that we're adding

in and if you look at this i mean this is about a about a 20 meg image total for an ssh server so tags images have tags um tags are are used to identify you know kind of figure out the difference between one container and another uh it's a lot easier than written in remembering those big long 256 uh uh shot to 56 hashes uh tags are unique per image so any any given image say alpine or centos or ubuntu you can only have one tag with the that unique name in it right but you can move tags so this is where the the sort of the latest standard comes in so as an example we'll take a look at

this is alpine um i just grabbed uh the latest existing image of alpine um and that image has four tags in it um it has the the latest tag which is the one that we're going to end up moving uh it has three because it's alpine version three uh 3.12 which is uh the first patch level and 3.12.1 which is the the second patch level so what they're doing is using multiple tags to identify you know the identify the image so that you can get whatever you're after so if you are after a very very specific version of alpine you can say alpine and give it the tag 3.12.1 if all you're interested in is making

sure that you have alpine three and you don't you don't want four when it comes out but you want to stay on alpine three you could do alpine colon three and you'll you'll get whatever the latest alpine three image is and of course if you want the absolute latest version you choose the latest tag and when that when they put a new version out they they go in and they say the latest version has this tag it moves that tag and then you end up with uh the containers that you see on the bottom so the original latest container which is that alpine three container still has the tags of three three twelve and three twelve one

but the latest tag is moved over to the other container so now when you do a docker pull or a pod man pull or you know any of the other systems you're going to pull the container uh with the latest tag on it which in this case is that alpine4 container which doesn't actually exist yet but as an example but where where are these stored like where we're talking about pulling containers where where where are they so containers are stored in what's called the container registry um when you build a container on your machine with build a again build a docker docker builds you know all the different building commands uh the containers are built locally uh

it stores them on disk in uh there's a file system um var lib docker is the common one and in there is all of the all the different layers for each container there's also as part of docker and i'm not sure how some of the other systems do it but there's a registry that it sets up a local registry it's only available locally you can't get to it remotely by default so it kind of works the same way as a remote registry does you'll see the a listing of the the containers the container names the tags they have the sha sums the sizes all that stuff but that doesn't do anybody else good we want to be able to share containers with

other people or have them in some central place for all of our servers to be able to get to so you can push containers to a remote registry remote registries can be authenticated um you know there's there's definitely some security in there uh it could be better internal registries are usually quite protected because uh some containers that are built have secrets in them or uh have uh specific configurations for the network you're on not the best practice in the world but that's how a lot of people are building them uh so the registry itself stores the layers so you've got the actual uh the actual files that represent each of the layers which is just a basically a binary

directory and some binary files it also has a manifest which is a json file and that json file lists all the layers of your container the the hashes for those different layers so that it can authenticate or can verify that it has the correct layers uh the tags for that container as well as a bunch of other info okay so now we have a container and now we want to talk about um the next step here which is sort of the isolation of the container so isolation uh consists of two basic features of uh and we're gonna talk on the linux side of things windows does something similar and mac is its own beast um so isolation uses namespaces

and it uses c groups namespaces um which we'll talk about in a minute are are uh kind of the the first layer um and it's it's very similar to what you have in development um you're you're giving a specific namespace of something to the object that's running uh and we can do this across network interfaces mounts uh inter-process communications so talking between processes uh uts which is an old uh if it stands for but it's it's a sort of an old original unix thing uh where that's the the host name and the now defunct uh nist domain uh as well as the user and the process ids so what is a namespace very simply namespace is mapping from one object to

another um so what i've shown here is uh a a the user pid namespace um so what we're doing is if you look on the left this is inside of the container so if we look at all the processes that are running inside the container there's a whole whopping five of them uh and if you familiar with linux systems you'll you'll notice that there's a whole bunch of things missing like where's init or systemd or or you know any of the the kernel threads or or any of the normal stuff that you see in a linux system and the answer is that it doesn't need those it's not part of the container because the container isn't running the

host system so it masks it away so the only thing that you're seeing are those those five processes and you'll notice the the pids are pretty low um you know as a linux system you need pid one um that's your your main process and in a container if you kill pig one the container dies um so you start out with pid one um you've got all the other other uh process ids in there and inside of the container it looks like this little small part of the world if you get onto the linux the linux host system itself and look you'll see all sorts of stuff going on now i've cut this down quite a bit but

you'll see the top couple of pids are normal things you see inside of a linux system you get systemd you've got some some uh some kernel threads some irq dam uh stuff going on and if we look down further uh we start seeing this is specifically docker but you start seeing some of the docker uh binaries that are running um that are the run c is the actual um container runtime that runs the container so that acts as like your bootstrap and your your your your uh root image and the uh further down from that you'll see those apache threads and if you notice the pids on those are much higher so what's happening in the background is

it is mapping those pids back and forth so inside of the container you see pid 1 and outside of the container you see pid 4054 which is the that that root uh apache process that's running inside the container now there's some bad stuff going on here as well uh we're running things as root which is not what we want to do unfortunately a lot of default containers uh stuff that's out there that's been built all uses uh root for for everything you can actually go into the container and tell it to use different users and map them to different things if you look uh user 33 which is the one underneath uh root uh on the host system is actually mapped to

uh dub dub dub data inside of the container um so so it does it does user mappings as well so this is what the namespaces do so you've got the view inside and the view outside this happens for network interfaces this happens for various different resources that you have between the two between the the host and the container the second part of isolation is c groups um secrets is a is is easy to understand a little bit harder to sort of put into practice um it's kind of like black magic um but it's pretty neat uh so c groups allows you to limit the cpu and the memory for any given group of processes by default inside of a container system

you pretty much have access to everything but one of the nice things is like you can go into a container and you can say you know running a uh say an nginx container uh we we don't need that much memory and we certainly don't need that much cpu so we're gonna give it you know uh a tenth of the cpu uh we'll reserve away for it uh and we'll give it a max of you know a gig of memory um and this allows us to uh run multiple containers on the system and not have any single container sort of overpower the system and and take away resources from everything else um it's it's it's all built in

all behind the scenes in all the different container runtimes um so when you use something like docker you don't have to know all the blackmagic commands to set up namespaces and c groups it's all a single command it does it for you containers are portable uh we already talked about building the container uh putting in a repository um and then you know putting it into a remote repository um so the idea is that you can build a container once and you can deploy it multiple times so we've got uh this container that we built that's our say our nginx proxy uh and we've got you know five boxes or five vms or you know whatever we're using

that are our front-end docker systems or our front-end container systems that we want to run uh a proxy on uh and the nice part about proxies for the most part the nice part about proxies is that they're stateless so we can run the same container across all of them there's no real store other than logs there's no real storage we have to worry about so we can take that container build it once run it on all of them when we know that every single system is running the exact same container so we've got a pristine environment all the you know we can still modify the configurations but all across all those machines everything is running exactly the same

so if we've tested that container in whatever environment we're in we know wherever we run that container it should run the exact same way uh you can do things like replicas so docker with docker swarm kubernetes nomad uh and some of the others have the ability to like their they're clustered container systems so you can go and deploy a container once uh and say hey run this nginx container uh we want to make sure that there's five replicas running at all times and it will spin up five copies of that container so all of them are running across there across the system if there is a failure of some sort uh whether something inside the container fails and

the container crashes uh or the entire system fails and it depends on the orchestrator but the the system will automatically restart a new copy of that container up to the number of replicas that you have you have uh given it um some systems are a little bit more advanced and will spin up replicas based on cpu and memory usage but in general it's you know here's i want to run three of these please keep three of these running and the system will always make sure that there's three of them are up um the other nice part uh about containers um especially considering portability is that the containers are great for difficult to install and configure applications

um so let's take something like uh a lot of the tools that are in say kali um you know getting those up and running in the exact manner that you want sometimes takes a ton of effort if you do that inside of a container you can spin that container up on whatever system you're on you have that container say in a remote registry that you have access to you can pull it down and no matter where you are what machine you're on as long as you have a container uh uh run time on that system you can pull that that kali container down um or whatever tool you've created and run that tool without having to

worry about conflicting libraries uh worrying about um you know uh configuring a different compilers or or you know libraries that are missing or sort of you know dirtying up the system that you're on and each time that you run it because they're immutable you go right back to what your your initial start state was um which is wonderful um so if you go through and do an entire pen test um you know when you're done you can wipe the slate clean and you know you can have a very simple little script that does that start starts the whole thing over again and off you go there are some gotchas with portability we've talked about we talked about state

a little bit before basically what this means is that any data that you are creating inside of the container is only created on that thin writable container layer inside of the container now you can mount in external directories so if you have like a local directory you can keep that data locally on that machine but again it's locally on that machine so if you're dealing with a clustered environment and you store something locally in a local share and the container moves somewhere else and that local share isn't there anymore that data is not available to the container there's shared uh file systems that you can use there's a bunch of different ways around that but

you know that's one of the one of the things you have to deal with when you're when you're building containers um likewise uh any state that you have in memory uh is only there for that container um so that's not shared across again you get into things like using memcache d or or you know other other uh shared memory systems um but you have to build your application uh to to use that um generally speaking if you're in a big container environment lots of clusters etc your developers are are looking to develop those containers to be as stateless as possible i'm still using databases on the back end and you know things like memcache etc

but you want your state away from the container where you can spin that container up on any one of the machines and it can get to the various different things it needs to keep to keep running um occasionally there's also environmental differences um hardware access and resources obviously if you've built a container that uses a lot of resources and you put on a low resource machine you're going to have issues that sort of exists with vms and normal normal applications anyway uh hardware access uh is a little bit it's it's not quite as bad um depends on the what you've built in the container so if you've built a container that is uh for say you know well we could use

kali again if you built a container that needs access to say some specific type of device for rf or uh you know some type of network device that you're expecting certain access on if that's not on that machine obviously that's not going to work again the same thing exists in vms and on bare metal uh we'll take a quick side journey into deployment here for a moment containers are portable we said that one of the nice parts about deploying containers when you're in a clustered environment is you can kind of deploy live so if you've got a container out there that's got five replicas and you say fire up that new version there's a bunch of different options

that you can set for this but by default most of the systems do a round robin deployment meaning that they go to the first container they'll drop that container down they'll pull the new image and that doesn't happen necessarily in that order sometimes uh depending on the orchestrator you're using it will pull the image first and then tear down the container just to speed things up some containers aren't smart or some orchestrators aren't smart enough to do that um but it'll it'll pull that old container down it'll fire up the new container with the new image and as long as it comes back healthy it'll move on to the next one so you get kind of a round robin

deployment uh if you if it fails um there's fallbacks that you can set up uh so that you know hey this container failed three times jump back to the previous one and if you do have a successful deployment you cros deploy across the entire you know cluster and for whatever reason there's a problem with the new container you've identified it and it's not something you can fix quickly you can drop back to the previous version which by the way is still there and should be a very rapid spin spin up again um assuming again any state that's recorded in databases etc isn't going to screw that up containers are cross-platform sort of here's where we start getting

into a little bit of the uh the more uh uh gotcha areas uh in general linux containers are pretty portable uh pretty pretty cross platform across mac linux windows they work across as long as you have uh the proper um processor family in the machine uh meaning if you are on a um on an x86 system or an amd64 system and the container is built for that as long as you're on that family or processor you're good you can't take a uh an arm built container and run it on an intel box and vice versa um so linux containers pretty much across the board no no big deal windows containers are a completely different beast windows containers can

only run on windows i have not seen any anyone getting them anyone with the ability to get them to run anywhere else this is a lot to do with the kernel that runs in windows and it's kind of only in windows the other gotcha with windows is that if the different base image that you're using so that that base layer image that you're using only runs natively on the same version of what of the host os so if you built a a windows 2016 windows server 2016 container uh the only place that that's going to run natively is on windows server 2016 and that's what's called uh process isolation so very similar to linux where it's using i don't know what

exactly what they're using behind the scenes and windows but it's very similar to c groups and name spaces uh in linux if however you have a 2016 image that you've built and you try to run it on a windows 2019 host you have to put the system into what's called a hyper-v isolation which is exactly what it sounds like this is a virtual a hypervisor it's virtualization it it runs a shim of some sort of windows 2016 so that it can run that 2016 container um i haven't done a lot of testing with this but you do need to have a uh the hardware that you're running on has to have hyper-v extensions in it uh and if

you're running in a uh virtual environment say azure uh there's a uh when you spin up a vm there there's a mechanism there's a option that you can choose to have uh hyper-v stacking i think it's called uh where the vm has those hyper-v extensions on top of the hyper-v extensions or whatever virtualization extensions they have in the hardware so you can run old windows containers 2016 can run on 2019 but you cannot run new containers on old versions so a windows 2019 container cannot and will not ever be able to run on a windows 2016 host um so there's the gotcha it's it is for every uh every major version of windows that they're putting out i

think there's if i remember correctly from the matrix there's like six or seven different versions they have right now microsoft has a big table that they show you which one works on which in which isolation uh then there's mac os containers uh it is possible uh the the instance that i've seen of this is nothing more than a kvm uh virtual uh machine inside of a container uh running on a host uh yeah you can do it um it's not really supported anywhere i don't know what the limitations are and i really don't know anybody who's doing anything like this in production um uh i do actually have a a kvm um container that i've built for one of the

apps that i have that i haven't yet yet to be able to uh containerize and it works um it's got a whole list of of issues and gotchas that you have to deal with but you can run a virtual machine inside of a container which i found pretty interesting all right security for containers uh i mean this is a security conference right so security inside of containers uh on a on a linux system um and i don't i don't know a whole lot about how it's done on a windows system um but on a linux system is exactly what you think it is uh it's all the basic features of any linux system that you

have essay linux plays a a big part in containers um if you have it turned on if you have it turned off then please turn it back on you need that uh so sc linux is sort of uh uh as with most things is layered on top um it's not explicitly required and it does exactly what you think it does um so your container images have uh certain labels applied to them and in theory if you're trying to access something that's not within the label you get denied um other things that you can do for security aren't uh aren't always thought of as uh security when you start um but a lot of the low-level security stuff that um is

recommended are things like on your dmz servers uh let's not install things like gcc or nmap or netcat or well anything that we can get away with minimize the hell out of that operating system so that the bare minimum of tools are there now as assisted man as and as most administrators know that's great in theory uh but when the server's broke and i need to be on it fixing things uh those tools come in really handy please give them back containers makes this a little bit easier where inside of the container you can minimize the hell out of this container you can bring it all the way down to not even having a shell in it if you wanted

to um there are containers that you can build uh which is simply a go binary um there's there's no os layer re i mean there is an os layer but when you're building it it's the only thing you're putting there is the go library so if you were to try to get a shell inside of it like it's just not there um so you can strip these things down to bare minimums and the nice part um what i find really really useful is inside of a container we go back to the immutability uh hey i'm having a problem with this container uh you know i get on the console and get into the container or um

yeah that's it the only way to get in there please don't put ssh inside your containers uh if you get into the container and you go well you know i can't really see what's going on um i really need hmm i need uh i need netcat and i need nmap and a tcp dump would be really great you know okay great uh you know run the commands to install those okay they're installed use them all done with that okay we figured out what the problem is great kill the container restart it poof all those tools are gone and by the way if you're running you know 10 15 instances of the same container only that one container that you're

working inside of gets those tools for the period of time until you restart that container so now you know your footprint your your threat landscape or your you know the the attack vectors that they have are greatly minimized because all the stuff that you would normally go after isn't there um another thing that really helps on the security side is container network isolation um so we always talk about segmenting networks um and we've gone through you know back in the day when networks were all physical you would physically segment the networks then we got to virtual machines and virtual environments and now we're virtually segmenting networks with containers um all of the to my knowledge all of the

container uh major orchestrators and the container runtimes have it's not quite a virtual network except it is a virtual network where on the system itself you can go through and create different sub networks so you can you can subnet the entire system and you can have the the i mean don't do this because you want different machines for this but you could have a dmz network and you could have a database network and you could have a you know a back-end network and they're all isolated away from each other on the network on the network itself and you can depending on which orchestrator you have you can actually put in uh different tools that will put in

firewalls or you know different policy sets that you can use to control what what traffic is going where um so you can actually net isolate the networks apart uh so even if somebody were to break into the container and they go oh great i'm in the container let me see what's here there's nothing for them to look at because they you know they're in a network that doesn't have anything else in it um and we talked about immutability a dozen times now um one uh i talked to uh a guy who had been doing containers for a while ran a big edge mesh network like a kind of like a cdn type setup and one of the one of the things that he

created or that they did was what they called the clean slate protocol and they had a standing uh effectively a cron job that at you know midnight every night it would completely destroy the entire container environment you know in in steps it would pull down all of the containers delete everything that was there and rebuild the entire system you know redeploy everything um you know and it took it probably took a couple minutes per per region or per environment they were doing but the theory was that even if somebody gained a foothold in the network in their container network during the day the act of cleaning the slate kicked them out and they would have to go through all the

steps that they did to get back into the system again um it made sure that everything was clean there's benefits for security there's benefits for making sure that your apps are not running for long periods of time i know we all used to jump up and down when our linux boxes had you know massive run times but you know we want to make sure that these these containers are are up and running and in pristine condition so if you can nuke them and restart them and and have a more stable system that way go for it so how do we make all of this work together um you know it's easy on your local machine to spin up a handful of

containers to do whatever you know i want to run a web server i'm a developer and i want to run this this container of this thing that i wrote just to try it locally but when you're talking at scale when you're talking about you know a dozen servers or 100 servers or a thousand servers we're talking about cattle we're not talking about pets anymore and you know we don't know the names of our cattle they're just numbers um so how do we manage all of this madness uh and that's through a bunch of tools called uh orchestration tools um so i've got uh kind of the big ones here um so docker swarm is sort

of the original you know docker is kind of the original uh commercial one docker swarm is their clustered version it is it's very very simple to get up and running it's very simple to use um it works fairly well uh it's all but deprecated at this point um it doesn't have all the bells and whistles of the newer stuff um and you know uh hasn't really been a whole ton of movement on it in the past uh year and a half two years so you know it's kind of with docker the company being split apart the way it has it's kind of going away uh the other uh the other big one that everybody sees

that everybody jumps up and down about is kubernetes uh kubernetes came out of google it's an open source product it is based on google's internal orchestration system that they call borg kubernetes is a monster it is complicated it is uh arguably resource heavy um it is it is not something that your average person spins up in their basement for fun um now there are those people um they exist uh but kubernetes is is a full-time job this is this is not for the faint of heart uh there's another one a little bit lesser known uh from a company called hashicorp it's called nomad so hashicorp is kind of famous for their vault product which is a secret storage product they

also have something called console which is a a service discovery system um and it also do some service mesh stuff uh and they have a bunch of other stuff that's been coming out lately nomad is um a fairly lightweight orchestration system that has an enormous scale they argue they can scale better than kubernetes i haven't seen that in practice but there are some big places that use it and nomad is is pretty simple it's a single binary you put on the system and it just does it does the orchestration uh it runs using the the same raft protocols that console and vault and all the other use um and it is cross-platform it runs on uh windows linux and mac

um so you can do uh and because of that it runs you know it's identical commands to run across all the systems so you can run entire clusters with this and it is if i understand i'm still learning a little bit about nomad but if i understand correctly uh it is relatively easy to set this up to run across regions as well as across cloud providers there's there you know the local areas need like 10 milliseconds or something of time uh where where it uh uh up latency rather between the the master so they can do things but there's there's ways to put it into modes that allow longer stuff uh and the last one is marathon um

marathon was sort of a i believe it was out before kubernetes uh it was intended to be the big you know we're gonna crush docker and and this is what you're going to run big systems on it's based on the apache misos project and while i love apache it seems like all of their products that they come out with are incredibly complex um i don't know how much marathon is out there anymore uh it hasn't been updated in a while um it's been over a year at this point um and i'm not sure how much it's used uh five years ago it was the big thing you know it was it was gonna win the world over and then

kubernetes came in and kind of you know stole his lunch so um so those are the big orchestration tools there's a ton to go into on all of them but that gets you kind of started in which direction to look at so to summarize containerization so we're looking at small footprints which is great because they're they're small they're easy to manage move them around all you want uh rapid deployment um you know once the once you get the first container out there and you get that that sort of base layer image updating those is is really quick as long as you're not changing those base layers updating them is really rapid and even if you are changing the base layers

again containers have small footprints um please stop deploying full ubuntu os containers that are gig in size you don't need that um you can get an ubuntu container down to you know a couple tens of megs uh or use something like alpine that's literally three megs and has everything you need uh pristine environments um you know test it continue build a container you can test it uh the one the really nice thing that i love about containers is um you know back back when you know with developers and developing and deploying things it would be that you build your uh you build your application you deploy it to development and you test it and it works and it's great and it's

wonderful and then you take that same application you deploy it to qa and suddenly it's not working so great because something in qa on the os is a little bit different or there's a config kind of out of place you still can have that with containers but it's much much less because the container that you tested in dev is exactly the same container that you're running in qa and the exactly the same container that you're going to push to production and the only difference that you should have between all those environments are things like passwords uh uh addresses to uh external resources like uh databases and configurations that's it there's really no other difference you

should have the exact same binary runs in all of them and once you test it it should test clean everywhere else so rapid deployments with a pristine environment make things much smoother to deploy uh containers are immutable if you screw up you know get in there mess around with it screw up the container kill it and restart it poof your back to start it's it's wonderful they are relatively secure um you know containers are new uh kubernetes is still new uh there's going to be like any new thing uh there's gonna be a lot of of exploits found um just like there was with virtual machines just like they're they're they're it you know continues to be with

with uh bare metal systems but they're secure they're designed to be secure and while you can break out of them and there are ways around systems if you build it the right way and you put you know your base level security in place on your hosts and and in your containers you you're kind of ahead of the crowd uh they're portable um we've talked about that to death and and ci cd friendly um you know building building a a continuous integration continuous deployment workflow for a container is brilliantly simple because the once you move it to that first environment moving it to the other environments is exactly the same so whereas it would take you know dozens

of commands to get things into the dev environment and then all sorts of changes to those commands to get into the qa environment you know these days i'm finding that i can do deployment to each environment using a single command and it works exactly the same the only thing i'm doing is changing the destination for that that container it's great