A Practical Guide to Securing OpenShift

all right my name is little cramp unfortunately as you may have just heard Chris Grimm is not here you know scheduled things better liam is out in San Diego I will be giving let's talk today on basically I have us here your openshift not too different than securing any of your your other systems or anything it's just a new technology so most people aren't too familiar with how to secure your kubernetes cluster and all that so there's me see you've been selling it Red Hat primarily in the public sector so mostly supporting government customers around the country with their various mostly OpenShift but also ansible and you know there were products Chris was on the consulting side as well

he recently switched to become a sales architect primarily supporting the army all right today were to talk about kind of the overall state of security how the hardened your OS which kind of the first primary I hopefully was familiar with us we can do that pretty quickly then kind of why you want to do container signing certificate management kind of how openshift handles that using your secure pipeline to kind of build a better image and limit things running in production and then multi-factor authentication

developers frequently just want to develop their code they don't really care about security although as we'll get to in a minute they they know they should but they they just frequently have their own deadlines they do know they need to get objectives done and they just generally don't have time to care too much about security security thinks you can just encrypt it super well and then no one get into it everything's good but you have to kind of balance that with usability it doesn't matter how secure it is if no one you can or will use it and then DevOps frequently is fighting fires one at a time they don't have time really to you root

cause analysis frequently something that a lot of companies need to do a better job out you you so report from about a year ago when say it was all 2018 might even summer but 33 percent of the people that were surveyed have reported a webapp in the last year 48% of the developers didn't think they had enough time to work on it because they have other priorities they've got other deadlines to me and 80% of everybody security ops developers agreed that everyone should be playing a part in it they just don't have time or priorities so people know it's a problem we just have to get management on board I'll try to fix it yeah and part of the thing one of the

most basic steps is compliance by keeping your your apps dependencies which is a usually a big one with development even those all of the day 80% of attacks use vulnerabilities that ever known most of them don't use zero days most of them don't come up with their own attacks there's something that's been patched frequently for months or years and people just never got around to it most of the ransom wires that you see at hospitals because governments a big one last year with the credit monitoring those are all vulnerabilities that have been out for months that people just never got our under passion well that's why the first thing whether it's whether you're doing containers

kubernetes just or just your OS you need to harden it so one of the the big ones is open s cap DoD's that lots of federal agencies use it it's all out there for free this helps write the guidelines for it and everyone can go through figure out basically some settings that are good general guidelines you usually can't apply every single one is strict as they want but you should at least know why you want to leave this vulnerability open or not necessarily butter ability but this this permissions level higher you need this service to run so out of the box pretty much all of our Red Hat products because we do support so many

federal customers as well as financial institutions though you can select a lot of these as you're installing the operating system and just have it at a base hardening already so you got the PCI DSS which is the one financial institutions use yeah the DoD DISA Stig's yeah from Augusta systems and you can kind of select all these out of the box go ahead and start it no good point we've already tested it it's supported you don't have to worry about breaking support if you run into issue you can open up tickets with us get some help so you can also use ansible to help so using things like this which will generate the fix and run it through

and you cannot go ahead and harden it in kind of a automated and repeatable way and some of them no there they go all right so one of the big things with with DISA and stage and all that there's a webpage into the lockdown IO which has a lot of play books out there a lot of them are under the mine point group for some reason that's the group that we usually work with so they have the official Red Hat content for kind of sharing your systems locking down and being able to apply them in a consistent and repeatable manner these are some of the common base lines that are that are out there that people

we're going to use same thing we talked about so one of the things that we've been doing a lot with our customers and more and more people are going to use kind of writing your infrastructure scale and which is kind of creating sort of securities code by having repeatable processes that are pushed but you don't have to worry about this person making a typo maybe this will this person wanted to put these security settings on but these other person's anything bigger necessary so by having it having answerable playbooks or however you like to do it you can basically have a push-button deployment and put some configurations in place put a few new servers Adam to the VLANs whatever you

need to do in kind of an automated repeatable back

so and one with one of the big things that a lot of people don't like to do because it does take longer to set up just doing your part back you're a role based access control and these are the same things you need to do on servers you need to do on your clusters basically set up your groups it'll take a little more time especially when you only have a few years you may not think me of the groups but frequently when you start onboarding people it accelerates rapidly and you don't have time to go back and figure out alright how should we set up these groups what we need to do so by setting them up early set up

your permissions you ensure that every you know what permission sets every 1/2 you're not doing one offs all right this guy's a cluster admin this guy's a viewer this guy's an auditor and then having to remember what everyone has thing especially with containers and Buddhas and limits one of the things you can do in an open ship just say I want this project to only have this much compute this much memory storage or this user and one of the problems people have is either intentional or unintentional basically ddossing themselves you can think of all the resources and not even realize it if you don't suck your quotas and limits ahead of time take yourself offline so one of the things that kind

of helps with limiting what's running on your system is doing digital signatures on your containers so kind of the same thing Microsoft has done with the updates and drivers and many other vendors by saying everything it doesn't prove that it's secure but you know where it came from you know wasn't some site loaded by the project you know isn't a rogue application or anything running on your cluster and kind of one of the ways that most people do that is you have here your built image either build it yourself so we do build the image a lot of times with an open shift or let's say you go out to docker hub registry that Red Hat or any other

docker container registry you take the add it into the open ship registry at the same time you can do when you open shift API call kind of sign it with the internal signing certificates that you set up it comes back down pulls the image you're signing thought it will spin up a new pod just a client hold that image over signs it with the signing key and then puts it back in the registry so why you care about that is in the traditional path you have here frequently with the deficit cops pipelines you've got the developer he does his stuff shuts it into the registry it goes over the test goes over to fraud and there's no guarantee that

the developer didn't just go ahead and push it all the way to operations so by signing in you had in this new path where it's signed and then you can restrict your openshift cluster so that only the sign for fraud runs and fraud so someone can't just run a rogue application you can't just skip the test process then you restrict you has those signing not just anyone can sign they have to go through whatever process you set up and that kind of helps for technical production so kind of just overall how did gets work there's a high level you got your root CA generally those aren't connected to anything general best practices people spin up a root CA been out a couple sub

CIA's shut it down just from the internet from everything that way that doesn't get compromised because if someone come around here ECA they can generate all the sub series and everything they want you keep that one on flying case you every to revoke any other sub CAS and at that point if you do you spin back up revoke a sub CIA open up a new one it knows what's revoked and shut your root CA back then so you can even have multi level sub CAS that are signing your different certificates as long as they have that signing authority so an open shirt open ship has its own internal CA that issues all of the certificates continuously on all the

rank pause as well as your nose your masternode your structure nose and your and these are generally they're only good for thing by default there's 72 or 96 hours and then issued the new one the one of the one of the thoughts behind that is as it is its own sign you see a bike quickly revoking the search name the key current with what's running out there and things don't expire and then as long as you trust the open chef CA then everything that it issues is trust as well so that stops those pop-ups being popped up do you want to trust the sir and there's other issues so there's a couple ways you can do that you can have

if they trust it a lot of federal agencies and others won't but you can have them issue a signing serve to openshift if not for example one of my clients we have our IDM issue of siding insert to it and then they trust our IDM but it doesn't have an official

you one of the big things that people are using in general it quickly his pipelines he doing kind of been more of an automated fashion you have all your different pieces tie-in you got your pipeline delivery so frequently people use Jenkins for that or builder and then usually way we have it we have it running on top of open ship that's another and then that can be running on top of whatever your infrastructure is with a Red Hat air metal whether it's AWS google measure and then you can lock all that down it kind of like we were talking about before use near open that's your big setting and then you've got your answer automation which can run across all

those levels and help do that infrastructure spin we talked about as well as automating the process so the first thing you that has you need your short school repository in our case we're gonna be using it and you're gonna build your hat so once your apps built it gets sent over to do some quick those unit-tests can be anything you want they can be either an automated approval method so let's say you want to check make sure one equals one and as long as you trust that you can just have it automatically check and I'm pass it on to the next level frequently you kind of want to start off with more of an a

manual approval process for two reasons one you want to make sure all of your checks are doing what you think they are kind of validate that you didn't accidentally put one equals 100 somewhere you don't want that accidentally making it all the way through the other is it kind of helps build the trust for management the management doesn't trust automating that's all the way through when they're used to having a test team that may take a week so by proving it you can slowly start turning these into automated gates and make this from maybe a week two weeks some cases a month process the minutes but from there usually you do static code analysis same thing you want to do

for your app running anywhere whether it's a container whether it's running on a bare metal server running about somewhere you want to make sure your code is so you can set up gates again for this there are plugins for Jenkins before to fight past all of your main deck code analysis tools that can go ahead and automatically ingest the code analyze it based on the thresholds you set so many criticals the code quality you may care about you can check all of that and make sure there's enough unit test documentation and then based on that either make it a push button to start or automatically pass or fail if bestest you go ahead on moving store it into

your artifact repository so frequently you got things like you kWe nexus j frog our factories another big one you're gonna build your image and store it up there so that you have something to reference for your test and development production per layer so once you got your container you can go ahead and then also do a container image scam so there's a couple couple different main tools you got Claire which is skin containers black Doug Whitlock as well as J frog x-ray are the main ones but basically it's what have you guys prefer you just plug it into your pipeline almost all of these either have AP eyes they can write your own plugin from

Jenkins or even have Jenkins plugins and the same thing you can do I'm a DJ make sure there's no vulnerabilities on the container image itself so your source code may be good but let's say you're building on an old version of my sequel that has the vulnerability in it this will change and make sure that the whole thing and make any reports or vulnerabilities and then you kind of do the same thing with the gates if there's too many vulnerabilities or they're too high of a level let's say they're criticals you don't want to have any of those and no more than one high and you can go ahead and do that and the thing you can do at the same time as

well let's pick off an email from Jenkins or the pipeline that says hey remoted it but these are the findings you may want to look at it make sure you get them scheduled into your sprints or whatever method you guys are doing in yeah actually so the way most of them were container or ingest this container and kind of analyzes the binary bits okay that's usually one of them so there are so you do have the doctor files though so you can do some analysis on that but usually these don't do that okay that's more kind of a how you want to set up the container so one of the things the OpenShift won't allow you to do by

default you can force it is we don't allow containers to run this route that way you can't just pull down anything and run it or install anything most of these containers for example some of them may not even have be I probably don't have W get they don't have your compilers on them and we don't want them to be able to have it if they need to add something they should go back

kind of more stands for compliance if you're if you're have a customer or you care about that so you made sure there's no vulnerabilities now making sure that your my sequel here for example is running with the right information sets that distinct required so you can again you're ensuring your compliance having a good baseline limits your number of exposures and vulnerabilities in production so at that point that's when you can start doing that that plugin that we talked about earlier with the tested QA to prod and you can go ahead and have this create your deployment it's already passed everything whether you clicked yes go to the next gate or it's automated doesn't matter once you get

into this step it starts moving things through to the next level the men you promote it frequently the way people do it is especially if they care about the security is like we talked about and sign your containers so your devotee wake Buster and only run images that are signed by development your your production cluster can only run images that are signed off by QA so that kind of ensures people don't skip steps there and limits what gets so one of the things is you kind of want to continuously scan your containers because as we talked about if people can run whatever they want they may for example in October last month there was a graboid krypter

krypter jacking images that were found on docker hub that if people didn't know better they could go hold down this my sequel image they thought was clean it had a crypto minor hitting in there and I went continuously scan for me just make sure you didn't pull down one of the images another problem is even before people have been baked in once people would find those containers running his route and go ahead and pull down just do like a double you get the website start up a project running on the background of your container you think this is only hosting your database it's got a crypto minor in the background also eating up your CPU and

resource but by locking down who can run his roof and continues to scan your containers so that there's not row containers and keep an idea of what's running in your system it's just like BMS when people spin those up generally these days people have process of jackin how many VMs are being used what the resources counts are and everything containers should be the same but since it's newer a lot of people kind of don't really fully understand it yet but you need to make sure that there's not ribbon just running on your

so one of things that comes up especially federal space is your authentication and authorization so with openshift with these are kind of decoupled processes so you can have the authentication service call out to a separate authorization service and that's how we frequently tie in things like two-factor authentication with smart cards things like that and just go ahead and offload that whatever your this thing infrastructure you have in place so with your typical authentication piece you have the user sign into a web console that go heads and passes along yeah Sanwa proxy on the back end so whatever your identity provider is whether it's any LDAP or some other source the user up then get their

credentials get passed over there it checks to make sure you are who you are and then it sends it on back and then you get into the system so you can go ahead and offload that piece outside of openshift handle the same way you do the rest of your applications services and not have to change anything there with authorization by the same thing here you your user administrator logging in its getting your group sinks so Elda the way we frequently have it set up at the moment is the users and groups to copy it over into the internal groups and users on open check and you continuously syncs those usually using ansible tower jobs make sure everything

stays up to date and then you can use your our back to limit who has what permissions what they can do are they allowed to create a new project they allow the start of project they had users who they have put as limits and they change those and they change the parameters on your application so one of the big things of course with containers just scalability so you can quickly scale up the from one pod to 100 but you might want to limit who has that authorization but two-factor authentication generally what people do you kind of use that same authorization model and with one additional step where it falls out here server smart card server validates that you are who you

say you are that you are Phillip Graham your cert is still good on the smart card and then it go to the head checks what your permissions are again against the LDAP and your groups and role based access you so one of the things of course with this is that works for the web but a lot of times either by choice or because there isn't a nice web GUI man port need to do things via the command line so via the OC CLI and one of the common things you see people using if they're using to factor for SSH is there's buddy cat which is out there for free you go ahead so here buddy authorization looking under

Spanish marker I pass that on to the server and get you an via SSH console so you can do those OC commands by the by hand

projects and stuff out there to help you we generally try to keep all of our stuff if we can and aren't limited by the customer we put it back up on github so others people at Red Hat as well as the general community can use it and learn from it you got to get help official a Red Hat official as well as the openshift repositories I have here your LDAP group sings his love your sam'l playbooks there's a nice redhead article as well as a blog post on how to set up a multi-factor authentication as well as Red Hat learning-dot-com has kind of like a forum style group so there's a bunch of people from Red Hat

as well as general community users if you have questions you kind of go get some community help if you don't have support you so one of the big things federal especially is one of the things that may be kind of weird if you haven't worked well worked much with government there's 140 - to says you have to use a supported and approved encryption algorithm or plaintext so you can't use you can't use a bad encryption algorithm it has to be validated and approved or plain text and the reason for that is they want you to know that you're using plain text rather than having a false sense of security that you're using something that's really powerful so you

can't do three days on your code or anything so so far OpenShift is not supportive but tentatively it's on the roadmap for open shift 4.3 that should be coming out by the end of the year and the so that's for open shift itself so one of the big things is the open container storage which kind of allows you the dynamic provisioning and growing you just say hey my container needs five bits of space or need the internet space the part that allows you to dynamically allocate that space reclaim it use it as ye as though inter storage they haven't definitively said whether or not that would be in the so there may be something you may have

the UI with Phipps but not turned on your storage nodes you do some more the research we mentioned so you've got the my point lockdown group that I mentioned for getting some Ansel playbooks open lock things down there's the compliance is good project that helps you kind of get some baselines for setting up here your applications to be infrastructure specified no CP multi-factor authentication that's the blog post I talked about I kind of goes through and also references those get home projects and Fear Factory software factory and it goes through helping you to write your code using the pipeline in those days that's about it I think I covered all the parts Chris was going to but since

he's not here I may have missed some stuff so you guys have questions you go back and you slide thanks man

you