Shining Light into Container Cluster Visibility

everybody it's time to uh go for the next uh uh talk about before that I have more uh or another prize to give away so um yeah Mo prizes and this time I'm gonna enlist will to take out a ticket I think we need to get a bigger so okay [Music] wait what are we giving away this time oh yes good good question oh this question uh this time we are going to be giving away a one-year hack the box subscription because thank you for reminding me to actually say what we what we were giving away all right who has lucky number two three three eight two four six this guy this guy [Applause] get your email um so real quick before our next presentation I have been asked to um make sure that if any of you ask a question please speak up a little bit we do have some people who are um checking in virtually and I want to hear your wonderful questions about horses and ducks and so on and so forth all right anyways uh our next presenter is nathanielquist he's a principal researcher at unit 42 in prismacloud his talk is titled shining a light into container cluster visibility [Applause] awesome hello it is so good to see people's faces it's awesome this is the my first conference in three years since before everything happened so um I'm glad you're all here um some familiar faces very happy to see some of you others not so much and I'm kidding all right so um a little bit about container security and visibility on it so I just kind of want to just kind of pull the audience a little bit um who's using containers either Docker kubernetes right now awesome who thinks that we have awesome visibility into containers right now okay awesome next question who's used service meshes few hands all right cool um so uh I I left this relatively kind of high level there's a few things we can drill down into I'm always open for questions um and with that we'll just get started um my Beard's gotten smaller I'm trying to be a little bit more professional these days I guess I don't know why I should just let it grow back and say who cares um but my name is Nathaniel Quist um sometimes I go by the letter Q I've actually just recently been promoted I'm now a manager of the cloud threat intelligence team at under Prisma Cloud um we are super dotted line to cross with unit 42 so like well I'm I work very heavily with unit 42 and I work with them I'm actually prismacloud so it's a bigger umbrella that does all that stuff um but unit 42 is a lot of fun really enjoy working there a lot of very very smart people um so yeah a lot of fun um some things about me I like computers I like Cloud I like RC helicopters I really love RC helicopters they're a lot of fun um first time you go inverted and then you actually go up you're going down right to the ground and you break everything it's a lot of fun so so all right all right so uh all right a little bit of the agenda today we're going to be talking a little bit about the Dark Side of cloud we're going to talk a little about the challenges that we have with Cloud just kind of in general in the higher higher view we're going to specifically zero in on uh service meshes uh and then we're going to talk about a couple other tools that we can actually use to get a little bit more insight into how containers are operating so the fun side let's talk about the dark side of the cloud I thought about doing kind of like a Pink Floyd sort of thing with this just because it's similar and all that but I was just like ah we'll just stay with our actual dark clouds so all right so uh first off where are we just in terms of cloud security all right um from an Intel perspective uh what we're seeing is we're seeing a lot of identity misuse so um IAM identity access management within all of the clouds within hybrid clouds there's a lot of complexity that goes into these environments um so with that we have a lot of leaks as Greg alluded to uh in his very first uh keynote here um is it your very first keynote I think it's your very first one too right second one awesome third it's even better okay um fourth it's the seventh um all right so um identity is huge and it sold um readily it's easily compromisable um and so that's some of the reasons why we're going to be looking at some of this um some of these big attacks that we see um IBM um huge breach on clouds um you know hundreds uh it's like 120 million uh different credentials are stolen and readily available for cloud environments it's it's insane mother of all leaks within Forbes um the one that I really kind of want to zero in on the most right here is in um solar winds the bottom right hand corner right here solarwinds goes under the radar a lot actually it durability doesn't because it's very huge and very large but when it comes to Cloud attack you know specifically it was a solar a supply chain attack but it was really a cloud issued aspect of solarwinds so solarwinds Orion which is their Cloud offering for um you know hybrid Cloud environments hybrid on-prem Cloud environments um and it's how they operate that over a larger structure and they were able to uh put a container an infected poison container into that supply chain make it part of the hot fix and that's how they gained access to not just um tens of thousands of organizations but 40 plus U.S government organizations as well including his U.S treasury um a number of different uh places on that so it's a very serious Tech and it's also apt-29 or nobelium um as as Microsoft is calling that um so very uh profound attack when it comes to Cloud attacks in general so some things that I want to look at these are I just pulled off of Docker Hub I actually pulled 50 of the very top most downloaded containers of all time right and then I compared these things across Showdown and I crossed it Acro compared that across uh um exploit DB so here's The Showdown attacks as you can see here um uh Alpine it's like in a crazy amount of downloads so just count some decimals here there's one two that's uh 7.8 billion downloads of Alpine it is the most used OS within Cloud operations okay then Ubuntu and then obviously uh you know nginx python BusyBox postgres redis shows up and it's uh 3.8 billion so it's pretty busy pretty common um sort of thing um within Showdown these are how many exposed um you know instances or products across all these environments uh obviously nginx tops the list by a lot um I used a a logarithmic scale on this one so um this is definitely in the 300 millions are that are are exposed um you know but you can see there's a lot of these uh particular uh instances that are exposed I was kind of interesting that memcache doesn't actually have any exposures maybe my Showdown search was off who knows but it's a it's pretty interesting um on this one so if we go over look at the exploits so these are how many are actually available to the cloud or to everybody if we look at exploits obviously WordPress is the biggest one has the most available exploits because that's awesome um as was it how many exploits are popped like 10 10 every five seconds or so um you know but there's some interesting ones in there httpd obviously used uh pretty heavily uh my sequel which is pretty easy so if we go back and we look at MySQL MySQL sitting there over uh just over a million exposed MySQL databases on any regular time in a regular basis and yet it's like the third most exploitable Cloud container that we have out there right now so kind of gives you a little bit of idea on that I love this graph because it looked to me it looks very pretty so what I did is I took all the exposures and I referenced and I cross referenced all of the different Cloud providers that are a part of this one and I'm just going to point uh to uh one of these particular things and I just kind of want you to see like which one that is right this one this one this one this one no which cloud provider is that digital oceans do we ever talk about digital ocean when it comes to x-fosed and exploitable databases on a regular basis attackers use it all the time sure I'll just set this up and we'll line we'll have payloads come from there we'll just use it as C2 infrastructure but how many people are actually using uh digitalocean as their their actual cloud provider and the biggest one on there believe it or not is Maria DB how much data is held in mariahdb on all of the cloud infrastructures so there's some interesting layouts and Footprints these are just 20 of the top 50 um Cloud containers that most downloaded Cloud containers that are out there so I'm going to Pivot from that just that's the escape the scope that we're living in right now and I want to talk specific about Cloud threat actors Cloud threat actors is something that is very near and dear to my heart it's something that I want to Define specifically so we just created a cloud threat report from unit 42 myself and j-10 who is the other uh researcher for this we looked specifically at who's attacking Cloud environments and if we can quantify them or index them in some way in order to do this we had to kind of Define what a cloud thread actor is it's not a normal threat actor normal threat actor from nist um you know 800 150 says that it's a an individual or group that is compromising uh you know infrastructure that's not their own so well that works for cloud sure to be a cloud threat actor we do came up with this definition an individual or group posing a threat to organizations through directed and sustained access to their Cloud platform resources services or embedded metadata you can be a threat actor and tangentially compromised Cloud environments and cloud services doesn't mean you're necessarily targeting Cloud environments you just happen to spill over into Cloud environments so we wanted to look at what are the actors that are specifically zero Landing zeroing zero Wing in man that's a hard one zeroing in on cloud environments so this is what we came up with and we created the very first Cloud threader Cloud threat actor index now some of these there's only five on there right now and a lot of these you might realize that these people organizations are actual crypto miners at least that's where they started however there's something more interesting about how they operate if we want to look specifically at Team TNT which is the very top one we're calling them the Adept and this particular asset X you know standpoint they're considered to be one of the most sophisticated Cloud threat actors specifically for their Cloud enumeration techniques and their credential scraping capabilities they yeah it's all bash scripts and it's all very boring and it's very open source and it's like oh come on and they're supposedly retired we'll see how that how long that goes on um but uh their ability to compromise multiple Cloud environments not just Amazon uh not also Google um very very impressive in how they're doing that Watchdog is considered the thief they're very technically Adept coders they do a lot of go language however they like to steal things too so they use a lot of Team TNT access they'll use kinsing scripts as well just to kind of hide and mask their operations inside of those um can sing it's all about money they're also really good at Cloud uh credential scraping rock is an old time I've been around a long time like what they started out in like 2014 um as a as a ransomware called iron ransomware um and they subsequently just kind of moved into Cloud they're still not very good when it comes to Containers they're sort of okay um but uh um they're really they're pretty good at just mass quantities of compromises and then 8220 which is spin off of the rock um they've returned they took a break for a while and they've come back and they've hit um pretty strong when it comes to um credential access and containers specifically so we kind of want to look at is we want to look at like the common tactics that these particular actors use across their environments um and we put these across miter attacks framework and we found that all of these particular techniques are common across every single one of these particular actor groups and lo and behold they're all identity focused every single one of them they're all looking for cloud accounts they're all looking uh for you know access tokens for sessionized application tokens if you look under the credential access they're looking for specifically private Keys Cloud metadata apis they'll just go into that C2 node and just do a metadata poll and well they have Cloud credits it's pretty fun and then they'll look uh pretty heavily on Discovery as well but they all have access to escape a host and they all have access to a container and uh resource Discovery so this kind of led me to this kind of idea it's like we haven't spent a lot of time talking about the actual techniques that cloud actors actually use against Cloud against compromise containers so that's kind of the essence for this particular talk um all of the unit 42 atoms all these particular access groups are part of a unit 42 atom um it's just kind of a common framework everybody's kind of familiar uh with six taxi formats and it's all Json encoded so it's easily decodable um build it right into your uh your particular security tools um and it's all free which is pretty awesome so please go download that um if you want links please let me know and we'll give you those those links so here's the challenge uh that we that I currently see within Cloud so here's the logic that I kind of want to go down there are a lot of containers used much more often in the industry uh you know we saw the top 20 list um there's only one of those containers that being redis is the only container targeted uh application that we know of right now currently uh for cloud environments redis is the only one that all five of those particular actors use on a regular basis the other ones they used and even make the list was Oracle weblogic I think think PHP um you know struts patchy struts those are all very common those are all used pretty much by all five of those particular actors however there's you know 19 others that are more downloaded and more exploitable than those five that I just mentioned so there's a whole bunch more that's out there there's a lot more exposed containers in our industry than I think we realized no one's really looking at digitalocean one because it's free and cheap and who cares um and because no corporations using it's all individual people right soda you know I know of a few organizations that actually do use digitalocean as part of their their actual production environment which I guess because it's cheap so um all right so um but my biggest thing is we are not seeing these Cloud threat actors specifically Target why aren't we seeing them Target WordPress you know are we just not seeing them we just don't have the visibility into their particular malware sets Etc so here's the the things that I think we're missing uh one is everything's fine don't worry about it we're all good uh don't don't don't care just don't even want to look at it another one is that the actors are just better at hiding their tracks they're just really good at skipping across everything we're not seeing things organizations also have very large gaping holes in their security perimeter in their Cloud environments I'm not going to go into a lot of this why I can say that that identity and access management the identity the specific roles behind identity and access permissions that organizations across the board we looked at over 200 different organizations 99 of them 99 of IAM policies are overly permissive it's insane the number three most used IAM policy in all of the organizations right now is administrator access within AWS in Azure the number two most used identity role right you know IAM role is owner that's access to everything in the cloud what happens if that key is stolen or compromised oh it's secure you know it's in our GitHub and GitHub is Totally Secure right sure interesting yeah all right so my argument is that we don't have enough visibility in two containers specifically and if we do have the visibility into those containers we are not doing a very good job at offloading that data and analyzing it in any way and why is that all right so let's go to the Rosetta Stone for cloud tools just real quick let me go over a couple Cloud workload protection all right so it is the ability to see actual containers in their operatings in runtime right so you can actually see when a container stood up you can see the process is running you can see it's Network traffic you can see all that good endpoint detection stuff okay so let's say an attacker compromises the front end system let's say this is nginx because it's the most uh used one that's out there they're able to move laterally from any number of reasons either they go to um you know they they run do run C and they escape that and go up into the uh the host and they move over to whatever container they just move laterally follow the traffic into that back-end database whatever it may be inside of that backend database not necessarily exposed to the public but most likely is as we saw with like mariahdb and stuff like that there's probably creds in there in some way or you got to the host and that host has creds available so we take that Dev key because most likely it's a developer engineer that actually just you know stood this up and they just have keys that are exposed in there just locked inside of that really nice you know dot AWS backslash you know configurations or credentials and they just take that key and then they can then if that IAM policy is compromised they can have access to everything that that particular IAM role does if administrator access is that I am keep which it most likely is or owner is that I am role like it most likely is um they have access to all this stuff but that's kind of where it stops that's all a cloud workload protection can give you it's only an endpoint monitor it can only see the containers right you cannot follow that IIM credential into that larger Cloud platform okay that's the end of cloud workload protection so let's look at what can look at that bigger picture the cspn the cloud security posture management tool this is GRC or governance regulence governance regulation and compliance uh for cloud environments it can see API connectivity you can see user logs in it can see hourly scannings of how uh your particular workloads are configured all of that good stuff you can also detect drift detection in a lot of aspects if something were to change in configuration from start to later on it'll detect those things which is great it also has the cool ability kind of mixes a little bit with uh Cloud identity Services where you can do IAM monitoring you can actually see users that are changing permissions you can see all of that sort of stock all in that one single dashboard so if you look at that look at that we can see uh AWS IAM sensitive activities by user that's great where did it come from we might be able to see a container but that's great we don't know what started it we don't know if there was malware on it we don't know if the lateral moved to get that we don't know any of that stuff behind it that's the end of cspm It's Too Tall right it doesn't see the granular aspects so it can't see workload behaviors it can see IAM configurations and behaviors so we're going to look at a couple other tools these are not necessarily ready for my import uh this particular talk right now but um they're important to know CCS which is cloud code security infrastructure is code scanning vulnerability fixes vulnerability scanning misconfiguration scanning real-time fixes obviously is a huge bonus you can scan infrastructure as code your terraform or cloud formation or whatever you have and it can it could fix it right away but it's like ye