BSides LV 2022 - Wednesday - Breaking Ground Track

Name: BSides LV 2022 - Wednesday - Breaking Ground Track
Uploaded: 2022-08-14
Duration: 10 h 48 min 30 s
Description: A track where hackers new and old can show off their latest and greatest while interacting with our participants and getting feedback, input and opinion. No preaching from the podium at a passive audience. It is a place where presenters can talk about their newest attack or defensive research, tools

BSides Las Vegas10:48:301.3K viewsPublished 2022-08Watch on YouTube ↗

About this talk

A track where hackers new and old can show off their latest and greatest while interacting with our participants and getting feedback, input and opinion. No preaching from the podium at a passive audience. It is a place where presenters can talk about their newest attack or defensive research, tools, new and novel approaches to InfoSec and to talk about the upcoming areas hackers should be digging into. Talks can vary from 20 to 45 minutes in length and typically include demos (live or otherwise). Some previous talks include reverse engineering malware in Go, network forensics in an encrypted world, anti-honeypot approach, hacking crypto currencies, 0-days in online services, mobile phone binary hacking.

Show transcript [en]

[Music]

do [Music] [Music]

[Music] do

[Music] do [Music]

[Music]

[Music] do

[Music]

[Music] do [Music]

[Music] do

[Music]

you

[Music]

so [Music] so [Music]

[Music]

[Music] [Music]

[Music]

[Music] do [Music]

[Music] do

[Music]

[Music] do [Music]

[Music]

so [Music] so [Music]

[Music]

so [Music]

[Music]

do [Music] [Music]

[Music]

[Music] do [Music]

foreign [Music]

[Music] do [Music]

[Music]

[Music] do

[Music]

you

so [Music]

[Music]

[Music] do

[Music]

so [Music]

[Music]

[Music] [Music] so [Music]

[Music]

do [Music]

[Music]

[Music] do [Music]

[Music]

so [Music] so

[Music]

[Music] do [Music]

[Music]

[Music] do [Music]

[Music]

do [Music]

[Music]

do [Music] do [Music]

[Music]

[Music] so [Music]

[Music]

do [Music]

[Music]

[Music] so [Music] so [Music]

[Music]

so [Music]

[Music]

[Music] foreign [Music] [Music]

[Music]

[Music] good morning besides day two i can see that we probably had a few people who had a really good time last night we're about a quarter capacity out everywhere i look so that said thank you very much for being here we're excited to be doing this in person again this year thank you for bearing with us as we're learning how to hotel again um i do have a couple quick announcements uh there was a change in the schedule for some of the talks so t profits talk this afternoon it was scheduled at five and has been moved to six o'clock and it's been moved to the tuscany room uh changes up should be updated on the

schedule shortly if it's not there already but if anybody's looking for that that's where that's going to be all right so uh without further ado we have our keynote speaker for day two this is amanda walker i've had the privilege to work with her across a couple of different companies now and i for some reason she still takes my phone calls and agreed to come talk to us about something that's i think pretty exciting uh and uh we'll ask you all to put your hands together for amanda walker and let's hear what you have to say

thanks a lot the slides were working early there we go okay hello everyone my name is amanda walker and i'm going to talk a little bit about logs and time series beyond logs in time series and observability in for security and privacy use cases i'll start out with a little bit of an intro i'm currently at google leading applied research for privacy safety and security before that during the pandemic i spent a few years uh working for a small company called nuna which gave me the other end of the spectrum from google of a you know 150 person company with a 10 person infrastructure and security team done some work on a host of prior

companies some of some of which i've met some of you but i want to make one one important statement here is i am while i currently work at google this talk is not about google so this is not a story of how we do things there a bunch of the principles i'll talk about uh are ones we do apply but i'm not going into tooling things like that this is more of a uh trying to get some guidelines on how to think about observability from a security standpoint and through a security lens uh regardless of what tooling you're on regardless of what platform you're on so talk a little bit about observability i know some of you are familiar with the

concept but it's something that's gotten very popular in the sre community where basic you know traditional logging and monitoring is no longer really sufficient and so reframing of how you keep track of a system in production and understand what it's doing has sprung up under this term observability this came from control theory and if you go to wikipedia you'll get a nice control theory definition which is you know how well internal states of a system can be inferred from knowledge of external inputs and really what that means is you can tell what's going on without having to stop and take it apart um you're not just looking at looking at things from the outside um

so how do we do that um the first place to start is logs we're all familiar with those they record what happens there tend to be events that happened in the past and they're used for evidence after the fact you don't you can't prevent anything with a log it merely tells you some things that happened so that you can go try to reconstruct uh how you got there um building on top of logs are alerts which are more timely this is something that fires uh well when a condition's met and i'm just defining these as this terminology so that we'll understand what i'm what i'm saying for each of these and the learning criteria are always uh

specified in advance there's something you expect to happen or are worried will happen and so you write an alerting rule that that fires when that when those conditions are met usually trigging triggering some kind of process based on that and these two have been thoroughly used in production in security over many years most platforms come with logging alerting centralizing that has been an entire sub industry for a while but that still doesn't give you a lot of information in the moment maybe 10 15 years ago people started thinking more in terms of telemetry rather than recording events that happened get snapshots of a system state at a given point in time do that at repeated time intervals so you can see

trends over time time series databases became very popular at this point so things like prometheus and systems like that that can store things over over a lot of time uh let you perform queries either by time or by signal draw pretty graphics on them things like that this revolutionized uh revolutionized monitoring because you could you could sort of see the state as it was progressing you could see request counts climbing you could see latency increasing before it hit some alerting threshold and the big innovation here was that the logging and learning conditions were separated from gathering the data traditionally in logs you know you admitted you emit a log statement in code you say if such and such has

happened or if i'm running if i hit this function you made a log statement or test the condition and generate an alert that way very hard to change you require pushing a new binary all of that if you instrument something with metrics and then have a engine running on top of that looking at the metrics coming into your time time series you're going to separate that you can tune metrics you can silence alerts you can create new ones without having to push something new into production and things like that because this is capturing behavior not just state you can see how the how the state evolves over time and deduce things from that so this is sort of a step more

abstract but it's still slicing things across time and so you get a snapshot and you get another snapshot and you get another snapshot and you still kind of have to stitch things together so if you're looking for more granularity then okay what's the load on my server or how many api calls per second am i getting things like that i need to move slice things in a different way tracing is the most recent of these approaches and this is really what most most companies and most organizations that are focused on observability take as one of the distinguishing pieces which is tracing slices across a context instead of time so you can follow the path of a request

say okay i've got a request it's my web server which of the dozens of microservices that i am running does this request flow through does the data come back through what credentials are used you can follow things like that and so it's much more focused on causal relationships so request came in did this did a database look up did did that and so on this is useful for debugging this is useful for uh troubleshooting problems that are that are happening right now and it is these three these three pieces are usually called the three pillars of observability or sometimes melt which is metrics events logs and traces you'll hear a lot about that these are these are these are the the

basic pieces of it and what all of this gives you when you combine all of these is something very important which is you can start to answer questions about your system with queries rather than investigations one of the challenges of working in security especially on defense is it can be very hard to figure out what happened something happens you get evidence of something you know an event and then you have to go back and reconstruct and you can often have to interview engineers go look at code it's very time and time intensive process if you can query these things from the system directly that reduces a lot of toil that increases your ability to stay

focused on the problem uh visualize what's going on and so on and questions are situational they're not known in advance you know they're not something you could have run you could have written an alert to anticipate you couldn't have written a log analysis pipeline to surface the things they're questions you have right now often based on questions you just asked and got got some answer to and so you you can anticipate what types of questions you need to ask though so things like pivots and joins so i've got something i've got a suspicious specific suspicious session going on it's like okay joe has not logged in from singapore he's based in new york this might be a suspicious session i

want to look at his credential i want to say okay what was that credential used for since time x or in this user session so you're pivoting on that you may want to pivot on the geo location you're saying okay is there other suspicious activity coming in from there all of these kinds of things are very sort of data analysis tool centric they're things that haven't been traditionally applied in a general generic fashion to production data to security data until recently you also want to do expansion and narrowing of scope if you try to look at all of your traffic or all of your activity in a very complex system especially when distributed across a lot of different services

it can be very hard to see signals in the noise and so the ability to expand or narrow scope as you have more questions as you do these kinds of pivots and joins is very important i want to say okay i want to focus just on just on this session just on this user credential just on this type of user credential just on this subsystem and see where all of the requests coming in and out of it are things like that and visualizing this helps a lot we all have eyes probably all had time zone our eyes have glazed over reading log messages scrolling through things trying to keep track of what is what we're looking at

and so the ability to take these kinds of queries and visualize them is important um and one of the things that i've noticed in working with teams that are responsible for both production and reliability and security is operational queries and security queries often cover the same data but they ask different questions about it and this is this is part of what i'm what i want you to take away with is some of these tools are aimed at different kinds of tasks then then we have as uh as us people responsible for security but you can use those tools you can give it you know if you add some data to it you can then make it useful

gain a lot of these benefits and i'll let's see we've got a tool thing okay there is a proliferation of tooling for observability there are whole startups that are focused on this there are a lot of open source projects there's stuff about code understanding um if you go and search for observability on the web you'll see lots of flashy screen dumps of uh you know dashboards and graphs and charts and things and you know aimed at aimed at the executive that's going to write the check to go license the product more than the engineer that's going to use it sometimes but where tools can go beyond that and integrate integrate with your security event incident management system

integrate with developer tools so that people don't have to do extra work to make stuff visible in the in the either visible into the observability framework or pull stuff out of it that's the better incident management in particular is an area where unifying security incident response and production incident response can be very very fruitful that said in my experience this works best at small companies where everyone has a fair amount of situational awareness it does not work as well with huge sre organizations and huge huge security organizations coming together i have stories about that more appropriate for other venues but the kinds of defects that cause security problems and cause production problems and cause just

bugs in business logic etc are all very similar and so you can apply some of this tooling to all of those you just have to know the right questions to ask so i've now been i've now done a bunch of sort of explain why this the observability as an approach is a good a good thing um as with everything nothing is an unmixed blessing there are costs and benefits um let's start with some benefits you know as i was just saying there's better you can have better alignment between devoxx devops and secops you know if your developers and your security people and your production support are all using the same tools they're all looking at

the same thing they can just say send a link to someone say hey can you look at this that can reduce a lot of confusion require many fewer meetings things like that the other interesting thing is that given collecting this data making it available via apis being able to issue queries against it is useful for automation as well as people just like you can write an alert once you've got data coming in and being stored as a time series and you can do sort of behavioral alerts you do the same thing with uh with an observability system and those can get much more abstract so you can you can narrow the scope you can be

pretty specific on okay you know when this very unlikely set of things happens you know we get a latency spike on this trace back to whatever the incoming traffic was you can write much you can write better rules that are more understandable than you can just doing going just on data this is also where a lot of organizations are trying to apply machine learning you know once you have this labeled data about what's going on inside your system it's like okay are there patterns humans are not going to see you know looking for things like high latency or error rates or unusual accesses and stuff is is something humans are pretty good at but some of this is pretty subtle especially

if you in a security situation where you have an attacker who's trying not to be seen so being able to apply larger scale pattern discovery pattern matching is fruitful although a lot of this is is pretty new it's it it's it's exploratory there are a lot of there's some products out there so we're applying ml to uh to observability this will this will solve your problems this will show you what you need to do they're not they they magnify what humans can do they free up human attention for the more significant more significant signals more significant power patterns bringing patterns to human attention to act sort of as the exception handler rather than crawling through analyzing all of that

data many of these many many observability systems come with visualization tools uh that you can use when you need to explain what's happened to somebody in a post-mortem or an executive briefing or things like that where someone who hasn't had their head stuck in this problem for the last week or the last several hours or whatever needs to know at least the basics of what happened what the consequences were what the impact was what was done to mitigate it similarly to that it's nice when you're sitting down with an auditor for your soccer or fedramp or whatever audit and they ask questions about what your system does it's it's really nice to say well we have an automated system for

that here let me show you and issue queries show results uh visualize them in a in a way that they can see and they can see that it's not you know someone manually collated a bunch of stuff into excel and and generated a report from that all of these things are good there are let's see there are costs as well there if there's a flip side to everything if you have a system set up where you can do dynamic tracing for example where you can start to monitor state or even effect control flow without monitoring a binary you know using something like dtrace or bpf things like that that's really useful for doing non-intrusive data gathering

but of course it's also useful it's useful for non-intrusive data gathering by people or agents that you don't want to be able to do that some of the capabilities that have been put in place for observability like kernel tracing things in standard libraries things like that also mean that there are new kinds of malware that are harder to detect that don't show up if you are looking doing static analysis of a binary and so access to some of these capabilities needs to be treated carefully it's the equivalent of root access in many cases and so how you deploy it within an organization how you track use of it uh how do you how you detect use it becomes sort of a

meta problem uh you know who watches the watchers is always going to be with us as a challenge another aspect that i've seen pop up two or three times now is it gives you an illusion of completeness now you've got this product it gives you a a single pane of glass it's a buzz phrase that i've been hearing that shows you all your operational state all of the things you need to worry about all the things you need to pay attention to one place you can feel as though okay now i know what's going on that's usually not true uh systems these days are complex enough that no single person is going to know everything that's going to go on and

everything that is in fact significant to them so if you're looking at adopting some of this these tools that have been been written by companies or or open source make sure that they can answer the question that you know you'll need to answer make sure it's not just hi this is sort of dashboards v2 lots of lots of pretty graphics make sure that they can magnify your ability to answer questions do things that you know do things that you need to do to find out what's going on inside the system help you build systems where that comes along there's an illusion of well you'll get observability for free if you adopt our sdk or if you adopt our platform you

know every cloud platform has has these days has tracing facilities has metrics gathering all of that make sure that you can you can use that make sure that you're not just sort of taking it for granted of okay this is gathering stuff i'm all set and beware of buzzwords of the day a lot of things have gotten lumped into observability that are kind of log analysis to traditional log analysis with uh with a new label stuck on it one piece of advice i have on on that is is to try things out uh build something you know pull over an open source project deploy it on a on a sample project get a trial from a vendor who wants to

sell you their latest observability system and and kick the tires try it uh experiment with it uh do a red team and see if it'll pick things up you know if they say well we have advanced ai for detecting anomalous access patterns say great let's set it up and we have a team that's going to go generate some anomalous access patterns let's see if you where you can catch them sometimes that works and sometimes it doesn't uh i know of uh i know of interesting cases of both uh but it's a way to it's a way to validate the claims because there are a lot of claims there's a lot of a lot of hype around this right now

backing up a minute we talked a little bit i mentioned sdks one of the things that is kind of promising is that there are there's a lot more support these days in in sdks in platforms for wiring up observability features wiring up metrics collection um the days are passing finally where you have to go and you have to add okay i'm going to add this counter and i'm going to expose it here and i'm going to log it to this take advantage of those with with as you're building things but this can't really bolt anything on a lot of this does not work for legacy systems we you know you have what you have those systems have outputs some of them

you can inspect in situ uh some of the capabilities that vm providers for example are starting to create and some of the kernel tracing work in in linux can be used to inspect internal internal state of legacy apps but you you either look out or you don't on whether that ends up giving you useful data let's see how we're doing on time how we're doing fine um so that's covering a bunch of security aspects i mentioned privacy at the beginning these tools can also be used for privacy use cases where you're more concerned with data and credentials than you are with with attacks you may be looking for okay someone's compromised this data someone has stolen

a credential passing it around someone like that um the better you unders but the better you've marked your data in metadata machine readable format the better these probes and observability tools can leverage that to line things up with other things going on in the system not all of this state is sort of runtime metrics and the flip side of allowing developers to use these tools to you know the the positive side is it's a great debugging tool being able to see how a big distributed system works and and how the logic that they are trying to implement uh is behaving if you're exposing data that way be very careful i mean some of this is basic

data hygiene but having the trace probes emit an identifier that's opaque uh still lets you trace it through the system without without exposing underlying data so going back to the central piece of the approach which is to frame frame the problem as how do i answer these questions uh when you know when i need to find out what's going on inside the system uh rather than firing off an investigation it does require you to build systems and processes that can answer questions and not just display info so logs and metrics can come across as being very read-only they're sort of reporting adding queries to that really turns that from sort of a static record of what happened

into something that you can work with to understand what's going on right now and with that take some questions if anybody has them um

what do you see as the next steps as it relates to

what do i see as the next steps right now a lot of the observative observability products that are out there and projects in the open source land are extending on things that have been built already building on things like prometheus and grafana and time series metric storage and trying to load you know plot machine learning models on those trying to plot more abstract models on those i think that's a fairly limited limited gain i think that the next step is going to be there's going to be something that's along the lines of going from logs to storing things in time series so that we can capture behavior so that we can go find the next level of

abstraction i don't think anybody's found that yet and so i'm very interested in where people get frustrated with the observability tools that we've built now that okay this this still doesn't answer my question i need a way to answer x and that's going to trigger the next step of okay let's let's pivot our our framing from gathering logs and events and traces what's the next thing past traces i don't know what that is yet uh but there's something and it's going to be born out of frustration all of this stuff is a product of somebody getting very frustrated they have a problem to solve they can't find the information they're looking for they they want to be able to ask a computer

and get an answer rather than tracking down a human and saying okay what what what does this mean so i wish i had a clear view of what was next there i think that's an area of active experimentation uh across the industry so i'm looking forward to finding out what that

uh is better and or like how can we improve in like viewing the non-investigation with the query like have you done anything around like how that would be an improved process so we can get better answers from the data so the question was uh how do you how can we structure the query process so that we can get better answers out of the data there are a couple different ways to do that i know that there have been some applications of large database technologies so dumping things into something like google bigquery or another sort of large database that can do very rapid queries around huge amounts of data one example of that that relates to observability

is depps.dev which is an experiment being run by google where every every day we crawl all the major open source repositories and grab all of the dependency data and put that into a database you can query so that if you want to issue queries against okay has the dependencies for this package changed has such and such happen you can issue a query against that instead of having to go crawl it or inspect it yourself i think there's going to likely to be more of that as we identify what corpuses of data are useful across organizations to query open source dependencies was kind of an obvious target for that it's a question many people have especially after log4j

and people worrying about the next log4j things like that so those technologies have some applicability they're a little manual now you know you you to use them right now you you exit your your observability environment you go start typing sql queries at something or have a wrap a ui around that i think the success of those will start to push towards more purpose-built tools people have been logging stuff into sql and issuing grades about it forever but the time series database was a was definitely a step up from that i think we're going to see similar stuff for other kinds of signals i don't know if that answers your question okay

[Laughter]

yes is is that just not understanding of the space or is it like like whatever the influences or is it is there some integration like how why do we have so much stuff okay okay so trying to repeat the question for the stream um question is i've talked a bunch about having one unified source of truth for a bunch of teams to be able to query but it seems like we have more and more of those as time goes on more and more sources of data more and more things we have to uh we have to consult uh why is that uh and i think the answer to that is that we don't have we don't have that good unified source

of truth and we may never you know teams solve problems and build tools around the problems they're facing sometimes having one grand unified platform is not does not actually make life easier those of us who work for big enterprises you know doing it doing it the one true way can often be harder than just rolling something yourself i do think that as the analysis tools get better and they allow you to to answer questions about the data you're gathering that provides some incentives for federation for other kinds of taking all those multiple sources not necessarily all dumping them into the same store but being able to query across them and correlate things across them so that

if you have something that's generating timelines and you have something else that's uh measuring production traffic being able to do that kind of join i think is going to be be the way to handle that kind of scaling multiple sources so big joins across multiple data sources is its own research topic indeed in database land but i think that that's likely to be the most fruitful approach to that i don't think we're going to i don't think we're ever going to get to one single source of truth for data just like i don't think that all uh you know understanding everything going on in in in a system is ever going to be one that one seamless pane of glass

but where we can reduce fragmentation the better and where we can move information from one context to another without having a human having to copy and paste it or type in you know type in a query get an id right things like that that kind of interoperability i think is going to be what's going to make the biggest difference

going once all right one more question

what do you see happening with a big

together

so to repeat summarize the question uh what happens when something that's gathering data so you know a traceability probe an observability probe of some sort has been compromised so something is feeding grad data into your big central store that's that's the same problem that we face all the time how do we how do we detect that something is anomalous that does touch down the watchers who watches the watchers question i had there are levels levels to this that we have to think about you know how do we know that the data we're asking questions over is accurate uh how do we audit endpoints how do we understand if an endpoint is under an attacker's control

or someone is misusing the data that we're gathering i don't think there's anything special about that when it comes to observability you know it's the same problem we have with endpoint management in general with any kind of remote sensor you can look at the data under your and the traffic under your control you can look at what that remote sensor is telling you and if they don't match that's something to flag you know that and that is something you want to be able to see so you know being able to detect that on your end of it is is a good place to put some kind of sensor some kind of probe and feed into your

observability system like are we suddenly seeing a bunch of flaky phone handsets or unusual activity for or lack of usual activity from laptops or remote systems that are outside of our security boundary things like that so being able to answer those kinds of questions is itself an observability use case picking but it does highlight the performance of the importance of picking what it is you're going you know you you you do need to just not you can't necessarily deduce all of these things from data you think of in advance and so when something comes up learn learn from every incident okay we should be we should be reporting this we should be gathering this data in addition to this

sensor we've already got deployed so that we can detect when when these kinds of inconsistencies happen and you know like any any kind of automation it's easier to automate cases that you've already experienced rather than to anticipate well what could go wrong in the future that that is a lot of all of our jobs that's something no one has figured out how to automate yet

all right well i took less time than i expected sorry if i was talking fast any last-minute questions otherwise give everybody a bit of a break [Applause]

[Music]

[Music] [Music]

[Music]

do [Music]

[Music]

do [Music]

[Music]

[Music] do [Music]

[Music]

[Music] [Music]

[Music]

okay good morning everybody welcome to besides uh very happy to be uh back in person have a few uh little things to announce first of all i wanna say uh thank you to our sponsors uh specifically uh diamond sponsor uh lastpass and palo alto networks also the gold sponsor just to name a few amazon and vision and blue cats second a couple of housekeeping things please science your cell phones the talk is being recorded and streamed so we want this to be uh to be clean and also as a respect for our presenters if you have questions uh when we have the the q a at the end please come back to the front so the

presenter can hear and repeat for the people we're streaming to um beside has a very strict picture taking policy you're probably aware of that so don't take any picture without the explicit consent of anybody in the frame and i think i'm probably done with the housekeeping except keep your mask all time now let me introduce matt matt is going to talk about malware families anti-forensic and some botnets i guess so matt you have the floor

thank you very much evan hey so hello and welcome to my talk a tale of two malware families overcoming anti-forensics and foiling botnets in the cloud so let's kick off with some formalities before we get started on the content for today so first of all i'm matt muir and i'm a threat intelligence researcher at cadal security prior to working at kedo i was a macos malware researcher and i've got a background in devops digital forensics and operational cyber security so you can follow me on twitter at the handle on screen and i'll share the slides after the talk unless b-sides do that for me of course so as part of kedo labs kado's threat research team

i'm regularly involved in conducting research into new software threats affecting cloud infrastructure as the name of the talk suggests i'm here today to talk to you about two recent malware families we've been tracking both of which exhibit some interesting anti-forensics or detection evasion techniques so without further ado i'll first go over the agenda of the talk before diving into some real-life examples of cloud malware we've seen in the world so kick off this presentation by introducing you to what we call the cloud challenge next i'll move on to the first malware family coin stump as we'll see coinstomp is a cloud native malware campaign that exhibits some interesting detection of asian techniques following that i'll discuss abc bot

a botnet initially discovered by netlab 360 that we've been tracking since 2021 and has a longer history than we first realized finally we'll wrap up with some highlights of a more recent cloud native campaign give you some tips for detecting this type of behavior in your environment and finish off with some further reading before the q a session so i'd like to begin by giving a brief overview of something we call the cloud challenge despite a sustained migration to the cloud from companies across the globe organizations are increasingly susceptible to attacks which are advancing in both severity and in sophistication recent cloud-focused malware campaigns have demonstrated that adversary groups possess an intimate knowledge of cloud

technologies and their security mechanisms and not only that but they're using this to their advantage cloud compute environments remain an obvious target for such groups however we're now also beginning to see a shift of focus towards serverless computing and containers on top of this the ease of which cloud resources can be compromised has led to adversary groups competing over potential targets so unfortunately defenders haven't adapted at the same pace there are a number of reasons for this with visibility into cloud infrastructure being a commonly cited one but it's an unavoidable fact that the campaigns i'm going to cover are both successful in achieving their objectives and are widespread clearly attackers in the cloud space are

utilizing this to their advantage by employing ttps aimed at evading detection foiling attribution and covering their tracks so this leads me to this slide which i stole from my colleague chris who did a similar talk recently about forward cloud sake most cloud threat actors see themselves as the gif on the left or is chris hemsworth in the movie black cat which is a great movie of course however despite the clouds being a lucrative and easily exploitable target for many threat groups the reality is that most cyber attacks on cloud infrastructure are not hugely technical ultimately when it comes to technical resource this means that most cloud threat actors actually have more in common with homer

on the gift to the right so what we'll see throughout this talk is that although the tools are rudimentary so most payloads are shell scripts for example the developers make use of some nifty tricks and linux specific knowledge to evade detection so the first of two malware families i'm going to discuss today is what we refer to as coin slump coin stump is a cloud native malware campaign that we've been tracking since early 2022 and it was notable for its anthony anti-hardening sorry and detection of asian techniques so to give everyone a quick overview of this family coin stump is a crypto jacking malware campaign which exploits resources hosted primarily by asian cloud service providers

so for example tencent and alibaba cloud there have been a couple of theories about why these csps are specifically targeted by this campaign it could be the case that it's simply due to the location of the attackers and familiarity with the csps in their region i suspect that since many of these csvs are newer than for example aws google cloud and azure the security controls that are in place are perhaps not as mature as other cloud service providers making it more likely that instances and resources will be deployed in an insecure fashion if anyone has done research on this specifically then please get in touch with me as i'd like to know more about it

so if you're someone that follows cloud security research you're probably bored to death at this point by a family of crypto jacking shell scripts but with coin stump we noticed some interesting techniques which hinted the attacker's awareness of cloud security measures and the incident response process so these included the use of timestamp manipulation otherwise known as time stamping an attempt to remove system cryptographic policies in order to weaken the target system c2 communication via a reverse shell and a reference to a prior crypto jacking campaign potentially in an attempt to foil attribution so now with the overview of the way let's have a deeper look at the malware itself and the payloads that are utilized

so the first thing that caught our eye when analyzing a coin stump payload was the presence of this date time string which you can see passed in as a parameter to the touch commands dash t option hopefully you all actually can see that okay but i've highlighted the relevant section in the screenshot so the function on screen begins with an existence check for user bin mod user if this isn't found the script then greps for a sequence of strings found in the jamaa's binary in a sub shell and uses greps dash l option to return the file name only this can be seen on line 16 here it then renames the chamod binary to mod

user and runs the touch command with dash t and a date time string of 22 23 on the 20th of may 2019. in hindsight that's a lot of consecutive 20s for a live talk um so this may be common knowledge to most people in the room but after consulting the touch commands man page we realize that this is of course a pretty neat way to perform time stomping with a command that's virtually ubiquitous across unix-like systems on line 21 we see the exact same technique employed for the chatter or change attributes command except this time with a slightly different date time strength so why are the threat actors in this case obfuscating usage of chatter and

chamod in the first place of course both of these commands are specific to file access control most cloud native malware campaigns assume that certain files of interest will be restricted for editing either via file attributes or permissions so using the mod and chatter commands to modify these permissions or attributes is of course the most obvious way to overcome this this is why we see this in virtually all malicious shell scripts that we analyze at kedl so clearly this is a creative example of living off the land to obfuscate system utilities typically leveraged in cloud malware campaigns but how successful actually is it well out of interest and since my employer happens to develop an incident response platform

we decided to run these commands in a test machine and import an image of it into kedo to see how this would look to an incident responder so as you can see on the highlighted portion of the slide kedo identified a disparity between the timestamps of both the mod user and chair user files it seems like the touch command updated both the modified and access timestamps to the date time hard coded in the shell script however most importantly for us as defenders or response responders the created time was still correct it seemed likely that this was an attempt to obfuscate usage of the chatter into mod command line tools as some forensic stools may prioritize

files recently accessed or executed when building a timeline of events using a copy of these system binaries also avoids alerting admins who may have set up monitoring for execution of these utilities so you may be thinking that using touch for time stumping is not particularly novel or technically advanced however this following technique we've yet to see in other campaigns so this seemingly innocuous one-liner that we can see on the slide here residing at line 23 of the same script we saw the time stomping behavior actually is some pretty interesting stuff going on first of all rm is used to remove all files or sub directories within user share which have the string crypto in the name

so this of course sounded quite interesting to us as malware analysts so we had a look at what might be stored under such a directory in various linux distributions this led us to discover that in rail and rail light distributions it's possible to define a system-wide cryptographic policy which is stored in config files under user share crypto hyphen policies these policies allow or deny application level usage of certain cryptographic protocols depending on the user's risk posture so for example american federal institutions are required to deploy computing systems which conform to fips 140-2 and there's a fip specific policy bundled with rail to help enforce this clearly this is an attempt by the malware developer to weaken the system by

removing such policies in order to enforce the cryptographic policies in the first place a process named crypto is used which interfaces with the linux kernel crypto api to ensure that the policies are indeed removed from the system and no longer enforced the malware then goes ahead and kills the crypto process after removing the config files so this is something that wasn't really picked up on when we first published about this threat but i think that it's particularly interesting so if anyone else has seen this then please let me know as i'd be interested to discuss it so this was another interesting and very linux technique employed by coin slump for the purposes of evading detection

so as i'm sure you'll know most linux distributions support read write operations to a remote host via the dev tcp device file naturally this is an easy and natively supported way of creating a reverse shell or c2 communications channel so as we can see on screen here the function curl is used to retrieve payloads and communicate information about the state of the system back to a c2 server line 4 establishes communication with this server over port 443 the port typically associated with https traffic we looked the ipe of the remote server up in shodan and saw that the server was running python's simple http server module so this suggests that although the traffic was going over port 443 the

traffic itself wasn't actually encrypted so clearly this wouldn't fill anyone with robust traffic monitoring in place but we suspect that it was an attempt to ensure that c2 communications passed freely as it's unlikely that 443 outbound would be blocked by firewalls in the target environment so we observed this function being invoked on a regular basis throughout the coin stump payloads usually invocation would occur after file existence checks used to determine whether it was necessary to retrieve additional payloads this makes this a stealthy way for the attackers to register additional implants so this was an interesting and unexpected finding when analyzing some more of the constant payloads coin stamp made use of cron as a persistence mechanism and registered a

cron job under the root user however rather than using this persistence to launch or relaunch malicious payloads as most malwares coinstomp instead used the cron job to kill the tail and mass scan utilities the latter of which is often used in these types of campaigns to find vulnerable servers to target so we noticed an interested commented line in the cron job which you may be able to make out on line 243 there at one point it seems as if the code hosting service and on pasta was used to host an additional payload for the coin stomp campaign we can see on line 243 that the url for this provider is still added to the cron job but it's

commented out resulting in it having no effect on the job itself when we visited the url we found another url pointing to the anon dns anonymous dns provider this url contained a number of strings that we recognized from a prior campaign the first of which was xanthe a crypto mining campaign discovered by cisco talos that we'll come back to later in the talk

furthermore one of the payloads in this anti-campaign that we'd analyzed was called fcy0 same as the resource in the url here unfortunately the url was down when we attempted to retrieve this payload so we couldn't determine whether it was the exact same installation script as we'd seen in this anti campaign we also didn't notice any overlap of infrastructure between these campaigns or anything else that would suggest they were linked so this led us to conclude that the url likely contained those names in an attempt to foil attribution so finally to round off our overview of coin stump we noticed this rather amusing spelling mistake when stack when statically analyzing the custom version of xm rig that the coin

stomp scripts drop so we're not sure if this is deliberate or not but it was jokingly suggested that it could be a reference to british crime actor jason statham who may well have had some involvement in this campaign so i'm not sure if everyone here will know him or not but i've included a photo of him on the next slide for visual reference

so now we've covered our first malware family i'm going to move on to another cloud native campaign named abc bot that we've been tracking since late 2021. to give a quick overview of this family similar to coin stump abc bot is a botnet which is spread via initialization shell scripts and targets asian csbs such as 10 cent baidu and alibaba cloud so the malware includes payloads consisting of shell scripts and elf executables with the shell scripts in particular displaying some notable capabilities these capabilities include insertion of attacker-controlled ssh keys to main maintain access to the target system self-propagation in a worm-like fashion using information about cloud security services and competing malware campaigns to disable competitors

and registration of persistence via common linux persistence techniques so the campaign was originally reported on in november 2021 by netlab 360. netlab 360 focused their analysis on the elf payloads used to connect the infected machine to the botnet i've included a reference to their research at the end of the talk if you'd like to know more about this so for this reason we won't cover the botnet related payloads today instead we'll cover one of the installation shell scripts used to propagate the malware and download additional payloads we believe this script reveals more about the attacker's capabilities and objectives since the botnet payload was based on open source code in fitting with the theme of this talk

the attacker's knowledge of the cloud environments which their campaign targets was also evident in this shell script so let's begin by taking a look at an interesting capability displayed by the abc bot malware family the killing of competitors although fortunately not in a literal sense so one thing that is immediately clear from analyzing this initialization script is that the developers behind abc bot are really invested in killing off competing minors and crypto jacking campaigns the function that you can see on screen here which is several hundred lines long is dedicated to removing artifacts of competing malware campaigns and mining software such as exam rig so we also observed the malware searching for processes associated with

other prominent linux malware campaigns so for example things like watchdog and kinsing this suggests that those behind abc bot actively maintain a working knowledge of the cloud security threat landscape in a similar vein the malware also searches for malicious docker images and removes or can or kills the containers as appropriate so this strongly suggests the abc bot relies on exploitation of misconfigured docker api endpoints for propagation which is of course a common infection vector in cloud environments and used utilized by many cloud native campaigns so clearly the abc bot developers had invested significant time into researching cloud security threats given the previous slide however not only that but the developers also demonstrated a knowledge of cloud

security mechanisms as disabling of security services native to the csps targeted was performed so this of course allowed their malware to execute unimpeded and also allows us as analysts to determine the targets of the campaign for example several lines were dedicated to killing processes associated with the alibaba and 10 cent cloud security agents as we can see on screen here similarly the uninstallation scripts often baked into instances hosted by the csps were used to completely uninstall monitoring software in some instances the ease of which these monitoring tools can be removed could well be another reason as to why these csps in particular were targeted so we'll move on now to look at some methods of maintaining access employed

by abc bot a key objective of most malware campaigns is to establish network connectivity to allow bi-directional communication with the attacker this is of course known as command and control we saw this in our coverage of coin stomp and its use of a dev tcp reverse shell in a function named ip iptableschecker the developer behind abcbot configures the linux iptables firewall to drop or accept traffic based on port numbers and source ip addresses this particular function gave us some insights into the state of this campaign at the time of analysis so for example it's clear from the function that the malware is under active development as the author has left plenty of commented codes in one of the commented rules

it appears as if the developer configured iptables to accept all english traffic from a remote ip this was likely a c2 server under the attacker's control so another commented rule drops ingress traffic from ports 2375 and 2376. of course as we all know these ports are typically associated with the docker engine api we suspect that this was added at one point to prevent halt attempts to haul execution of any malicious docker containers the malware creates a check is also done to see whether these rules are already in place but if they aren't they're no longer added instead a more generic rule is added to allow all ingress traffic to a non-standard port number of twenty six

eight hundred interestingly urls embedded in the malware also made use of their support so another notable technique of abc bot was the ability to infect related hosts with a copy of itself firstly the malware removes ssh keys found on the host which appear to be from similar attacks it then goes ahead and inserts its own ssh key to guarantee ongoing access to the host after this as we can see on the slide here the malware checks for the existence of roots ssh known host file and a corresponding public key if these files are found in root's ssh directory non-hosts are enumerated in a loop and a copy of the installation script is run on each of the remote hosts found

this ensures propagation of the malware in a worm-like fashion and could result in an organization's entire cloud estate being rapidly compromised so now that we've covered some of the notable capabilities of abc bot let's discuss an unexpected finding that emerged during analysis of the campaign when analyzing abc bot we were initially under the impression that we were analyzing a relatively new malware family continued analysis revealed that this malware had a longer history than we initially thought so back in late 2020 cisco's tallow security research team reported on an emerging cloud cryptojacking malware campaign the name named xanthe xanthe was originally discovered after an intrusion was found on one of talos docker honeypots so we discovered a link between abc bot

and xanthe when conducting analysis on the infrastructure behind the abc bot campaign once we began comparing samples from both campaigns similarities and features and capabilities began to emerge additional comparison of the code used in samples in both campaigns further confirmed their suspicions so before discussing the similarities between these families let's have an overview of xanthe itself xanthe is a family of cryptojacking malware with the primary objective of hijacking system resources to mine the monero cryptocurrency in order to mine monero on target systems the common open source miner xm rig is deployed so similar to abc bot xanthe also spreads via exposed docker api endpoints with an initialization shell script responsible for propagation network scanning and downloading of additional payloads

xanthe's additional payloads included an open source library for hiding processes a script to disable security services and kill competing miners and the xm rig binary itself so if you're paying attention to the previous section of this talk then this will probably sound familiar to you so let's take a look at some of the signs that demonstrated these campaigns were linked in their report published in late 2020 talos researchers commented on the coding style present in those antiscripts they analyzed they highlighted that in the samples analyzed function declarations were located at the top of the script and function invocation was conducted at the bottom talos suggested that this likely aided testing of new iterations with function

calls commented or uncommented as necessary so although this is of course a fairly tenuous link it's interesting to note that samples from the abc bot and xanthe campaigns both followed this convention so diving deeper into the samples themselves we see several of the functions in xanthe have an identical name to those in abc but some of the functions also have the string go appended to the end of their names and this is another convention that we observed in both campaigns so we identified five functions with identical naming that you can see on the slide here subsequent analysis of each of the above functions was performed and they were determined to be semantically equivalent so these functions were mainly

responsible for adding public dns servers to resolve.conf to ensure outbound dns requests could be made registering persistence via cron and rc scripts creating and modifying iptable's rules as we saw earlier in our analysis of abc bot and downloading of additional payloads such as those used to connect the machine to the botnet in abc bot's case and the downloading of xm rig in xanthe's case so we mentioned earlier the propagation via enumeration of known hosts was a notable capability of abc bot this exact same technique was used in the samples of xanthe we analyzed albeit with a slightly different implementation examples of the codes responsible from both campaigns can be seen on the slide similarly in abc bot a number of

malicious users were added to the system to facilitate a backdoor the user is added to the system were identical in samples from both campaigns and included usernames such as logger sys all system and auto updater both abc bot and xanthe searched for and removed hard-coded users when analyzing abc bot we originally believed that the user names being searched for were from competing campaigns however we now believe that at least one of the usernames searched for by abc bot was from a historical campaign from the street actor the username in question was opsec underscore x12 and both abc bot and xanthe included code to remove this user when we first analyzed samples from xanthe we realized that this username was being

displayed as ascii art at the top of one of the payloads so while this could of course be an attempt by one threat actor to copy another we believe that our prior findings indicated that this was more than coincidental so now on to our final and most interesting finding although each of the similarities we've discussed were enough to give us reasonable suspicion that these campaigns were linked we still had some doubts so code reuse is of course common amongst malware developers with payloads such as shell scripts where everything is in plain text copying is even more likely so in light of this we needed one final piece of evidence to conclusively link the campaigns we already discussed a

function from abc bot named iptables checker which was responsible for configuring ip tables to allow ingress traffic from a non-standard port an incredibly similar version of this function was also found in the xanthe sample we analyzed not only that but rules used within this function to allow traffic from the c2 server included the exact same ip address in both sante and an abc bot so the lines from payloads in both campaigns that demonstrate this are viewable on screen so to us this constituted an overlap of infrastructure which was fairly strong proof that the campaigns were linked the server at the hard-coded ip address would have to be under control of the the developer behind both abc bot

and xanthe for it to be usefully included in the script of course there'd be little reason for the developer to include this if it wasn't a part of their own infrastructure so we believe that this is the strongest indicator yet that these campaigns are linked and at the same time actor is responsible so in summary the abc bot and xanthe campaigns demonstrated the sophistication of malware developers in the cloud security space low code reuse is common in malware particularly malware involving shell scripts we've highlighted an overlapping infrastructure and identified reuse of unique strings which would be difficult and or pointless for someone to copy if the same threat actor is behind these campaigns we believe that this indicates a shift

away from cryptocurrency mining which is of course a common objective of cloud malware and the main objective of xanthe on to potentially more destructive botnet activities as we highlighted with abc bot so this should give you some idea of the destructive potential of cloud threat actors if they decide to broaden their horizons from crypto jacking that is so i know that the title of this talk is a tale of two malware families but since the two families discussed in the talk are relatively old now i wanted to include an example of something recent which fits with the theme of the talk i also thought it would be a good idea to give some tips to defenders who may

be concerned about preventing or detecting this type of threat in their environment so this snippet is from a recent campaign from the threat actor watchdog who targeted our harry potter infrastructure it utilizes a similar time stamping technique as we saw in the overview of coin stump but and perhaps more interestingly we can see the attackers implementing a very rudimentary albeit effective process either first the binary for ps is copied to another file named ps lanigiro a very simple shell script is then written to bin pdfs the sole purpose of which is to call the renamed ps binary and pipe the output through an inverse grip to filter out processes with the names with the strings ddns

and scan in the name so as you might expect from this ddns and scan are two malicious processes run by the malware amazingly this actually works and it's perhaps the most unixy process hider i've ever seen in my life so more importantly this demonstrates that you don't need fancy root kits to have effective detection of agent so this was another simple but very effective but effective technique sorry by employed by watchdog for hiding artifacts on the target system so when analyzing their payloads we saw multiple references to paths containing directories that were named with three full stops or an ellipsis so it turns out that this name is perfectly valid for files and directories on linux systems

and has the added benefit of looking similar to the two-dot alias for the parent directory and long listings so as you can see on the screen here the ellipsis directory is hidden and could easily be mistaken for the parent directory by an unsuspecting admin obviously this wouldn't fill proper edr solutions but include compute instances or containers where you may be manually investigating a breach this is the kind of thing that could be easily overlooked so we briefly touched on this technique when we discussed coin stomp but it's one that seems to be a favorite amongst cloud threat actors in this screenshot we can see existence checks for a file named cd1 this is actually a version of curl

that's been renamed in order to obfuscate its usage the malware then sets an envar with the path to the renamed version of karo so that any attempts to retrieve additional payloads make use of cd1 and not the carbiner itself it's difficult to see just how effective this would be but i suppose if you're monitoring for invocations of coral then it's a way for the attacker to avoid generating an alert this technique has also been observed when obfuscating usage of other data transfer utilities so for example wget not only that but we've observed it being utilized in campaigns from watchdog rock group team tnt and so on so moving on now to some tips for anyone

fortunate enough to have to defend against these types of attacks so first things first make sure you have the basics covered remember cloud threat actors don't typically make use of advanced zero days or nation state level tooling although their attacks are increasing in sophistication they still have more in common with homer than they do with chris hemsworth and black cat so in terms of basics ensure that you've got proper auditing in place in your hosts or containers and you're not exposing services to the internet unnecessarily of course docker and redis are still some of the most common infection vectors in cloud attacks we actually got an idea of how common it is to exploit these services when we

deployed a docker honeypot that was first compromised only 12 minutes after we put it up so watch for new additions to user bin or bin directories both campaigns described today relied upon the renaming of binaries under these directories to evade detection if you monitor for rights to user bin or bin you'll easily be able to identify when this occurs and trace the usage of the renamed utilities so thirdly implement a bastion host most people probably know to do this already but cloud today actors rely heavily on ssh propagation so if you have a known host file on your host with ips of all the other hosts in your organization this ensures rapid compromise of your

cloud estate if you're targeted by some of these campaigns so ensure adequate csp logging is in place this is of course an obvious one but it comes up all the time cloud malware campaigns are successful partly due to misconfigurations in cloud environments but also largely due to the lack of visibility the defenders have and to resources in their estate a comprehensive csb logging strategy can help with this so finally monitor for execution of the touch binary using touch to modify timestamps on a file isn't of course an everyday occurrence if you see execution of touch particularly when combined with the dash t or d options then it's a good indication that something strange is going on

i'd like to wrap up now with a note that incident response in the cloud is hard and hopefully this has given you some knowledge of techniques currently in use by cloud native threat actors if you're interested in this type of thing here are a list of references that we use when conducting this research so as you can see here there are blogs from netlab 360 talos unit 42 etc these organizations are all well worth following if you're a fan of cloud security research or you work in this area i've also included the blogs that we published which detail this research and you can find many more on the blog section of our website at kdosecurity.com blogs

so i hope that you all enjoyed the talk and of some idea of the techniques currently used by cloud threat actors as i mentioned in the beginning i'll post the slides afterwards if b-sides don't do it first then please feel free to reach out to me if you want to discuss any of this further so i can be reached at muir cadosecurity.com or mapmuir on twitter so i think we've got some time for a q a session now so if you have any questions about cloud security or the content in the talk then please let me know otherwise i look forward to seeing you all around vegas for the rest of black cat and defcon

[Applause]

uh so they won't be on the blog but i can post them publicly i'll post them on twitter after after the talk most of the screenshots that were used in the slides come from the blogs so if you're interested in that then check those out yes [Music] yeah so you're not actually the first person to ask me this um we didn't actually do that but it's something that we probably will look into in future does that answer your question

yeah yeah i mean my background's in malware analysis so my focus really is on the payloads itself quite often but yeah i would agree that is definitely a key part of of the puzzle

any more questions at all

cool thank you [Applause]

[Music]

so [Music]

[Music]

do [Music]

[Music]

do [Music]

[Music]

[Music] do [Music]

[Music]

so [Music]

[Music]

[Music] do [Music]

my [Music]

[Music] you know

[Music]

[Music] [Music]

[Music]

okay hello everyone welcome to uh besides las vegas uh back in person um i just say a few words uh first of all i want to say uh thanks to our sponsors uh diamond sponsor lastpass and palo alto networks as well as our gold sponsors i just named three here amazon envision and blue cat a little bit of housekeeping please silence your cell phones the talk is being recorded it will be available on youtube we also streaming live so you know make it make it nice here if we have questions at the end make sure that you come closer to the podium we don't have a microphone in the room so the presenter can hear you

repeat the question for the stream and answer it um picture policy uh you should know by now besides a strict policy for taking pictures make sure that you have explicit content from everybody in the frame should you take pictures uh keep your mask at all the time before i introduce brad here vladimir durkheim how many people can speak french okay so we'll do it in english but it's just because it's you so valemia is going to talk about a certain form of pollution from what i understand so hey you have the flow mercy dimitri thanks for coming to my talk uh i hope you will enjoy it there will be a lot of javascript inside but don't worry

we will cover everything uh feel free to sign very widely if i'm not speaking properly in the microphone i'm vladimir you everybody calls me vlad i'm a staff engineer at datadog i'm also a node.js co-collaborator i've been working on application security for the last six years in a startup that got acquired by datadog last year so we are building application security products feel free to reach out also feel free to follow me on twitter at paul defeat today i will take a chronological approach on that talk so there will be a lot of context stuff technical stuff a bit of uh self-soul-searching uh it will be kind of a mix of a lot of things don't worry

the technical part is still the biggest part um and a few disclaimers there's a lot of storytelling that's a very personal presentation it will cover a lot of things i've done as an open source contributor not as a data dog no screen employee and there will be a lot of back and forth about what i had in mind back then and you will understand why i say that now also last disclaimer i've built some commercial content online trainings about javascript security including prototype pollution and i found it was fair to put the items the disclaimers okay so it all started uh with the bugbrunty program so how many of you are familiar with npm as

a package ecosystem okay a fair amount of people so just quick reminders it's the most popular package ecosystem in the world and you can take this graph at any time in the past few years it will look the same npm as the javascript package hosting repository is the most growing and the most popular package repository in the world and that's been so for almost a decade now there are millions of packages on npm and npm serves a huge variety of content of course they're javascript backend packages node modules but they're also javascript fountains modules like people get react bootstrap from npm so front-end libraries are also downloaded from npm you have web assembly code that is

distributed over npm because it's used on the web so wait no why not and you have native executable those are the native add-ons used with node.js so if you want to for instance use sqlite you're likely to use the sqlite3 native add-on on node.js and that might be pre-compiled or not for instance data.org as a couple pre-compiled native executable on npm to patch node.js directly or even code in cc plus plus rest that can be compiled at install time for node.js so why do i say that well in 2016 the question of vulnerable slash madison's package was a bit undefined npm did not really npm as a commercial entity did not want to do any strong effort in that

direction they had a lot of other things to do and that was fair from their point of view um sneak existed but was still fair new and that's that's a commercial tool and the node security project which was basically an alternative to sneak was deprecating because they got acquired by npm to do something totally different than vulnerable package management so the node security project came to the node foundation and say hey we've got this database of vulnerable and malicious packages do you want it do you want to be in charge of npm security as the node.js foundation and we said yes we were probably young and crazy um so we created a node.js ecosystem bugbunty program and hacker1 was

extremely nice they offered us a free plan for the ecosystem and we also have a free plan for node core so if you find minorities in node.js feel free to report them on hacker1 i will be happy to review them and we started the backbone team without rewards and depending on the case sometimes the owners of the vulnerable packages would give rewards so for instance the versailles company had vulnerabilities in a package themselves and they rewarded the hackers with five hundred dollars from time to time which was very generous and that was a lot of work handling this burgmanty program especially when i was by my own for the first six months because we had hundreds and of reports

and usually reports there are trends when people find a vulnerability in the in such ecosystem people will do everything they can to find the same vulnerabilities in similar packages and that's actually important for for the rest of the talk and then in 2018 january we get a super report because it impacts 12 packages on the apm ecosystem comes from a security researcher from canada named olivier and there are 12 impacted modules but the weird part is it talks about prototype pollution and we had no idea what prototype pollution was back then that was the first time we heard about it all together and we did not find any literature so the summary was utilities function in

the listed modules can be tricked into modifying the prototype of object when the attackers control part of the structure passed to this function this can let blah blah blah blah blah blah blah how how can you override methods in the javascript standard library and that's where we start to go technical and we forget about history for for a bit so in javascript you've got basic tips types you've got booleans objects numbers big ends strings symbols and undefined and pretty much everything else is objects so dicts are objects strange objects are objects regular expressions are objects null is a specific object and objects are objects and map and sets all this higher level data structures there are objects too

and objects have method so for instance if i call underdict foo one has on property foo it will return true and hazon property is actually a property available it's a method available on that object uh if i check has on property on has own property it's false and hazard property tells you if this object owns this property or if it's come from somewhere else so now we know that we can call a method named has on property but it does not exist on the object itself so how is it available where does it come from well let's use a debugger because the debugger is probably the most marvelous piece of software in the history of

software yeah i'm terrible when i'm asked to mentor juniors i force them to learn the debugger rather than putting prints everywhere i never force anyone to do any anybody ah anybody i'm doing anything don't worry so if we debug it we realize that there's actually a property named prototype and that has on property is actually attached to this prototype so how does that work in javascript objects can't can have a prototype and when a method or property basically methods are properties in javascript is not found on an object we look up on its prototype so if you want to access a property on an object and if not available on the object directly we take the object's

prototype and we check all that and the prototype is actually an object itself but this prototype might have a prototype so if the prototype does not have the property you're looking for well by recursion you look on it on the prototype of the prototype or on the prototype of the prototype of the prototype until there's no prototype available and that's why we talk a lot about prototype chain in javascript so let's talk about prototype chain on the left hand side we define functions constructors and on the right hand side we use them so let's create an object that's called my proto that will be a prototype for certain objects and it has a function a property

named foo that is a function then we define a class with the old style in javascript where you define just the constructor and you attach a prototype property on that constructor and here we put our own prototype with bar and hello world as properties and then we defined a new style class that extends the old style class and as bars as a property so if we create an item an object from this class here well it has the property hello available because it's available on the prototype of old class old styles class that is actually implemented in our new class so that's just basic inheritance in any object-oriented language but with prototypes and if we call baz it's available

because it's on the prototype here and if we call bar it's available because it's on the prototype here same thing if we call item two it will share the same prototypes and if we create item three and we call the method set prototype off because why why not we can this method exists and we use myproto well now the prototype for item 3 will be myproto and we have selected an arbitrary prototype and what's interesting is that prototypes are objects and they are not defined multiple times in the hip there's only one instance of the prototype as an object in the hip so for instance item 2 and item 1 you know they are created with new

on the new style class so their prototype is the same object we can check equality object equality on their prototype value but this prototype has another prototype that is the one from the old style class because inheritance because the new class is actually extending the old style class and all of these prototypes they actually have another higher level prototype that is the object.prototype the default prototype for objects in a javascript heap and on the other side item three it's prototype it's my proto and the prototype of its prototype is object.prototype okay so the gotchas are objective prototype properties come from the prototype or the object when they are not found on the object we look upon them on the

prototype and prototypes are the same for two different objects multiple objects can have the same prototype that are exactly the same object in heap and we can access the prototype from the object directly that's where javascript is a bit special so here i've got a class and i create two instances of this class one with b sides and the other of the gas and if we call my item but that has on property show prop it will tell us false also it's worth not noticing that has on property is actually available on the object prototype it's coming from the higher highest prototype in the prototype chain okay so we can call object dot get prototype off and that will return the

prototype so in our case show prop is accessible on the prototype of my item i don't think anyone has said that much uh prototyping a talk in few years then you can also access the prototype by calling underscore underscore proto underscore underscore that's a kind of outdated way to do it it's deprecated officially but we can't really remove it from the web platform whether that's the front end or the back end because it will break everything so there are options to disable it we'll see that later but you can access the prototype through the underscore underscore proto underscore underscore property or each object in javascript has a reference to its constructor and remember the prototype is just a property on the

constructor so if you access the constructor of an object then the prototype value on the constructor you have access to the prototype tool and just to re-explain that prototypes are single instance of objects if i take my item and my item 2 and i compare their prototypes they are exactly the same objects okay i think i'm good with explaining your prototypes work in javascript hopefully it was not too confusing so what's a pollution now that we know what a prototype is let's pollute them a prototype pollution happens when an arbitrary payload handled by the javascript codebase can overwrite properties or methods on the prototype chain and that's pretty much it and that happens usually when a

merge function is called we'll see why now so here i'm using the hook library in an outdated version it was part of the original report by olivier on prototype pollution so i create a malicious payload with a property name underscore underscore proto underscore underscore remember that one of the way of accessing prototypes from an object and then i place a property named oops on it with it works i create an object named a and i don't mutate this object at all anymore because on the next line i say before and i check if a has a property name oops either on a is on the prototype chain and it will say undefined because there is no oops

in the prototype chain or on a then i call hook dot merge on a totally brand new object nothing to do with a and json.pass my malicious payload i will explain why we use json and pass right after that and then i recheck if a has a oops property and actually tells us it works because the call to the merge function actually mutated something in the prototype chain because when the merge function was called it checked if it could merge the proto property of this object and it did so it accessed the proto property the prototype of our brand new object and added a property name oops on this prototype but this prototype is object.prototype the global prototype

for all objects so basically all our objects in this javascript group hip now will return it works when you call oops except if they have a own property name oops why do we use json.pass here because if you define the object directly well it will already have a proto property and it will be kinda ignored by the engine but there's a gotcha in the historical implementation of json.pass that makes this proto valid only if it comes outside of json.pass i won't go in the depth of the implementation of that but the gotcha is that for this exploit to work it must go through json.pass but so why then as i told you let's check the merge

method and the merge method is actually recursive it calls merge again on itself because you want to merge objects recursively because you want to merge nested and sub-objects of your object and that's where the prototype pollution happens because once again you will check if the object that is the target right now the first one here has underscore underscore proto underscore underscore as a property this will return true so you will merge property on the prototype because require seamless i hope that's not too confused okay other example in that case we don't need the json.pass version because remember all objects in javascript has access to their constructor so their own prototype through the constructor so there's another exploit uh there used to

be another exploit in lodash well you would do basically the same so here we've got constructor prototype is that mean true we create a b object before we check b dot is that mean it returns undefined we merge the payload on a totally different object and when we call b that is that mean it will return true because we polluted the prototype chain again why does it matter well lowdash is actually one of the most popular javascript package on earth it's been downloaded more than 200 million times in april that the downloads per month and it's been downloading 8 billion times since the package was created it's pretty much in most javascript code bases and it has been having multiple

prototype pollutions meaning that the world javascript ecosystem was at risk with them okay let's take a break from the javascript and saying prototype prototype and prototype of prototype and recursion and recursion and request and go back to the report to to go a bit slower so we started to discuss on hacker one and the report is still private for a lot of reason even four years later uh but he's the gist of the discussion first thing we did we acknowledge having the report and we just gathered as the bug boom team manager team and be like okay what do we do with that now is that really a security issue or is that a quality issue is that a widespread

defect in the javascript ecosystem or is it something that actually can be used to impact applications on their security posture and discussions went on and on and at the same time i started to reach out to nathan la forest nathan la forest here with the main maintainer of hook the first library i showed today and i told him hey nothing i i don't understand how it works under the hood what i explained in the previous slide i did not understand back then just reading a report and it was hard to understand so we we sat together over the internet and started to put debug points and try to understand how how how that worked and that's how we

understood prototype pollution and it blew our mind and we know that there is a systemic bug in a lot of javascript code base based on that uh we know that everything that's merged or deep clone with recursion might be vulnerable to that as long as it takes arbitrary payload in input is it a security issue that that's a one billion dollar question maybe i'm a bit exaggerating but not that much so we challenged a bit and the reporter actually provided an exploit in an existing code base node.js code base some kind of cms slash blog management i won't name it because it's the reason why the report is not public yet because it's been fixed in it

but never published properly and they actually highlighted a crash in the application so node.js applications are writing in javascript and javascript is by definition single threaded like worker threads are an exception to that but no js applications they are thing they are usually single threaded and all http requests are handled by the same thread at the same time with a lot of asynchronicity so if you crash one thread in a node.js application well you crash the whole process because there's only one thread that's why error management is really critical and if you manage to find a crusher you're not crashing one request you're crushing the web server itself so they found a crash by replacing a method by a

string which is fair and the code base was calling this method and that was crashing because it was a string at this point um so we found the denial of service because that's that's the definition of a denial of service you crashed to your application application server and we also found cwe 471 a modification of assumed immutable data that was a good candidate back then with our knowledge so based on that we had okay we have an exploit at least one real life exploit that has an acceptable outcome has been acceptable a malicious outcome and we found some kind of reference in a referential we found this cwe saying hey this has been acknowledged as

a group of vulnerabilities by the industry so at this point we are mostly thinking yeah that's probably your proper vulnerability let's accept it as a vulnerability report so a bit of housekeeping we asked olivier to publish 12 reports instead of one single one because we had to communicate with library maintainers for them to fix and we didn't want them to know the list of other vulnerable packages we reached out to the maintainers shared the reports and we started publishing fix and publishing reports and we had like a few cwc see the cve going out in 2018 and what happens after 2018. well that's a good question yeah since 2018 there have been 193 cv published for prototype pollution

2022 is not over um and not all projects attribute cves for instance if i'm not mistaken sneak does not apply publish cvs on a lot of packages when they're not popular enough and the hacker one project for node ecosystem for npm ecosystem has been closed meaning there might be hundreds of other occurrences of prototype pollutions in the wild i kinda remember a stat why it was one of the most the fastest publishing cv machines in term of attack classes there have been one sce in kibana we'll go on that later and in december it was a repo there was a report about one cve on past server we'll get to that a bit later too

and more recently last month the royal institute of technology in sweden published yeah published a really cool paper uh on the topic the reference will be at the end of the of the talk it's a good read so i told you let's talk about an lce because we found dos but what about sce over prototype pollution okay so as i told you node.js is single threaded so if you want to do tasks that are cpu intensive you will be blo you will be monopolizing the single thread and node.js has been designed for ios for asynchronicity so if you block the main thread you block the wall up and you kinda does it temporarily so kibana use a hack when it has to do

some computing it start a child process which is very fair until we had worker threads in node.js two years ago that was the only way to go and node.js shares the environment of the parent process to the children one and this is defined as a javascript object i think you kinda see where this is going right and node.js has an option that's called that's an environment variable called node options and basically that enables you to pass command lines argument to node.js but through an environment variable so instead of doing node.js dash something than your executable you can do node option dash dash your flags then node your executable which is something handy well what happened in kibana

is that someone found a prototype pollution to actually write the no adoption property on the prototype chain so when the child process was starting it was inheriting the environment from the parents including the node option newly defined and their environment viable on the prototype chain and they would be able to call dash e for eval and run arbitrary javascript code from the command line through a prototype pollution and since you're running arbitrary node.js code you can call another child process with a shell in it you can call the fight system you've got an sce directly on the server so that was pretty bad that's been fixed and i know that the kibana team has been doing a tremendous

job on that the past vulnerability is more recent uh the report is from december but it has been published only this year if i'm not mistaken past was a standalone startup that got acquired by one of the major companies of the valley i'll let you google where that is but it's not google and they deprecated the project and the current version of pass is the open source community version of that and that's basically a rest api in front of mongodb it's used to build backends for mobile applications and basically it's it's an api it's a web api over mongodb so you can write objects you can read objects all of that and it's venerable to prototype

pollution because it's called the merge function at some point that's that the exploit but also it uses a library named b bison.js which is the default library used by the mongodb driver in node.js so the attacker can upload the document on mongodb through pass that's basically the point of this piece of software you can upload upload and retrieve documents in mongodb so good and those documents are formatted in bison and bison actually allows you to register functions that will be executed when you deserialize code from mongodb but this is by default disabled you need to enable the eval functions property on the bison library to make the code executable but there's a prototype pollution so you

can pollute eval function for all objects in the heap and when bison is checking oh is eval function to set to true yes it will be because of the pollution and because eval function is not default defined on the object in bason library because if the bayesian library had a real default version not a per default default property meaning that if the option object was updated with is this property existing on the object yes no if no its value is false then it will not be vulnerable but what they do is just does this property exist on the object or its prototype chain because it's easy to do and because of the prototype pollution you could do

that so basically the outcome run arbitrary javascript code on the node.js server once again you can call any node.js module fork star spawn a shell do whatever you want that's that's a se and that's a terrible slide ordering for me i'm so sorry let's go back to the cube and i want for one sec how we think that node does not allow the dash e in node options anymore so you can't evolve keyboard as fixed you can check the report sorry about that how to prevent well remember that's our merge function in hook here is the fix basically we just needed to prevent the property proto to be taken in account and that was good enough because at the only

vulnerable path to hook herc was not vulnerable to the prototype dot constructor.prototype path so there are other ways when you call an object in javascript when you call a property it's a good practice to actually check if the property exists on the object directly that's why we have hazon property and you can make sure that the property is actually not inherited and there's an alternative you can actually create objects that won't have a prototype at all so null despite being an object does not have a prototype and you've got a method called object.create that takes a prototype as an argument and create a new object with the argument as a prototype so let's take our example again and here

i create a with object that creates null so we run the attack and a is actually not vulnerable to this attack because it just doesn't have a prototype so there's nothing to pollute but there's there's a drawback is that i can't call has on property or the usual methods of the objects in javascript because they're generated from the object prototype i think one of the main way to prevent prototype pollution and that's actually pretty straightforward is data validation and sanitization just make sure that the objects that are coming from the user that i by definition are interested don't contain properties you're not expecting so there's a very very very cool library named joy for that and you define the

shape of the object even the size of the integers on it regex to validate properties whatever you want it's a very good library to do proper type and validation on objects in javascript and by default it will prevent objects from having and unplanned properties so shape your object and if properties are not expected they will be marked as invalid which is pretty cool sidenote object decentralization is typeless in javascript it's mostly json.pass and that opens the door to a lot of other issues over object injections one of the most impacting one is no secret injections you you can check my previous stroke just google my name and mongodb and you'll see a few talks on the topic if

you're interested so that's the conclusion now what do i think now what do i think four years later almost 200 published vulnerabilities later did we open the pandora box or did we do the right thing well during the process of accepting the first prototype pollution report there was a lot of uncertainty i think we were good because we involved other people we trusted including community leaders as maintainers so we did not stay as a close mind team especially important because in that triage team under berg bunty i think i was the only one working full time no liranto there was only two of us working full time on application security and the other one were just

node.js developers with no professional background in security especially so we did i think we did the right thing by gathering uh group intelligence on that uh i think we were fair in expecting an example of exploit saying hey okay that's brand new that makes sense can you demonstrate that it can have a real life impact on an application that would help us a lot and try to link to as little as much literature we can even if it was just finding the right cwe um i think we missed a few opportunities under process 2 because we had a brand new vulnerability class we knew it was largely impacting javascript as an ecosystem and we did

not do what we should have done which is sit together with the node.js core team and say hey we need to compute the threat model and evaluate if there's something in node that could be uh abused over prototype pollution and maybe we would have found the kibana case one year before it was published uh by saying hey look you can redefine environment variables maybe and i think i was really bad on communication i let olivier gave the talks on that which is fair he was the one who found the vulnerabilities uh fun thing uh the next year at uh os california i was giving a talk about node.js security model and he was in the

next room giving a talk about prototype pollution and olivier moved to something else after because he's a security researcher and we probably should have uh run that bike as community leader and security leaders in the javascript ecosystem evangelize that's actually my first talk ever on prototype pollution and i think that's a missed opportunity on my side for the fun story couple years ago someone tried to tell me there's a prototype pollution in my code and i demonstrate that it was unexploitable but sometimes you know [Music] conclusion what now monitor incoming objects don't let proto or constructors come inside your hip from outside objects there's an option in node.js to disable proto altogether use at your own risk it

might break some of your libraries but a good a good thing with same grip will help you know if you're uh if your code base or their dependencies actually use underscore underscore proto underscore underscore i'm saying i'm mentioning some great not only because you're in the room and enforce sanitization and prototype less objects a few links the two last ones are actually my classes if you want to deep dive into that the one that's here pollute me it's actually a github repo where i have written a vulnerable code base you can download it start the docker repo and try to exploit it i did that in 10 minutes yesterday so maybe it's not very good feel free to provide feedback

reach out on twitter if you want the slides because i always forget to upload them between jet lag and busy conference time and the two first links are um report the first one uh i think it's the yeah it's the kth paper and the second one is actually uh the erc in kibana i think i have time for some questions now yes

so the question is did we do any research to see if other languages were very verbal to similar class of attacks um i i'm not aware of that and i was actually reading the page about prototype oriented programming this morning it was like oh there's at least a few other prototype oriented languages that's probably worth checking i think depending on the way inheritance is implemented in other languages it might make sense on the top of my head i would consider that language that trend on the jvm won't be vulnerable for that but that probably needs to be checked

something like yeah you can't modify the application but you don't have any access on the server side is there uh

so the question is is there a go-to payload a cheat sheet to find them and because of those i i don't know if there is a list yet i actually think this one that replace has on property by null and same thing on that one that should do the trick and just check the object prototype on ndn write a script and you can use one single payload to override all of these values so you will soon see that i think you've got a 70 chance of crashing the application if you manage to override them so yeah i i can try to upload something on github later yeah

why was the back bounty shutdown for node ecosystem not not core mostly because npm was acquired by github and that was the best move for the javascript ecosystem and no guitar also npm started to do bug bounty i mean vulnerability report as a feature in npm with npm edit then got acquired by github that has security features so it did not make sense to have like an extra an extra initiative on that especially since we were doing that on our free time and people like github are paid to do that so they will do it ten times better than we do

uh do we have uh regression testing on this vulnerable project um i i don't know but that's the magic about um about fashion in the bug booty world is that so when we started the ecosystem bugbunty the first report we get was a directory traversal in the application used to serve static content and if you check the outcome of this bugboonty i think the probably the first 50 reports were that then when proper time pollution was published we had probably 50 reports for prototype pollution so i would say that people who are getting trained to get karma and hacker 1 all the regression testing for that if i may

um if i rephrase does googling search the code base now

um i i i don't think we will online we've been using that at all i was actually discussing with two people in this room earlier today about the fact that this kind of attacks are really hard to find with static analysis if you read the kth paper one of the point is that they had a lot of false positives if i remember properly don't misquote me on that they had a lot of false positive and static static analysis so i don't have a i don't think scanning works yet for this class of attacks okay if there are no more questions feel free to follow me on twitter ping me provide feedback i'm looking for them

too and ask for the slides and thanks so much for being an amazing road [Applause]

[Music]

[Music] [Music]

[Music]

do [Music]

[Music]

[Music] do [Music]

do [Music]

[Music]

so [Music]

[Music]

[Music] so [Music]

[Music]

[Music] [Music]

[Music]

do [Music]

[Music]

[Music] do

[Music]

[Music] do [Music]

[Music]

do [Music]

[Music]

so [Music]

[Music]

so [Music] so [Music]

[Music]

so [Music]

[Music]

[Music] do [Music]

[Music]

[Music] [Music]

[Music]

[Music] do [Music] foreign

[Music] so [Music]

[Music]

foreign

[Music]

pepper [Music]

[Music]

[Music] so [Music]

[Music]

so [Music]

[Music]

[Music] [Music]

[Music]

[Music] do [Music] do [Music]

[Music]

[Music] so [Music]

[Music]

[Music] do

[Music]

[Music] [Music] do

[Music]

[Music] do

[Music]

[Music] so

[Music]

do [Music]

[Music]

[Music] do [Music]

[Music]

[Music] so [Music]

[Music]

testing check one two three all right hello good afternoon welcome to b-sides las vegas you are in breaking ground uh this talk is the north northern virginia shuffle lateral movement and other creative steps attackers take in aws cloud environments and how to detect them be given by philippe proteus just a few quick announcements real quick before we start we'd like to thank our sponsors especially our diamond sponsors lastpass and palo alto networks and our gold sponsors amazon invisium flex track it's their support along with our other sponsors donors and volunteers to make this event possible regarding cell phones these talks will be live streamed and as a courtesy to our speakers and audience we ask that you check to make

sure that your cell phones are set to silent thank you if you have a question use the audience microphone which is the one i'm using right now so that people on youtube can hear what your question was as a reminder the b-sides las vegas photo policy prohibits taking pictures without the explicit permission of everyone in frame these talks are all being recorded and will be available on youtube in the future and last but not least we'd like you to please keep your masks on at all times thank you very much with that let's get started please welcome felipe proteus

thank you everybody thank you for coming for my talk so my name is philippe as you you heard from pierce and my nickname is proteus so just my mom called philippe so you can comfort us wherever you find me whenever so a little bit about me i have at least 10 years of experience in security in general i'm a proud father of a girl i'm also a bluetooth instructor back in brazil and i'm a security researcher at entity security teaching security is a start-up company based uh focused in cloud security and third-party risk management and as i said as my role as a security researcher part of my job is googling around reading documentations and writing box

to test for their abilities and this kind of stuff and sometimes we help our clients uh with with questions so i i was there one day googling around reading docs and one of our clients came to us and asking about his network diagram on aws and he was doing all kinds of nasty stuff he was doing things like pissed appearing between dev and broad environment cross accounts between dev and fraud environments and i say no you're doing something really neat here you shouldn't doing this because there's the problem of ladder movement and he says what and yeah and then i explained him and then something popped in my mind i just realized that cloud security is more complex than

on-premise security and i know that's a bold statement but i do have a analogy or a metaphor to explain that so by the title of this talk when i submitted this talk to besides i was thinking well what things walk laterally and doesn't work a lot of league so i can think get a thing for my talk and the only thing that i could think was crabs but i don't like crabs i guess nobody like crabs so i was talking to my boss sierra and he told me about this kind of dancing it's called texas shuffle and for those who doesn't know which texas shuffle kind of dancing style is is something like this

seems nice right but i'm a brazilian and i'm from rio general actually so i have to bring something for you for you guys to do a contrast between the texas shuffle and the dancing that i brought here today is something that we call pacino it's like a brazilian thing so it's kind of like this

yeah two different dense styles with their respective differences but we'll get a little bit more on this later but just keep those dancing in mind so when we talk about ladder of movements and possibilities of lateral movements uh how many of you guys already did a pen test on on-prem's networks how long did you guys take to get us domie and me more than a day or more than two days usually you just pivot pretty fast between all the networks find the domain admin uh get the credentials and becomes a domain administrator right so this can also be done on aws but on aws you can have separate accounts so you can have account a and account to b

and for do these kind of things in aws must have to be a vpc peering between those accounts and those vpcs right so when an attacker usually attacker doesn't land when he went when they when when their objective is so he has to move laterally right so if he hackets a instance on the vpca he can find out on other networks and move laterally into the vpcb and so on this thing in aws we call data plane and data plane is where your data lives actually but on cloud we also have another plane that's called the control plane and the control plane is like it's all the apis that you call to make requests to your cloud provider for

example in aws you call it the control plane to create a new vpc to create ec2 instance or three buckets or something things like that and the the thing with the data plane to get in you just have to get something exposed like ip address or things like that but just getting on the control plane you're going to need a credential or login password or aws access keys and so on in the control plane you have entities like users roles groups and things like that so how do you movement laterally on the control plane so here you have a role and an in account a and this role can assume another role the role too

in the same account but the thing is the the attacker who has who got the role well he could also zac and create an instance on the vpca and then he can jump from the control plane into the data plane and if the attacker got a first hack ac2 instance and he find out that his ec2 instance has a role attached to it he can assume that role and jump to the control plane and from there he can also find out another networks another account in aws and jump from account a to account to b and he could also from the same ec2 instance if he find out another account he can jump from the institutions in

account a to another role you know in account b so you can think you can pretty much see that this this thing's getting even more complex than opram's network right and even though uh the same attacker who has a role in account in account to be he can also exact and execute to create an instance on vpc on account to be so it's really really mess right so just to bring that comparison between those dances and i promise to my sister-in-law do not make comparisons between dances because that's not something that you should do but we can compare features between those dances so who thought that the brazilian flank would be more hard to learn

and well who thought that the brazilian fans would be harder to learn than texas shuffle anyone yeah but the thing is the brazilian is only fast he was dancing by himself fast moves yeah but dancing by himself the texas shuffle uh he has to interact that has interaction between the partners so uh they also can dance by themselves if you watch the the video clip again they also dance separately between them but they also have to coordinate the weights and balance between them so there's much more aspects and complexity in texas shuffle than brazilian passing you so the texas shuffle should be our cloud environment the comparison between data planning control playing and the balancing between those

so uh i created not created actually i just uh copy it from the cupid shuffle some steps to introduce you to the lateral movements so everybody knows that music is like to the left to the left to the right to the right now kick now kick now walking by yourself but that's a really anthem um at least through cruisers and things like that in marriages so when we talk about ladder movements on the data plane uh you between two two ec2 instances on different uh vpcs uh connected by a vpc period some things doesn't work as expected for example iarp protocol rp protocol doesn't work as expected because in aws you don't have the broadcast broadcast

so if you enter in console and do a arp dash a you won't see any machine just your machine in the router so there is no easy ways for an attacker to find out which other networks are connected to the to this account so the solution for this is to do a network scan so you can scan the whole range the whole slash 8 range it's nice but effective it's gonna if the if the blue team of that company has some flow logs and things like that you're gonna catch him uh but it's effective for the attacker so he could define a new instance on those on other vpcs and on other subnetworks and then hack it their way into this new

instance another kind of lateral movement uh is when the attacker came from the ec2 he hacked the ac2 instance for example and or aluminum or ecs or a batch job there's any kind of data plane uh things that can assume a roll and he can just get those information enter on this machine if there is a role attached to it he can install this credentials and jump into the control plane by doing this something pretty much like this so the attacker lands on dcc2 instance so he does a sts get caller identity to find out which hole he has and now you can see that he has the thing g2 roll so he's going to dump credentials to

jump to the control plane by doing this he is working he's using the metadata metadata is a service from aws who gives a lot of information about the the role that that the htm instance has and credentials information about that instance so here the attack would be able to get the credentials and then he can use uh some tools to post exploration that one of the tools is the paku it's a open source aws exploitation framework developed by the renal security guys so if you guys don't know this tool just download and play around really good too has some some defects i don't know how to say that but it's pretty good to enjoy so

there is another kind of letter of movement that i call here step down and that means when when an attacker is able to get any kind of credentials or access keys or login and password without mfa from a user or perhaps with mfa with phishing the attacker get this credential connects the control plane and if he find out there is a vpc there and he wants to see what else has that received he doesn't have enough permissions to walk around that networks well he created a new ec2 instance then he send a command for that institution for example using the ssm or loading the data inside the user data for that institute instance and then he can jump to that

instead of instance and execute code here in this example the attacker sends an sscm send command to an instance and creating a reverse shell for here attacker ip address on port 3137

so and when we talk about lateral movements on the control plane things are a little bit more trickier than on the data plane right so uh let here we have three accounts let's assume that we have three accounts accounts a account b and account c if the attacker gets a credential an account a and the account b has this following a policy here that says that the principal is aws any entity coming from the account a so a user is an entity from a country hey can assume road so an attacker would be able to jump from the account a to the account b without no problem and assume this role but when you think

uh when an attacker got access to account b and he wants to jump to account c you think you you're tempting to think that this policy must be enough for him to jump between accounts from county b to account c but that's not true because when it happens when uh the the attacker assuming the role from account a to account to b and then try to assume a role in account c he is not actually coming from the account b he's coming from account a and this policy won't work the true policy that has to work it's something like this it has to be a assumed role for an account b uh with a role and when you assume a

rolling account you should you should usually you put a value like a testing or your name your username something like that and it also has to happen here in the policy on account c i will be pretty honest with you guys i've never seen this in any of our clients nothing like that so i suppose not work like this and the thing is that that that thing that we call uh assuming role a a sumo rule on b and a sumo ronal c it's called rolly chaining and this has a hard limit of one hour from aws so even though if you could be able to assume a roll and be in a summer roll on

c uh the maximum time that you can assume that role is for one hour that's not much a big problem because all the apis from the aws are really well really well documented and attacker might script their way and creating persistence in back doors all over so this roll chaining is not a big problem but an attacker can be creative right and usually they are so there is another kind of movement that are called now the economic because when this policy comes from the account b the account b is the route an entity on an account to be right so if the attacker has the action that ability to create an institution he would be able

silence i kill you share the picture whatever all right so when they are county when an attacker and a county would be able to to exact uh create an ec2 instance on the same account the ac2 instance is a entity on this account so he would be able to assume a role give a permission to entities institutions to assume a role on account b and he jumps to account to be as an entity assumed by account a but he also could be able to create a new instance on account b and then jump to account b and then assume a rolling account is c and would be something like this so he he went to the control plane down

to the data plane back to the control plane down to the data plane and then back to the control plane again until he reached the account c so that's occurs because aws the doesn't reach ac2 instance uh institutions using roles as a roller chaining thing so there's no limit there's a limit to up to 12 hours but it's a way to bypass that so what are the attacker requirements to make this thing work so he has to have enough privilege on aws so he must have sts assumed role and perfectly resource star that means that he can assume a role in any resource and he also has to know a related account in the same organization

so this can be found in clown trail i am policing through synth sometimes you go to the github account from that organizations and find out they have like three or four other accounts which with account numbers shared there because account numbers are not really a secret right so you you can share it's not supposed to but you can and they also has to know uh of a role to assume in the new account so when you do an sds assume role you have to know the iron that you the the irony of the role that you assume in the the new account this can be brute force because usually it's created by humans so you can read first like a a row like

accounts number role admin administrator and things like that but there's a special case because when it's coming from the master account there's a special case for that so let's talk a little bit about privileged escalation privilege escalation is a set of actions that allows a principle to increase their privilege for instance you can imagine a user with a really low privilege but with enough enough actions to improve his his actions his capabilities there was a great research by the late spencer guitar from reno security labs and he found out at least 28 28 no privilege collation techniques and there's a there was a really new one released by spooker labs today in his talk if you guys saw it

on apple stream about app stream and there is even more possibilities because with combinations with ein pass road as this is an action on aws and there's a really good research by noah and dahm and when he find out new kind of privilege escalations combining iampa's role and other actions from aws but the thing is not all privileged collisions are created equal let's take some examples uh not all the techniques have the same effective effectiveness and allow you to become an administrator a sap so here we have two techniques the first one is the iam create pulse version that allows you to become assistant administrator as just creating a new policy version and becoming an administrator right

but there is the another technique called set default policy version that depends on you have another policy version that allows you to become administrator earlier so if you have the pulse version if you want here that allows you to set default pulse version but the pulse version v0 doesn't allows you to become a 16 administrator this this technique of privileged escalation might give you some more permissions the permissions of the version zero but not become uh assist assistant administrator right so there is a tool that's called thoughts planning anyone have ever used it yeah it's a good tool it was created by kinner mcquaid uh he he was working on salesforce at that time and released these two on april 30 2020

and the current version is zero five oh but the good thing about this too is that it downloads your ian policies uh in line policies custom policies and aws policies and analyze those policies looking for privileged collision techniques that exfiltration possibilities of latex situations resource exposure credential exposure and infrastructure modification so you generated these kind of reports a pretty pretty html report with those techniques and which is bar graph bars and things like that so it turns out it's pretty easy to analyze privileged escalations with these two so another attacker requirement is to know the i the account id of a related account on the same organization right so this can be found on cloud trail and

here we have an example of a cloud trail of uh em wrong getting away the psp aws api call uh disclosing an account id and you can also be found in aim policies because if you have cross account policies and those cross account has to have the number of the accounts uh to the destination right so you can find out new numbers there and also through sync as the example of the github that i gave you guys before so and the the third thing and the third trick actually if the targets use aws organizations or awsc control tower the role that organization that aws organization creates and control tower creates on the the child accounts on aws

it's a well-known name it's called organization accounts access roles and for the control tower is at aws control tower execution and they have admin privileges in the child accounts by default so if an attacker gets access to the master account he will know by just doing our organization list the number all of the the number of the accounts he has and he can also jump from his account from the master accounts to any other accounts because he already know the roles for that like organization access or aws control tower even though the aws control tower's efficient role has two principles one is defining the services so control tower can also assume a role on on your

account but the the thing is there is a principle here from the master account as any entity on the master account can also assume this role so you if an attacker got access to a master account it's done game over so let's run some some true scenarios on aws some usually patterns that people do on aws so the first pattern is called hub and spoke when you have multiple accounts which accounts for a production account for a development staging and so on and a shared account who shares cycle database and things like that so as a aws administrator you can do some kind of mistakes for example for example creating an ac2 instance on the shared

vpc with two enis one routing to account a and another one route into account c what that means that means that if an attacker could hack any accounting at any ac2 instance in account a find out there is an instance on the shared vpcs hack the charge vpcs here you will see that you have two ni's and then another one will allow you to allow him to jump to account c and the other things that you can do on this kind of scenario is do vpc peering so if you do vpc in between account a to account b to account c you are basically making your network flat and there is another pattern that you

called cicd account it's like basically like this you have a master account and you have your ci cd account and this icd do deploy for that for stage and for project accounts and the master can access the dev and the stage a user in a master can access the dev end stage but he can use usually access the project account what an administrator can do to break this pattern is create a cross account between the dev and the ci cd the backwards you see and when he do that the user might be able to get a credential and have then jump to cicd and then jump to prod just by assuming roll between control planes

so in order to help our customers and find out really easy ways to find out previously lateral movements between accounts and those kind of things we tested a lot of tools but none of these what we expected them to do so it's time to build our own right we tested the mapper from ncc groups we tested uh a aws px from f secure labs and cloud mapper from dual labs and a lot of those but none of them did what we expected them to do easily actually so we had some some requirements right so we should use open source in free tools we should download automatically through the aws apis all the necessary input to

plot the graph uh from also from the control plane in data plane and also has to have a query language to allow razer alerts automatically so we decided to use gremlin as a graph database carry language and for doing this we are using a patch tinker pop behind we are also using cloud spaining to allow us to log in multiple accounts and download the all the information and also some uh splitter scripts to download the information about the the control the data plane things that we need to to our project and we decided to use the 3js that's a amazing graph library pretty easy to to make things on that so our model how we model that

well all the squares that you guys saw in the presentation uh are things from the data plane like ec2 instance lambdas ecs or anything like that and everything that was circle uh is from one account and the colors differ different makes the difference between accounts and also we have an arrow with a direct directional relationship and so it's demo time the thing is this tool has no name yet so vote on twitter i just make a booth this morning and you guys have this only these four options of names sorry but yeah that's what the twitter let allow me to to put on percentage um on as options and please vote and we will publish the two as soon as

possible so it's demo time all right here this is the interface of our no name yet too as you can see right here on the left and we used we based the this interface in a project from the github the project called graph xb from a handheld guy that i don't know him but i was kind of we just uh modificate a little bit the code to make this work the he has a javascript gremlin carried interface with the the past tinker pub so just to grab i already loaded all the information from the aws and just click search you'll be able to to graph this information so you can see rounds are control plane stuff

squares are data plane stuff you can zoom in zoo out bend for the left to the right

but the data that i show you guys here it's like a mock-up data from one of based on one of our clients who has hacked so the attacker would be was able to create to grant his credentials create an easy to instance and just do data mining but the thing is if the attacker would be able to it was a bot so it was not smart enough to find out lateral movements but if it was a person if it was a real attacker and he find he could be able to find out later movements our clients would be screwed and i show you guys why

you can you can move the the pieces around and this is the user who got the credential leaked and the attacker was able to create an ec2 instance and this tcc to instance was in the master account so he would be able to jump to all those other four accounts that he has the or the green the blue the orange and the light orange accounts and this client has also has a vpc peering between other accounts the orange account so this instance he would be able to perhaps hack this instance and then jump to another instance and then also become assistant administrator because this instance here was on a private sub network but he has the role of the

administration role attached to it

and the easiest way to analyze things like that is to look for clusters like this one and this one this second to second cluster is from his cicd so there is account who will deploy here and account for deploy here and that is a easy instance with jenkins here to deploy for those accounts and things like that are usually lumped with us with a role because london has a role attached to it and they're like sparse they are not connected to the whole graph

and you can also use a filter that privileged collision here and you would see that you fade out or raise the transparency the other nodes and there is a possibility of privileged escalation here from between those two rows and if you click on the edge you'd be able to see the there is a label edge for privileged collision with the create policy version so you can also carry information between those nodes

so we have some future work to do or our to-do list as we used to call so one thing that we want to put on this tool is to be able to evaluate restrictions of all privileged escalation techniques because for example the example that i did that i said before there are some kinds of uh is the privileged correlations that needs a c2 instance that leads lambdas if there is no role within that that that account with the permissions enough to that lambda will not be a privileged validation at all you won't be able to get more actions we also like to edit support for other resources like s3 and rts support for resource-based policies and

a better visualization and filtering for these two and we can draw some conclusions from here well first of all is try to apply the least privileged iam role permissions and i know that's these are said and done but you can abuse of scps and im boundaries to limit uh most of most of what an attacker can do in your network and never use a resource start because when you do resource start you can do those actions that it has permissions in any resource so you should a good thing would be historic the kind of resources that uh that attacker can make moves so other thing is don't run local workloads in your master account so your master

account in aws from organizations should be empty as much as possible don't cross account don't cross any account to the master so that case on the cicd when the there is a connection between the cicd and the dev and the that back to ci cd if you do this kind of thing things you are alone allowing privileged escalation a privilege collision i'm sorry you're allowing lateral movements and always review all your trusted across account policies and for sure segment your networking accounts as much as possible so thank you guys that's all i have for you today any questions the code will be released on github tnt security and hopefully soon any questions i've got the microphone here if anybody

wants to have a question just raise your hand and i'll try to hand it off

i'll ask a quick question a lot of the stuff you talked about today does that is it also apply to other cloud providers like azure google compute platform uh the lateral movement yeah it's possible you know on all other cloud providers but the things like aws organizations are basically on aws thank you thank you very much

his last name is stanchi

yeah i was kind of nervous because english not my first language yeah i get nervous in english this is my first language but you did you sounded great man very confident and the presentation was really except it was really interesting thank you very much with a lot of effort on this do what i put a lot of effort yeah it showed it showed it went really smooth it was like i figured this i mean i know the content is new it's like it's not like you've been doing this presentation 10 years or something you know it's like you know you're very polished so i appreciate you coming out and giving the talk ah thank you

very much yeah can i grab the mic from you real quick yep [Music]

talker at village which village yeah perfect awesome all right well thank you we appreciate it man yeah make sure you get all your power adapters there i don't wanna

hey

i want to be comfortable

that's right

[Music]

[Music] do [Music]

[Music]

do [Music]

[Music]

[Music] do [Music]

[Music]

so [Music]

[Music]

[Music] do

[Music] so [Music] do [Music]

[Music]

good afternoon and welcome to b-sides las vegas lucky 13. you are in breaking ground this talk is titled cookie monster exfiltrating data and more bite by tasty bite will be given by eric and mick whitehorn gillum just a few quick announcements before we start here i would like to thank our sponsors especially our diamond sponsors lastpass and palo alto networks and our gold sponsors intel google and bluecat it's their support along with our other sponsors donors and volunteers that make b-sides possible regarding cell phones these talks are being live streamed and as a courtesy to our speakers and audience we'd like everybody to silence your cell phone if you would thank you if you have a question

use the audience microphone this is the one i'm holding so that youtube can hear what your question was we appreciate that uh as a reminder the besides photo policy prohibits taking pictures without the explicit permission of everyone in frame again these talks are being recorded so these will be available online youtube in the future last but not least we would like to please ask you to please keep your masks on at all times and with that let's get started please welcome eric and mick thank you i hope everybody's having a good time and is excited to be here is actually making ir to be here talking uh we've spoken at other b-sides but another las

vegas first time here so really excited about uh being able to present in front of all of you so as we said it's a cookie monster actual trading data and more the and more kind of got dropped off we'll talk about that towards the end yeah it kind of dropped off and this really has been here it's been like four years in the making we built this utility that we're going to talk about to serve a specific purpose and then over the years we thought maybe we should do more with it back and forth back and forth and finally decided that we wanted to make it publicly available to people but overall this tool while it serves a

purpose it's really more about getting us thinking about our networks right as most of the conversations and talks have been here it's about knowing your network and visibility into your network and what's actually occurring and that's what this tool is here to help do before we start talking about it a little bit about us we both work for secure ideas it's a consulting company security consulting company based out of jacksonville florida our ceo kevin johnson former sands instructor started it back in 2010 it's our 12th year hard to believe we've been around that long but we do pen testing et cetera right so what we're talking about is something that we've tested it's not just us

coming up with a theory it's something we actually tried my name is eric keane i'm a principal security consultant uh before i started working for secure ideas five years ago i was responsible for very large active directory infrastructures at fortune 50 companies like windows environments i know that probably gets me booed off stage when i say i actually kind of like windows but i do and when i'm not working i'm a movie enthusiast i love movies i was actually a film major acting minor i love watching movies now that my kids are older we're getting to watch all of the really good movies uh it's not just you know disney movies which are nice but i

like what action films too uh so i'm mick whitehorn gillam uh i'm from canada but i'm i don't live there now you probably tell that uh i write code i've been doing it for a really long time uh started when i was six basically uh on the command line migrated to batch files within a couple of years and then actual real programming languages not long after that and if you can't guess my age that makes me old um so yeah uh i used to run long distances says i do on the slide but it's been a while uh covid unfortunately i used to lift weights and then covet yeah so yeah it's harder now i'm heavier now um

because you know that happens too so we're here to talk about data exfiltration and i'll admit for us when we're doing our pen tests this is usually not something that's on our radar at all right unless you're a very very mature organization and you have some controls really probably all i'm going to have to do is just browse out to an s3 bucket and i can take whatever i want off your network right that's not true for everybody and and you know there's all these wonderful options you know sure you can use command and control channels you can use network protocols ftp https uh cloud services like i said one of our favorites code repos then there's all

those physical yeah i need to move a little further away from you evidently we'll get an echo physical things that you read about that sound so awesome but i don't think i've ever had to do or probably never will but you know using lights on a device and doing morse code or something strange or fan speeds or cable harmonics all this stuff but like i said that's really really high tech and not often what's needed and i mean i love my picture there of the the monitor on the copier right for that physical access in case uh that's all we have is the ability to print something and when we're talking about trying to get data off of a network there's all

these controls that organizations are putting in place there's a considerable amount of time effort money etc all they're trying to prevent what we had to do you know our small list you can limit the open ports which most orgs are doing yep only http https although you'll still find even large orbs that somehow forgot about some random port that suddenly you can get access out there that's not covered by a firewall or a a proxy smb oh let's not yet please know although it does happen smb although i've seen smb in that's even worse let's move past uh then we have next-gen firewalls right seeing what's going on doing that deep packet inspection all those intrusion detection and

prevention systems the inspection proxies that are ripping apart everything that we tried to hide over that tls encryption right let's break that apart then you have your dlp software making sure people aren't opening files that they're not supposed to with credit card information or whatever it might be then your drive encryption so that people just can't walk off with your laptop hopefully uh and then you know of course we have netflow and other things that are showing what devices are actually talking to and where right all of these things that somebody is going to have to try and defeat of course once again if you don't have all of these things maybe having somebody trying to

exfiltrate data should not be one of your top priorities but that's what we were trying to solve way back in 2018 hard to believe it's been four years we were actually doing a red team assignment different from what we typically do we're pen testers but we do red team as well right pen testing if you're not familiar with the difference red team and pen testing pen testing is down and dirty we have like this much time to get wherever you want us to get to red teaming we're going to try and be really stealthy we've been on the network for a while and our problems really came down to that last bullet point the blue team was

really responsive you messed up at all once and they were on you we made a small mistake on one device trying to test a website to see if we could get a phishing campaign to work they caught it that domain was burned i'm sorry if i'm echoing i don't know what i did wrong but our client wanted us to exfoliate data right they had these controls they only allowed https out not over not even http they proxied everything they were ripping everything apart they were inspecting it uh they limited where we could go right we had very limited domain options um content filtering dlp software that everything we knew that they were ripping the traffic apart but we couldn't tell what

they were using right and they wanted us to get this file that they gave us that was full of credit cards and other sensitive information and see if we could get it somewhere we didn't want to try that the good old s3 bucket they weren't using the cloud it was 2018 this institution hadn't moved there yet and we knew we had one shot so mick myself and another gentleman we're sitting around we're discussing we're like what are we gonna do that might mask this and let us get some data off of this network we need something that's quick we didn't have a lot of time we're getting towards the end of the engagement and we need

something that's low-tech right we don't want to invest the time and effort to try and get one of their devices to do some strange harmonics on a cable or something so we actually said what's normal in web traffic that's there that probably isn't going to be looked at by anything and that brought us to cookies right so uh just to do a quick overview because people are at different levels with cookies most people probably if you deal with http traffic on a regular basis this is going to be why is that guy explaining cookies um but for the people in the audience that might not uh so example there that's a request there's a cookie in it that's

being sent to the server uh in in normal traffic uh that would be set one of two ways uh would be common so either a response header would have come back that said set cookie and provided the value or a response of some other kind like a json response would be a common one sent the value back and then javascript on the page picked it up and shoved it in through the browser's api uh those be the two ways they normally would get set and then when the requests go to the server whatever the scope is for the cookie those cookies get included in a header just like in the picture there and the important part is as you see

from our website secure ideas right it's pretty much unintelligible right there's nothing there to give you any indication of what it is and if that's what a system is looking at to determine if this is normal behavior or not i we were pretty sure we could do something that would look kind of like that and bypass all of those wonderful controls which brought us to cookie monster so cookies have a max size right you can't just send that that whole file all at once it's going to be way way too big so we said let's you know come up with this idea let's encode the data take that file just encode it read everything read the contents and

code the contents break it apart into nice sized chunks send all of those out and request as cookies merge it all back together on the back end bam we ever file right and of course we needed a mascot as soon as you build a utility if you don't have a mascot it's not a utility right okay so cookie monster v1 back in 2018 built on node we had a web server in our case it was apache because we wanted to have that tls encryption just because we didn't want to send anything over http and it wouldn't have worked anyways through the proxy so we had the cert and then we had the feeder the one lim

the one thing that we found on their network which was very nice for us it was 2018 so powershell was still a very good attack vector back then right not quite as good now we can have discussions about that i think it's still a good attack vector but powershell was enabled we could use it so we built the feeder the script around invoke web we create web request this is sending just like it sounds a request as if it was like a browser to a website so we encoded the file on memory all right took up a little bit of space we used that normal commandlet to bypass them if they were looking for something

using a.net library because that is a little odd in most cases we made sure to set the user agent on the on the request so it didn't say powershell right that would be kind of obvious uh and then we also said let's allow it to send a whole bunch of files not just one and then put that random sleep in there so it's not just you know sending requests over request of a request let it take some time somewhere between oh like half a second and a minute whatever you want to make it look a little odd and not sit there and be a consistent state it was simple but effective and you know you look at that

and it looks very different from our cookie and i'm kind of embarrassed to say that this is what we came up with but we were running out of time and the funny thing is it worked right this bypassed their content filter their inspection proxy everything to us any person looking at it who has any idea what a cookie looks like this looks pretty odd right we've got double slashes in the front and that was because we were encoding certain things we didn't want it to be able to see it that's not normal uh we have some random integers in there that we're telling cookie monster the back end hey this is part one of x so that incremented and

then we had our payload which was the first that was part one of the file and then we had some other bad things in there right the web server did things like said okay whatever but overall it worked um if somebody would have seen this while we were going through the 45 minutes or so it was sending the data um you know they would have instantly known it was wrong right if they would have gone to that website that we had picked when we got that domain name it would have errored because it didn't have cookies it was down and dirty but it worked it's not good enough we know it's not good enough right that that was like gen

one let's let's do a proof of concept we needed to be better we need to be hiding in plain sight better than we were this is a picture of a snake do you see the snake no she's right there do you see him it's a copperhead okay can you see him now that's the nope rope that's the bad end of the nope rope you don't want to get bit by it would be really really bad okay this is one that mick saw when he was running when he was running that's not true yeah this this is what we needed to be we needed to be like this guy so we came up with version two

we said all right let's let's take a step back we want to stick with our same premise but things have gotten better right we're sure detections have gotten better so let's set up some other stuff for us number one let's set up a unique id for for our device because before it would accept like if we had four computers all sending files at the same time it was going to get confused really fast so let's add a unique id let's do some better padding let's get rid of that slash in the front right let's do something a little bit better do some graceful handling in case the server goes offline or something strange so that we can do some retries because

when you encode the contents of a file and you missed a section you pretty much lost everything uh we make made a beautiful ui all right a functional ui and a whole bunch of other enhancements and then we put it on github because well once again we think this is going to work i admit we don't know if this version is going to work so when we get to the question time and you say is that going to work we're going to say maybe we have no reason to believe it won't but we admit as of right now no one has paid us to try and exfiltrate data out of their network if you're interested in helping us test this

please let us know reach out right we'll be happy to do it or if you test it yourself let us know or test it yourself let us know we want to know so the server enhancements all on mick oh that that's right that's me that's you uh so uh i rewrote it like pretty much the whole thing uh updated it to note 18 is what i was using but it would probably work on 14 or higher uh i did stuff i so user interface uh is there still doesn't have built-in tls but we do that through a reverse proxy definitely if you're using it for real data even in you know a testing environment be nice and tls encrypt the data uh

keeps a lot of stuff in memory at this point in time when when eric wrote that bullet point that's because that's what i told them but it's not actually entirely true anymore oh i need to change it all right um it does write a temp file at one point uh but yeah so it still reassembles still decodes and it can send well i can send one command actually i've it can send am i premature on this well go ahead it can send one one command to the server it's kind of hard coded in there it's who am i so you can send who am i's up to the server yeah we didn't really want to push out a full

c2 framework because you know ethical concerns and all that uh so we did it that way because that way if anyone uses it they'll probably get detected that's right hopefully because everybody knows who mi is the immediate notice that you've been compromised right who am i equals you're in trouble the only people that don't know who they are people that don't belong there so the feeder still using powershell i like powershell right windows guy i like powershell i know it's protected right you can do all sorts of things to see what's happening if you've turned on the logging and everything else many organizations still haven't it's a little bit more customizable you can set any user agent you want it has a

default that looks like a windows 11 box you can set it to go through your own proxy you can reconfigure the cookie size to be anywhere from well pretty much anywhere but you don't want to go over 40 4 000 bytes i think is the limit uh and and now it adds some other things in there right it goes to a random page itself it kind of picks some random page to send you to uh and it has that heartbeat option and yes we did originally have the idea that we wanted to to accept commands and do functions based upon commands but that was a little bit more we like the idea of this being a way of testing your

exfiltration controls right now more than a command framework it probably could be expanded out but we'll let that up to you if you want to do it so now everybody needs to like with me hope please work that the demo is going to function because as you all know they never do but hey we'll see what happens all right no minimize thank you all right so here we go here is our file of credit cards i don't have dlp on my box right but you know we'll assume we got past it just random credit card numbers and just to show that it's a new thing uh where are we we're in b-sides las vegas all right

so over here we have our feeder so it's a simple little ui can everybody see it here let me make it a little bit bigger here there we go so you know you can change a couple of settings you can add some things oh we're out of frame we're not a frame sorry i'm going to start the server

and so over here we can look at the powershell if we want i just have my simple little uh script it says invoke the feeder cookie monster feeder give it the url i want to go to the base url a proxy because i want to send it to burp so we can see what it looks like and the file name and now everybody crosses their fingers hey all right we are we are successful it worked at least it started so here we go we could see that it's getting data and if we look in burp right it's sending us to random pages and to top it off the server is responding back with data right

yeah this is a big win you know it's not just something you know that says hi uh yeah it's a small selection you could put more in if you want but it's our small selection and it's running through you can see it's doing all the gets we have it doing it get it'll do any web method you want what we found is you know miter and many of the other groups that are seeing how data exfiltration happens it's via posts so if that's one of the things your data exfiltration is looking for it's not going to catch us because we'll do a get or a put or whatever we feel like a delete it doesn't make a

difference trace or we can make up our own method whatever we want yeah and we can see here we go the cookies look a lot better that looks a lot more normal right like a cookie and that has everything so we also included we decided that our identifier should be like a google identifier because everybody's using google for tracking their website so there you go so that identifies the box we still have our our session data that's holding it but we added a random letter in the front all right so in theory my file is sitting over there

i'm just going to scp to the box to grab my file oops helps if i change the ip address

all right and in theory if this worked there's my new file the exact same one it worked oh always a good day always a good day so but just to show in case you want to do the other thing we have this other option for you so now cookie monster is running in the background every 30 seconds or if you want to change that he goes out and he just does a quick heartbeat to the server

uh and we can see there there he is that should match him but i'll trust it matches him it matches and there we go we're going to do the one command here who am i and then we just sit here and we wait and we'll go back to burp to do there was the response back from our seat from our our server saying hey please do something so we said set the cookie my server and cookie monster responded

and there you go right so normal web traffic it looked just like web traffic once again we set this up just to do this one method right using powershell it doesn't need to be powershell it could be anything you want uh all right so let me go back so i can

we didn't really explicitly call it out there but so a lot of people probably recognize that cookie was base64 encoded you we threw another character on the front of it just a random character that we ignore but it's enough to throw off the padding of the base64 so if you grab the whole thing and decode it it looks like junk yep so if you just grab the actual encoded part you're good but once again we're not trying to defeat people right at no point am i trying to defeat a real person who's looking at this because if your job is to be looking at every single web request going out that's a really bad day i'm sorry for

you right we have systems that do this and we just need to defeat that system so pros and cons of cookie monster pros standard high use port right web traffic happening all the time everywhere it runs as any method we could make up anything we want once again it doesn't care it's just running in the background looking for cookies right the cookies are are you know once again cookies themselves are typically encoded or encrypted or some random stuff that isn't going to be inspected or useful cons there are cons you can't make a perfect tool unfortunately it requires a lot of web requests to send a message right depending on your file simple math if we're only going to send

500 bytes at a time you're sending a one gig file that's going to be a lot of requests we admit that but once again you can space it out over a long period of time too you don't need to send it all at once it can make some files bigger because encoding things the data that it could make it a bigger file than what it was originally um and then of course you have to host it somewhere so if they have some really good content filtering or based on reputation filtering it may not work but it's there and once again we we know it's it's listed in mitre it's listed in other places the this is a attack that

is known we just tried to tweak it a little bit and pick an area that might not be used setting a custom server header is is easier to pick out than trying to inspect cookies i i don't know why a real legitimate detection system would be trying to break cookies apart but there could be a valid reason and then once again your max size is just over 4000 bytes because that's the biggest you can actually set with a cookie or the the entire um it's the the entire header yeah that header and everything breaks yeah but it that would be that would look weird that would also look very weird so yeah please don't send it that big

so where do we go from here well you know we think more feeders would be good not not just powershell python we've talked about it why even limit ourselves to just a programmatic way let's build an actual website that's just sitting in the background looking for the cookies and then you can browse there yourself and just send the data and it looks 100 legit because i'm not actually going to move even as fast or as normal or repetitive as something with a sleep or you upload an html file on it i'm relatively confident i could implement that that feeder in the web browser uh we want to enhance the server some more we want to store the information

somewhere in a little bit better we still need to work on breaking it apart by device we have some potential limits there even better disguises because we know we can have them better web pages like we said better things to see uh yeah so a couple options i floated around there where one would be use use an interception proxy to collect a bunch of responses to then pass to this pass the project file in there like a burp project or a oasp zap project or proxy it and actually point it at live sources and get your responses from there just forward them on perfect better obfuscation redirects et cetera we want to have other encoding options

why limit ourselves to cookies let's use jwts right the signatures in the jwt that is 100 unreadable it's supposed to be it's always opaque anyway yeah it's always opaque i mean that would be incredible uh we just didn't have time to do it uh do some iterations of encoding uh have the server actually dictate the cookies because right now it's hardcoded in both right so this cookie monster first thing before it sends a file goes out and says hey what should i do we get the cookies back to actually look like the server is setting some cookies uh all sorts of things oh and let's actually have the ability to have cookie monster tell the feeder to stop doing

the heartbeat that would probably be a good thing to add in there yeah um and you know then we have these other questions like we said the whole command and control framework is that something we want to get into i don't know we think once again we think it would bypass it but i don't know if that's something we want to be responsible for you know people have written command control frameworks and like this is awesome let's release it to the world oh boy what did we do right that might not be where we want to go with this so what's the point of talking about this awesome tool other than saying hey we think we made this awesome tool

how do you detect or prevent it well all right so you're gonna have a large number of requests going to a single source or will you why not have like one central server just getting everything from a whole bunch of websites that you create out there that's going to mask it even more all right so maybe that's not a good one uh reputational based filtering yeah that that'll work oh wait all of the people who do the reputational based filtering will be more than happy to tell you the web domains and what their reputation is right now all right and it's easy to find domains that are expiring okay so we can't use that uh sending cookies without getting it

from a server okay that's pretty strange except all of its websites that say remember me forever yeah because of long-lived cookies it wouldn't be that weird to see traffic today that you didn't see a set cookie for so yeah uh let's see cookie's being rotated without the server yep uh the endpoint protections and heuristics absolutely but as we all know those you don't want to put all your cookies ha in one basket right or all your eggs in one place they can be broken and and let's face it that may not actually catch it depending on on what's going on in the background it all comes down to what everybody's been talking about like all the keynotes

and everybody what's normal in your environment where are people talking where are they suddenly going that they weren't going to before that's what's going to detect it right knowing that level of information but maybe we missed something right is this did we write a utility that nobody needs did we solve a problem that didn't exist anybody have any ideas i think we came up with something different and something a little unique a new take on stuff how would you protect against it in your environment that's why we came to breaking ground right we're like hey we think we have something we thought we've come up with ideas to break all of the ways that a system is going to try and

detect it what did we miss anybody have any ideas if you have a question let us know um you know we're here because we want to get your thoughts you know know who's using that sensitive data where it is don't let it float around and i that's it so we're really early we'd like questions or we could go through the demo again and watch it break since we have to get hubripo on there i i just want to say there's some really bad code in there fair enough and i'm gonna i'm gonna repeat that i said i was a windows guy i can write powershell that doesn't mean it's good looking powershell it's functional it works but it it's not good

looking so eric has an excuse i know better i run a dev team and i told my developers don't write stuff like this we will have a hard conversation if you write stuff like that it works uh it's not pretty it needs a lot of polish but it's there it's based on an idea we know it worked four years ago we think it's going to work even better now we'd like to know if it does for you if you ever give it a try any questions please give us questions yeah i've got the microphone here if anybody has a question or wants to try to answer the question they had at the end about how you would prevent against

this against this i can just raise your hand and i can bring the mic over please give us something coming from the back it came all the way from the back um so one question i have is in your previous slide you had um you know sending cookies without getting them from the server cookies rotate being set from the server or you know stuff like that so i mean to me that sounds like maybe a spec change on the http side you know to deal with that because implementation in the browser but a question i have and this is this is the question what in your research is anyone else legitimately rotating cookies or setting

cookies without receiving them first so is there any false positives we might get based off of that so absolutely the seeing a cookie being sent before it was set will absolutely happen right because okay let me take a step back if you're sending all of the traffic from your devices all the time through the proxy even when they're at home and not vpned in okay then that would probably be pretty rare but how far back are you going to take that data set right how long do you say it did i see that cookie a minute ago or did i i look back a month ago right because right websites remember me forever so absolutely the looking for a cookie

being sent before it is set by the server would give you false positives uh the rotating that would be weird that would be a weird thing that would be uncommon so yes we do admit that is a limitation so i think i saw another hand over this way awesome

hey uh just like say like really awesome tool so far i love it man my question was when you said you have the reverse proxy handling say cls connections and whatnot um do you have kind of a fear of being fingerprinted using ja3 because i know uh ciphers and things that are offered to income to clients are being fingerprinted now like i don't know if it's on the internet but ja3 is a way to fingerprint c2 frameworks and certain endpoints based on the offer tls ciphers and whatnot is there any changes that have to be made to this framework specifically to accommodate you know randomizing that or is that more on the proxy side no so get

your if you get the certificate from a legitimate source right we're not using it for the encryption of the program we're just using it as normal tls encryption if i heard your question right so the question was do we need to compensate for specific ciphers or anything as far as going out and being used for the the reverse proxy correct yeah it was more so just you know will you account for being fingerprinted by offering the same tls ciphers over and over after each deployment of the tool uh so right so if we would stand this up in multiple locations or we'd stand it up again and like we use nginx is what we used if

we've got a cert for that server it would be one right whichever website we decide to stand up at the time to host cookie monster could have a new certificate so you'd go through the renegotiation as necessary but once again it doesn't need to be the same certificate everywhere okay cool thank you yep i think if i wanted to make it distributed i'd probably stick it in there as an end point behind a bunch of like cloud front instances or something like that who else has a question up front anybody have a better way of detecting it come on i know there are a lot of smart people in this room i know you're smarter than

me too yeah like i guess if you want to try like more uh obfuscation techniques to it or maybe make it so that there's not so many patterns would it add too much complexity if you have like a well i know that there are some a lot of links that have like really crazy parameters or like when well when you visit google sometimes people saw what it that it has like a bunch of garbage um what would look like garbage uh parameters and it looks like a really crazy link so i'm wondering if you do application and maybe throw in some parameter and use that to alternate sophistication would that have to uh too much complexity to

the project we could i mean we we absolutely could have it send parameters it would not add that much complexity to what we're doing or having the web server respected right it's just extra development programming um yeah i don't think that would be difficult at all so yeah uh to add some kind of generated parameters in there right we just don't want to we actually just don't want to send any real data through it right the whole idea is to keep all of the important data stuck in the cookies cool and would it help at all like uh to kind of alternate um the obfuscation technique you're using in the cookies yep possibly possibly like we said talk

about jwts is the way we definitely want to go next yeah jwt signatures would be a neat hiding spot you could do possibly some like base64 encoded image uploads you know you could stagger it in there there are options there are a lot of ways to go with it in terms of more variety and another simple one would be to just renegotiate maybe outside of that part but like any yeah i mean so extensive research no i mean when we need a site we have our places that we go that list recently expired sites you can check those through any of the tools to say what's your reputation do you have one right they provide it for you so

beyond that no i have not done extensive research into you know using this specific item or one that will absolutely be this uh utility versus another

hey uh i have two questions sure uh the first is have you thought about using websockets to create that connection and send the information that's funny it's like a continuous connection we actually did discuss that and so we went back and forth we think it could work but would that be inspected more than not so that's one of those it was one of those takeoffs and something we debated of potentially doing um yeah it really came down to we we were afraid that that would look more suspicious to somebody watching which kind of feeds it to the next question because if that is the typical way that client communicates on the web application could you take that into

account and maybe first of all clone that way of communicating via web sockets and use that as the base for your communication but then also could you take the cookies they typically send on their applications and then name them accordingly to then bypass any filters that might come on to the names of the cookies from before absolutely right if we wanted to once again we get to dictate whatever we want the whole idea is as long as what i am sending is understandable by the server it doesn't make a difference right so we could pick up the cookies if we found a website that we really wanted to clone we could right assuming it had the right

number of cookies in it for us which is four right now four or more or more um so we could but then of course then you have the other problem of hey that's saying that's rsa.com but it's not right so i'd rather go and pick up my sites now like my five six financial sites that have expired or whatever rated sit up there put up a good website that says coming soon and then when we need to exfiltrate data we just throw a cookie monster in the back end right and it's waiting there yeah ready cool thanks really interesting also the idea of a c2 server sounds really interesting with this as well so risky exactly that's where we were like

this would be awesome because i didn't say it i have a hate relationship with metasploit and many of the other frameworks out there i will type something say go and i get that horrible exploit ran but session failed whatever a peer of mine nathan would go up there just hit enter and it works right so me no bad i'm not good with cg framework so i like living off the land who else has a question there we go hey thanks for the talk uh quick question so you mentioned that the um the file you're trying to exfiltrate it'll break down and have like an id as part of like each request going through right so the files split

into smaller sizes if you're expecting a large file and maybe some of the requests time out and they don't send those chunks can you from the server side to say hey request part 5 or something so yes so there's another part that we started we went down that path of hey had the server tell me it failed or not and what we decided instead is to just we're doing it sequentially right so server goes offline the feeder pauses tries pauses tries and once it gets a good message it'll continue again um we did discuss the whole let's send a final act and do everything it just came down to time before we got here we do

pen testing a lot and this was a side project for mick and i that was yeah that makes sense that was the scope creeping on me because i decided to rewrite a thing yeah we write a thing to do a thing yeah and so we did change and uh so let me find i've been heartbeating in the background as we've been talking so let me find somebody so we changed the entire mechanism right this itn there oh you can't see it because i'm not sharing the screen

this entire itn there is actually the the encoded you know this is part we have 28 parts this is part two or whatever so it still gets that information it's just encoded in a better way okay cool yeah i mean my main thing was like obviously if you're exciting a large file and it fails you don't want to repeat the entire process all over again so just be able to say oh i'm missing this part can i just request this only yep that that is definitely something that we've talked about and so it would be a good addition to put in uh just time all right thanks yep any other questions make sure nothing on the front

hello i wanted to ask one thing uh i suppose this is quite stealthy on the network level but what about the detection when you have like windows defender and you're doing the base 60 for encryption or use the invoker web request functionality does this get it detected a lot or what's your experience with yeah no by by default okay so as of right now the feeder has not been submitted to anything that's saying it's malicious that could change and then it would right and then defender's going to pick it up but the behavior that we are exploiting underneath the hood no it's invoke web request it's it is an expected behavior of powershell we are

doing nothing malicious at all right it's not malicious but i think nowadays when you're basing coding for example or when you're using the invoke web request it sometimes gets flagged because it's unusual activity so it might yeah so if you have the full defender suite identity and all it cloud all the enti it very well could i i can't speak to that the default defender endpoint protection on its own isn't going to care does that thank you yeah any other questions thoughts is anybody going to download it and try it awesome who's going to download and try it all right eric secure ideas.com make it secure ideas.com let us know all right not a difficult email address

please let us know how it goes because we only get to try it when somebody pays us to so thank you [Applause]

[Music]

so [Music]

[Music] [Music]

[Music]

[Music] do

[Music]

[Music] do

[Music]

so [Music] so [Music]

[Music]

[Music] do [Music]

[Music]

[Music] so [Music]

[Music]

[Music] do

[Music]

so [Music] so [Music]

[Music]

do [Music]

[Music] do

[Music]

[Music] so

[Music]

[Music] so

[Music]

[Music] so [Music]

[Music]

all right everyone welcome back besides las vegas um could i ask you guys in the back to move forward a couple things we got a small crowd today and it'll just just be easier to pass around the mic for questions later on thanks um hopefully you guys were able to get a little bit lubricated so so to speak during the happy hour uh for those of you joining us at home hopefully you may might have joined us from home um but regardless welcome back to besides las vegas this is the afternoon session this is breaking ground and we have klaus here speaking to us with the title of the talk detecting log4j on a global scale using

collaborative security a couple announcements before we do begin as always want to thank our sponsors for especially our golden diamond sponsors for without their generosity along with our other sponsors volunteers and staff this event truly would not be possible uh and and i think i share the same sentiment with a bunch of other people that i'm glad that this event in person truly did happen um so hopefully this will just go uphill from here but regardless one important announcement uh if you are here for the six well i guess the next talk the next talk six pm with the exclave experience relocating to almost canada by t profit that has been moved still 6 pm but it's moved to a different

ballroom it's now at passwords con and the tuscany ballroom so if you're going to t profits talk make sure you go to the passwords con and not common ground instead for those at home same thing the the schedule has been uploaded updated on the website so you can just refer to the website that has the most up-to-date information with all the speakers as they are being changed but uh a few other announcements again this is being live streamed just as any other talk has been over the past few days this will also be posted later on youtube so please do make sure that your cell phones are silenced for respect to other speakers as well as those

listening on the live stream and later on during q a i know it's annoying it's a slightly bigger room there's not many of us here so i promise i'll walk a little bit fast but please do speak into the mic because otherwise the people online would not be able to hear your question and we really don't want to that's happened so without further ado klaus flow's yours take it away thanks yeah i'm taos welcome to my talk on detecting log 4k on a global scale using collaborative security but before but before we start this is my first talk in vegas so uh that's i'm really happy about that um but i'm of course i'm mostly i'm most

happy that about the fact that my first talk will be at b-sides because we all know b-side is always the nicest crowd that's that's how it is in the at blackhead they would have killed me so but then yeah another thing i want to say before we start is that um the first thing that the best thing that happened in my professional career was when i got fired a year and a half ago from one of the biggest danish retailers online retails and that's relevant because if i hidden if i hadn't then i probably wouldn't have been here today i've been an infrastructure professional for almost 20 years and i don't know about you guys but i just

never came around to finding out what i wanted to do for the rest of my professional career but i did find out some at some point that what i like best about my professional career about my job was all the things that i did in my spare time which is uh i've always been involved in all west i've been active an active member of overs copenhagen for 13 years some 14 years sorry and i'm co-founding besides copenhagen and what i what i've loved about that and and always have loved about it is arranging events and gathering people and see in their eyes that have had a wonderful time and and helping them out and and that's really what i wanted to

do and at some point i was so lucky at that to find out that there are people out there who does this for a living they ins they interact with the community they give back to the community which i love to do of course and help them out and and eventually i found crowdstick or crafts they found me depending on how you look at how i look upon it and now i have the greatest job in the world so that's it um first a little bit of background about how crowdside works because if you if you don't understand just a little bit about how it works then you won't be able to understand the data and and

understand and appreciate the method around log4j that i will be talking about later i won't be going into details this is as high levels as i could do it but let's start with the beginning the beginning is that as a cyber secure professional or at least cyber security interested you know there's something wrong out there right there are people out there spending a gazillion dollars on on cyber security and they're still getting breached so something is clearly up i mean what it it seems like the whole world has agreed that if you just throw enough money into a big bottomless pit then everything is going to going to be all right except it's not all right

that's evident for everyone right so our point is and and the reason why crowdstick is around is why don't we try something else for instance we tried outpowering the bad guys we tried outsmarting the bad guys why don't we try and outpower them because if you think of it there are more of a regular people like like us than criminals so why don't we work together and get those bastards right why don't we do that and then that is basically what crowdstair games should do but what does this mean in practice you can think of crowdstick as the ways of cybersecurity and if you don't know waze waze is a gps app which that basically collects information about

how fast your car is going if there's any in the accidents or any or any road repairs or anything around you so and sharing it with everyone else so that everyone has an efficient trip as possible the crowdstick does kind of like the same in that crowdsack detects locks or sorry cardstock read locks on the service you're running uh in it and it and it detects um it takes those threats um and then and mitigates them afterwards and there are a number of way different ways to mitigate um then it sends signals back on the attacks that it sees back to our data lake thing and after that it's been there all the ips are being assessed and turned into a

block list and shared with with the rest of the community and that happens that happens automatically um yep carson can detect a number of different attacks uh crowd claustic was originally i guess meant to replace fails of ban filter ban only does brute force attacks and that's so that's fine but crosstalk can detect a lot of other things on things on layer seven for instance the log4j that we'll be talking about and that's in that in that sense crowdstrike agrees the the web server log and intersects those special strings uh it's not uh ai or ml it's basically rule based and and the log passing and the log scenario file consists consists of some part of grog

which then passes the log file and based on that it detects attacks and and sends them back like i described before cardseg this is relatively current stats that we'll be saying we collect around 1.6 million signals a day we have around four million ips in what we call the smoke database which is three days of of bad guys um and there's around 20 to 30 thousand ips in this curated blog list that i talked about before and in a later shot i'll i'll be talking a bit more about the strengths of the crafted network meaning how how big a number are we talking about but before that it's important to understand that krausteg is free and

open source and the license we chose for making the open source is called is the mit license and the whole point of that is that once you have open source something during the mi with the mic license then it's not possible to close the source again and crosstake being a startup there is a risk or change or possibility that some big company wants to acquire us and maybe they want to close the source we we we will do what we can to prevent that because crowdstick is nothing without the community we do want to screw it over so instead we sort of set up like a fair deal at least we think software is free to use

um and if you want the blog list you share your own signals and if you don't well then you don't get the signals but crosstalk still works crowdsake is still able to partial log detect attack and stuff like that you just don't get any signals from everyone else a question that's that i often get asked is what about privacy and especially when we're used to as a security personnel hating everything that's free on the internet because that means that you're the product in this case you're not the product because honestly we don't care about you we care about the bad guys so what we're so what crosstake is collecting is literally just a an offending ip a

source ip a timestamp and like a behavior behavior is like a scenario that describes what's going on for instance like ssh brute forcing or credit card stuffing or whatever um this that's the only thing that's being um that's been recorded uh cardstick doesn't send your lock anywhere uh and also if some of them if some of you are from europe we have something called edpr which basically means that that we have to have a dpo a data protection officer we have to have made private privacy impact assessments and we also need to have processes to remove ips from this list in case somebody somebody who identified himself would do so and so far no criminals have done that so

that's good another question that i am often asked is how do we do or how do we deal with poisoning and false positives because the bloodlist is being distributed without questions asked so to speak it's being sent to the agent and the agents just accept it without questioning it this means that if we simply can't accept that something is wrong with this that there are false in this and uh to prevent in order to prevent poisoning or first let me explain a little bit about how it works when signals are sent through crowdsack it's it's going into the smoke database because you know there's no smoke without fire then the consensus engine as we call it is assessing the ips

finding out whether it's bad or not and if it is bad then it will end up in the fire database and consume and get distributed back but in terms of poisoning we have a mechanism called the trust rank basically whenever and all agents submitting signal surprise they have a trust rank based on how long time we've known them and have long uh and how long time we've known that that they are reliable so a new agent will start with 12th rank of zero and then if they send consistent signals for six months without any errors then they have achieved this trust ring of 99. the reason for this is of course to make it time consuming

to to if you want to poison us so if so if you're a bad guy and you want to poison the crosstalk database first of all it takes time the other it also takes takes takes a lot of asms because one in this voting process basically on the level that ip needs a certain a number of votes or two in order to be malevolent and those um the votes are given based upon the trust rank meaning that that that that simply an agent needs to have a certain trust ring in order to to be able to affect this process and on top of that um an asm only has one vote this and and this means that if you are

a bad guy and you want to poison the the crosstalk database you don't you can't spin up like a thousand dps's on gtp or somewhere else because there simply won't won't be enough asn to make any difference so instead you as a criminal would have to you know have a fleet of bad uh agents but spread across asn essence all around the world in order to you know ever have the chance of affecting the the consensus process but obviously that's not all uh all all that we do because we also don't want false positives and the one way to do that is that we have our own uh fleet of honey parts that um that we use to compare signals from the

from the crowd or test compare with the with the signals that we get from the crowd and then i need some water

that was nice the second thing or the second mechanism of this is um it's basically a white list so all google dns is all google seo bots all cloudflare cdns whatever the thing services that you that that you simply don't want to block they cannot be blocked and thirdly in the it sounds a little bit like a pre-crime maybe this but the way um the way crosstalk also works is that it it looks at ips from the same net block and at some point if karzak has seen and deemed enough ips from the same net block malevolent then it will just you know block the rest of the um of the net block regardless of them

having done anything wrong or not we just assumed that yeah you will also do something wrong so we'll just block your head and then if it ends up in the fire database then things are distributed back to to crosstake agents and if you want to know more about crosstalk then this is not the talk for it i won't be talking about with os crowd 6 supports or anything more faintly technical about that but if you want to know more then i'll be around at the pool party and i'll also be around the def con just follow the trail of the crowd stick crowdtake stickers like but like breadcrumbs i'll be around [Music] before we move on to the look for day

part it's important to understand that crowdstick is not a buff and this this means that the way crowdstrike works as i explained before krausteg needs a lock entry in order to block something right and that is that causes an inherent problem if you know a little bit about log4j because you know that if you're vulnerable it only takes one connection with just one log entry and then you're screwed so we very much want to first of all as a security professional i cannot recommend only depending on one control but if you use crowdsake as one of your controls then there is a risk that krausseg may not detect it and block it if if if it if it's an if it's if it's

an ip that the crowd doesn't know already but luckily um we did a little experian experiment with two identical servers just to try and try and explore how much of how much of what crowdstick is blocked is being blocked by community by reputation by the block list or or by local local signals and local processing so in order for that we set up a two servers on ovh cloud provider in europe they were completely identical both had the agent installed and another one had what we call the bouncer which is the ips part installed as well on the basically the the bouncer plug plugs into the to the host firewall so it blocks the connection

and it turned out that after three months we compared the the signals or compared all the attacks that had seen and it turned out that 92 percent of the bad traffic was aimed aimed at the server is blocked just based on ipv reputation so basically the um the the server that that was protected only saw like the orange currents part and that is good that is good news for if you're vulnerable for lofote and and plan to use car for the crowd car sake or for that or of course similar vulnerabilities

yeah and conclusion of course community matters as we all know um then let's move on to the part you came for i hope look for jay if there should be a symbol or a few person in the room not knowing what love for days i'll give a super short or super short resume outfit on december 9 2021 the apache foundation released information on the critical bug in the log for gay library which was exploitable via remote remote code execution um and i didn't know that before that before then but but as it turns out love4j is used everywhere and by everywhere i mean exactly that and this screenshot is from a from a tweet from a

guy called the cast vancouver cast vancouven and that's where it dawned on me how bad this is because basically he said his name into this dndi string which complains the contains the exploit and the payload and he got back connections from apple servers meaning apple servers were also vulnerable and i was like this is going to be a joke right so um yeah that quickly escalated into a worldwide panic and everybody was either patching or releasing free tools and resources to help out and so did we

as i said before on december 9th apache release information about the vulnerability on december 10th uh crosstalk released our first scenario uh detecting this lock for day thing and then on december 12 on december 13th and 16th the scenario was updated because as it's it's a quite natural um development because in the in the beginning it was just like a totally unobfuscated judge ring contained the payload then then over time it would be obfuscated in more and more in more and more obscene ways and of course the zarya needs to be able to take care of that and since um it's grog it's very static so in the end it ended up containing 34 grog patterns for for

matching and the last update was in the swimmer on december 20th where somebody from the community added support for unicode unicode encoding so to me this is a really good example of why why this cloud-based approach works because basically we we create a signals single and then it takes i should i'll show you a little bit a little bit later how long time it takes and then we start getting data back and signals back of active active malevolent actors in this and that is uh to me i find that pretty amazing because that is that is really what the community can do when you work together so as you can see or brought up this is a timeline of

from starting from december 9th and ending on december i don't know 16 17 something like that no sort of anyway the big spike is when um is on the december 12th where we start getting signals we end up we end up at that point with around the thousand ips that we knew were actively exploiting this but as we looked at the data we saw something strange in the sense that that one one ip was had a different scanning pattern and we looked them up and it turned out to be a german security resource institute that were they were basically trying to find out how big how big is this problem by by scanning and we decided not to block them because

we would then mess with their research and also given that it's not it's not it's not a risk so um so they will fill it out this is um an overview of of the signals we have received from love4j until now the big dive is is on the may 18th um approximately three months ago we still get we we got around 100 100 signals every day but today we get around 40 to 50. so love for day is not dead rumors are greatly exaggerated um if you look carefully there are a little bit of spikes after um may 18th but two biggest one they are quite interesting so i'll talk a little bit about them

um we're not really sure what happened because as you may know we don't really collect much about what's going on we just know that we don't know exactly what which payload or whatever bad actors are doing we just know that the scenario was triggered so based on that is relatively impossible to find out or try and find out what they were doing but we know who did it on june 21st our data scientist tweeted this and that that we saw a big spike and that was actually the small spike of the big ones uh and the ips 13.89.48.118 there still they're still around doing their business with love for day um so you can see the graph the two the two

spikes there the first bike is on um it was under through well what was on the june 21st and the second wasn't it was a on july 7th and that was double as much and now i need some more water we don't know why as i said but we do have a theory at least for june for june 21st because as it turned out on june 20th on june 23rd two days after the first spike cesar the cyber security infrastructure security agency in the us released a bulletin saying that the vmware horizon i will look for gay related vulnerability in vmware horizon was actively being being exploited so it is at least to us it makes sense that

this may this this may have been what they were scanning for they just knew a little bit in advance um and as as i said before in terms of the love for gay being everywhere it's also highly unlikely that that we've seen the rest of this i'm pretty sure that there are still our love for gay things out there so love for days definitely not dead

and i guess when you think about it it makes sense that we can also see other interesting things in the data that we see that that we receive but to but to put things a little bit into perspective um this is a number it is an overview of the signals that that that we that we are receiving for the last three months in average we have we've had 29 millions a month um broken down into weeks and hours um we have around 57 000 agents in 168 countries right now and it's it's crazy to to think that that the number of number of countries just keep rising i mean a month ago two months ago it was

150 and i was thinking well now that's it but it's not um we get around twenty thousand ip addresses shared every two hours there's a two percent renewal rate every 12 hours which brings about 400 new ips every 12 hours and 12 of these ips are are then seen for the first time so there is a large number of ips or there's a big change that they're being changed shifted there's a big shift percentage of the there's a big percentage of the ips that have been shifted in the beginning whereas over time uh it's it's a it's a bit more static um there's no view of that here less less than one week 12.63 percent but

over time um two percent of uh two percent uh 2.79 of those ip over time i knew so the the the part of of um of ips that i knew will are decreasing over time and one thing that that that that we realized in the beginning of the crosstalk project project is that the sparse resource on on the internet or the scarce resources on the internet that's ip numbers so basically the more difficult we can make it for the bad guys to have new ips because they knew they need the ips for anonymity so basically they just hang around and try to find ips that they can use for the for whatever they're doing the more difficult

we can make it for them the faster we can burn their ips the better and that is basically i woul

BSides LV 2022 - Wednesday - Breaking Ground Track

Related talks