Log in Your Own Eye - Exploiting a Stealthy C2 Channel in Azure Logging Infrastructure

our next speaker will be dimitri berioza dmitry is a senior security researcher with vector ai working on threat detection in the cloud and on-prem's network and today he is going to be talking to us about log in your own eye exploiting a stealthy serial network in azure login infrastructure take it away dimitri all right all right thanks a lot hello everyone thanks for coming uh thanks um so i'd like to second what the previous presenters uh said the the presentation right before this one it's awesome to be back at conferences and let's hope that uh covet soon soon becomes uh distant unpleasant memory so um um here are my handles on discord and twitter and

my linkedin information in case you want to sync up later i do security research at vectra ai and as a company we do thread detection in the cloud and in on-prem networks using ai and machine learning before that i was a pen tester and a secure software development advocate with ibm security and before i made a switch to security i was a developer for many many years i'm passionate about all things security um but in particular about reverse engineering secure software development and i just love ctf competitions i live in ottawa canada so uh hello from your friendly neighbor to the north so what are we going to talk about so i'll start with um going to brief some brief

notes on azure and azure log analytics we'll talk about custom which is a query language in log analytics and about how it allows you to query external data and then we'll jump into how we can actually build a covered channel writing on that and i'll do a quick demo of a proof of concept code that i built then i'll touch on some mitigation strategies and we'll wrap up so i'm sure you you guys know about azure it's one of the big three cloud providers it's massive been available since since 2010 and as of now it has over 200 services so it's just gigantic and it keeps growing just like the other cloud providers and of course these days security is one

of their focal points they pay a lot of attention to it they have a lot of security specific services and i would say as far as cloud providers in general my in my estimation promise of better security in the cloud does hold uh but you have to be careful because when we talk about security in the cloud there is something called uh shared responsibility mode model and that means that depending on the type of service that you purchase from the cloud provider responsibility for securing it either either split between you and the provider or it's it rests fully on the provider or it rests on your shoulders as a customer and that's where i feel most of the breaches actually are

happening it's where our responsibility for securing things is on the customer as far as infrastructure infrastructure is concerned like from what we see in the news there are not that many breaches that are happening in the core cloud infrastructure or they're happening and we don't hear about them they're fixed quietly uh there's a stark contrast with how many vulnerabilities are reported for uh software that's on-prem but the most of the problems occur when customers don't quite secure the functionality that's in their area of responsibility properly azure log analytics is a log management system in azure used by many other services such as monitor and sentinel and it's used to ingest and process logs from azure itself from

on-prem environments and third-party logs so it's really universal in that way it has a variety of connectors you can connect can consume log data from a variety of sources it's used for by nature it's used for log analysis but in particular for security log analysis as security analysts the logs are really our only window into what's going on in our cloud environments so using logs is just invaluable for threat monitoring and incident response so log analytics has something called custom it's a query language it's a sql-like query language i'm not sure why they didn't choose to implement a sql a language with the sql syntax but it resembles sql if you look at this example here

you will see that it's sort of reminds you of sql there's a there's a where clause you can summarize things you can group things there are all kinds of operators and functions you could include so if you know sql learning to use custo is very easy i've used it a lot in the past few years and it's it's great the syntax is powerful you can do a lot of different things you can you know do things with multiple tables and logs are presented as tables in this environment so essentially you're building data queries over these different log sources uh you could join you could intersect you could do all kinds of stuff and another important feature of it is that it's

really fast i'm not sure how they do it exactly i think it's just massively indexed on the back end but your queries are very quick and that's very important in incident response because you can almost kind of write queries and execute them at the speed of thought you don't have to kind of write a query and then wait for it to complete so that's really important when you're trying to track down a specific incident uh breach of some kind and these queries are used uh in different areas of azure so azure sentinel for example uses makes you use of some built-in cousteau crease some queries that community provides uh they're crowdsourced and customers can also

write their own hunting threat hunting queries plug them into azure sentinel run them on a periodic basis so it's a very good tool for trying to find malicious behavior in your environment the interesting feature of custom is ability to reach out to external data sources so in this example here there's an operator that's called external data and it allows consumption of web data sources as tables in your environment so in this example here we are basically loading a csv file that's hosted on github and it's presented as a table in custo and you could then join it with other tables to produce interesting behavior so for example you could do a you can consume a list of tor exit nodes

and join that with your table that tracks sign ins in your environment and you could have a query that would alert you if there is a sign in to your cloud environment from a tor exit node so you could do some pretty cool stuff with it this operator supports http get requests only and it can ingest data in many many different data sources we primarily use it with csvs but it could be text it could be an html page data sources could be compressed so it's it's really powerful so as you as you noticed you just need to provide the url for it to work and this was a concern to me um so why is

that this is not a bug but i think it could be exploited for malicious purposes and i'll explain as we go along so malicious user could build a command and control channel writing on this functionality so you could have commands transmitted over it malware downloaded you could exfiltrate data just using get requests and you know in some cases it's it's not a big deal because with uh you know with zero trust uh infrastructure you have lots of remote users that are not only connecting to your cloud environment they have access to to all kinds of other websites so for those cases that is not really a big deal because we've seen malware that's writing on all kinds

of interesting third-party http channels like the malware command and control centers could be hosted over facebook over twitter they could be on google drive there's you know sky's the limit so it's not as interesting in those environments but i think it becomes interesting when you're trying to protect an on-prem network and you're trying to regiment what what data sources what what what external websites you allow your users to reach out to us i think that's that's when it becomes important so what's what's the big problem here so first of all if someone were to use this functionality maliciously they would be using legitimate azure infrastructure so if you have firewalls in place you have endpoint

monitoring tools to all of them it would look like benign use of azure functionality and probably would not be raising any alarms even more than that even if you decided to somehow try and monitor that what do you do you would have to open up encrypted channels and try to filter by specific api requests which is which is not a trivial task so it's it's not easy to find something like that on the back end so let's say you used it to tunnel through azure infrastructure and on the exit end you were let's say reaching out to a particular command control server in that infrastructure it would look like request is coming from inside azure proper from their core

infrastructure so also not very suspicious and last but definitely not least this functionality log analytics is usually used for for threat hunting for incident response so it's a security tool so to me it feels like you know there were these recent incidents where malware was using antivirus update channels to download second stage of malware so they're essentially we're using antivirus infrastructure for malicious purposes to so to me that had kind of uh feel like that that you this is a security infrastructure security service and yet there's a possibility that it could be abused so what's what's the scenario here so let's say user is compromised let's say clicks on some attachment malicious email and malware executes on an endpoint

and that user has access to log analytics malware could then execute greece through this functionality continuously to reach out to that command and control server and once the connection is established data could be exfiltrated via http urls and commands and second stage exploits could write down on that channel back to the uh to this exploited endpoint so there's some considerations in play here so first of all victim needs to have a log analytics reader role in azure but that role is not really a high level role it many people have it because you're not really changing anything you're you're reading logs and you don't really need access to any log data source in particular because you're

not using any any log data source requests will not work with ip addresses so hosting your command control server on the just anonymous ip would not work you need a registered domain but you know there were many cases where legitimate domains were taken over and used for malicious purposes so that's not not a big deal responses from those get requests they could be pretty big like in my tests i i could download megabytes of data and from those data sources and that wasn't a problem but data coming out by nature of get requests very limited the url is about 200 bytes in size so you really uh it's hard to exfiltrate something significant over the channel you have to have a lot of

different queries in order to push through i don't know the word document for example but you could use it just for critical data so let's say you're trying to exfiltrate passwords encryption keys certificates so for for little pieces of data like that that would be a valuable uh valuable tool microsoft applies rate limiting to these kinds of queries so one would have to be careful when executing many of them or executing them at a high rate so that's something to keep to be aware of and so with with communication being limited in volume and being victim initiated you have to reach out the command control server you have to have some kind of heartbeat setup so that you periodically ping the

server in order to establish a session and you uh because of size limitations you have to split your payload across multiple requests so i built a small proof of concept for this so there's a server in python it's hosted on aws and a powershell client that reaches out to it through this method client just runs in the loop it pings the server periodically and just to kind of play around i introduced some delays and jitters so that things would not be easily as easily detected as automated requests and requests are encoded as part of a url path just as a parameter in the url path so client pings the server asking for instructions and ships the data that was

let's say some private data that's found back to the server and responses come in the body of get get requests that come back and uh it could be a command to execute a file to download or recommend to exfiltrate something from the client i uh compress and encrypt communications with pre-shared key so that you could not easily look inside and uh maybe try to trigger detections based on that and i just uh to make it look as unsuspicious as possible i masquerade this as queries to some thread database so let's say imagine you have there's a database that you query with hashes of files that you find and it responds with information on those files whether

they're malicious or not so here's some simple uh communication kind of unpacked so let's say server is awaiting request it receives a ping request from the client client reaches out um asking for for instructions and the server responds with the following command so they the server the supposedly there's let's say there's a malware operator sitting on that server and they say they want to execute who am i command i mentioned before that you know because of size limitations you have because of the need to maintain a session you need all kinds of housekeeping data in the request as well so there will be you'll need some counters for chunk sizes chunk numbers so on so forth so

there's some housekeeping data added to that command and then you can uh encrypt it and and compress it and it'll be shipped uh in the body of the get response to the malware on the client side and once the command is executed and you get request sent to the server with there is with the result so in that case it's my username as a result of who am i command and then we compress and chunk uh the the data as necessary and base64 encoded because it has to you know encode it like that in order to be a part of the url so at the bottom you would see uh how it looks like when it goes back

to the server so let's see okay all right that's encouraging so let's start the demo so let's this is a demo how this actually looks in real life so on the left you have a server um you know in the cloud in aws and the victim machine running the powershell come on you can do it all right right so the server is starting down the client you can tell again type verbal so you'll see a little heart symbol blinking there it means that the communication is established and the communication goes through azure through log analytics through that channel that's established there so it's sending heartbeat requests requesting commands and operator on the server asks for to execute a who am i command and

that's what the client does and that's the response so it's for small requests it's it's pretty fast then uh we'll try to exfiltrate a file so on windows there's a small text file that's out there on every windows machine it's very easy to find and it's very small so it's good for for testing so you could either type uh use type command to dump its contents or this i implemented a um an upload command to actually uh steal binary files from the client if necessary there's just uh you have to be mindful of how long that takes because of the 200 by 2000 byte limitation in the ui you need many many requests to transfer a large file

and uh and finally uh we'll download malware from the server to the client let's just give it a second so we say we need another dll that contains some other malicious stuff and because get responses these sizes pretty much well it's not unlimited but it's it's uh you can transfer significant chunks of information the download to the client goes very very quickly so this is how it looks in action all right so let me see

let's go to the next slide come on all right so uh you may be uh asking why implement uh c2 channel from scratch uh it would be a valid question to ask uh there are a lot of c2 frameworks out there and i actually tried to implement this with cobble strike with a malleable profile uh most functionality is available in cobblestrike but there are some and are there are some limitations that could be bypassed creatively but unfortunately some other limitations they ended up being roadblocks it imposes restrictions on your right size on header size it needs multiple your eyes to actually implement a proper beacon so with cobblestrike out of the box it didn't quite work however

with some code modifications i don't see why you couldn't have a cobble strike beacon or some other c2 framework beacon implemented based on this or intraday integrate this with metasploit on any or any other frameworks so uh as a defender how would you address something like this so first of all for this to work you need your victim needs to have access to log analytics they have to have either log analytics contributor or log analytics reader role and not everybody should have those roles as a matter of fact even ignoring this scenario your logs are just treasure troves of all kinds of interesting information they're just very valuable for recon if somebody wanted to learn about your environment

sometimes their keys or passwords dumped into those logs so in general you shouldn't give this role to just anybody it should be a restricted set of people to get this role and azure fairly recently introduced a way to monitor your log analytics queries there's actually a separate log to to log those queries that's kind of kind of recursive so the query you run on the logs gets added to the log but there's this log where you could see what somebody is is doing in your log analytics environment what queries are being run so maybe you could look for some patterns in that table i thought that you know there's no reason why this function should be wide open so i actually

propose to azure that maybe we should restrict this a little bit and just give sort of a white list of functionality to administrators where you only define the websites that you really need to reach out to using this external data operator and everything else would be prohibited so microsoft looked at it and said uh no we don't want to do that so okay that's fine they weren't as impressed um but i think the uh the issue here it goes beyond this particular issue cloud offerings are just growing at amazing pace so azure is over 200 services aws over 200 services gcp over 100 services so like almost like weekly or monthly something new is being added and you can you can bet that

in other services in azure and elsewhere you could have similar functionality unrestricted http requests going to wherever wherever you want basically and you could have uh malicious functionality writing on that so i think it's just something for cloud providers to keep in mind maybe uh maybe do something about it there are all kinds of you know database services big data analytics automation you name it that could have similar functionality that could be abused for hidden c2 channels so it's something i i think should be included in cloud providers threat modeling and same thing for their customers so to wrap up um so we wish we saw how some cloud functionality even though it's behavior by design it's

not a bug could be abused for for stealthy communications i think it's uh as i said in one of the slides it's a threat not only because of the stealth it's hard to monitor and also its azure apis they have an error of legitimacy to them so somebody using the api for malicious purposes is not a red flag so it's something that that easy to overlook and also this functionality is specifically used for security for threat hunting for incident response so for it to be abused in this way is sort of unexpected this is not where you would look for your for your threat so that that adds some additional gravity to this um there are ways to restrict this

particularly by not giving away a log analytics reader roles to just anybody i think it should be restricted group and our queries and i think there there are similar opportunities for abuse elsewhere i think it's uh it's something to to think about do we do cloud providers basically have these opportunities for covered channels tunneling through them that are hard to monitor and that's the end of my talk and if there are any questions i'll be happy to answer them and thank you guys for listening [Applause]

uh i wasn't sure it would be interesting enough but since you you're asking yeah i'll the slides will be here and i'll add a link to the code there as well in a few days give me a few days

well i'll hang around in case somebody wants to talk thank you guys [Applause]