BSidesCharm 2022 - Threats lurking beneath the subsurface (WSL) - Danny Adamitis

all right so thank you everyone for coming my name is danny ademitas and i'm going to talk about the windows subsystem for linux i also want to give a special thanks to my partner steve rudd who helped me with this research but unfortunately could not be here today to present with me so we're going to do a quick overview we're going to talk about what is wsl some of the samples we've observed how to detect that and then a couple closing thoughts because we all love doing that stuff so for those who don't know me my name is danielle midas i'm currently a principal information security engineer at luma technologies and i focus obviously anthrax to telecommunications

and isp networks um i've been doing this for a couple years now and you know my spare time if you see me afterwards i love talking about cooking running and my of course my dog cookie who is the better half so for those of you who don't know just real quick um i work for lumen technologies this was a rebranding of centurylink uh level three quest service um we do cdn dns resolvers um a whole bunch of things so basically what i want you to take away is we just have tons and tons of linux servers and embedded systems everywhere so one of my jobs is to try to find linux based malware that's obviously

going to impact some of our systems and that will come in handy in a second and then as part of the luma technologies we have our own special threat intelligence division that we call black louis labs so we have wonderful stickers and t-shirts if you want to come see me afterwards you can get one and it's all cool so real quick we're going to talk about what is wsl or the windows subsystem for linux uh as you guys might have heard this is finally going to be the year of the links desktop almost um so for those who don't know wsl allows for almost a near native running of the actual linux system on your

windows machine um version one was actually first launched in april of 2016 and it was really good but it was still a little bit buggy but then they relaunched it in 2021 with the version two and it was much smoother it's very good it's awesome um and this allows you to run a full linux kernel um you can actually install this either through a command line with wsl install with some conditions supplying such as admin privileges you have to power cycle device and do a couple other things or you can download through the app store um and this allows you to run things like ubuntu or debian cali alpine basically whatever your flav your favorite

linux flavor is they probably have that and then again this kind of allows you to run all of those native linux terminal commands you guys all love so if you like doing you know apt get install package you can get through that you can mount file systems um it just kind of allows you to do some of those tasks a little bit easier when you're like a security engineer or administrator via that linux command line but still using your windows system so you don't have to have two laptops and again this is actually from the windows website about why you should actually use wsl um it's really good for things like version control on get so if

you're doing software development or if you're a programmer it kind of makes your life a little bit easier you can do cross-platform development with visual studio everyone loves visual studio and makes life easier now you can code your linux projects using that you can set up docker containers that's the misspelling i should have caught that um you can also set up and connect to things like databases um you can even connect um usb drives and you can set up lux containers and again this is just all taken from the windows website so once i saw that monthly list i was like this is literally everything you would ever want us if you're a red teamer you can find the guy who's

actually if i go back the person who's running diversion control of your get version that seems like a great box to be on someone who's continuing your docker containers connecting and setting up all of your databases this seems like that box you'd want to be on to give you the credentials to all of those other databases that you would be interested in in the future so when i saw this i was like this must be a huge vulnerability because or not vulnerability a risk because if you were gaining access to this particular system it could allow you access to all these other more sensitive systems start deeper within the network so that's why i kind of took a keen

interest in this because i was like well again this could kind of lead to other things it also allows you to read files you can set up things like lux containers you can interact with removable media this is just a whole bunch of bad things and then the other really interesting aspect i want to talk about today is you can run an l file on the windows machine using the windows subsystem for linux and then as i started moving into this and i kind of talked to some friends who worked for edr firms there might be a couple people here they kind of adopted this approach where they're gonna treat l-files like disney characters treat

family members named bruno we don't talk about bruno until it's about two hours too late and the house is literally crumbling down around us so i would like to take the next 20 minutes to talk to you about bruno set with your house does not come crumbling down around you so this kind of started off with again my routine job of just looking for linux based malware they could be affecting our systems and we kind of create these things i call generic militias so it just looks for like weird heuristics weird file names and one day i came across this lovely file called bypass.elf and i was like wow what are you trying to bypass exactly this seems really

interesting it was originally designed for a dubbing system it was compiled in python and it had a detection rate of zero and i was like congratulations guys you did it you bypassed edr i think you named it bypass so they were so close just so so close so for those of you who have been around you guys might know this um but if you're not um python compiled files are actually super fun and super easy particularly if you're kind of a newer reverse engineer there's a number of really cool projects already on things like github called pi instructor um this will allow you to actually take that compiled l file or compile pe file and

turn it back into python by code so this is kind of that first step where you kind of like are unpacking it and there's gonna be a second step where you can turn the bytecode back into actual human readable python so again this is another kind of fun thing you guys if you come across it's fun it's easy i thought this was actually a little bit more funding some of the other stuff written and go so when i start running this um this is just kind of like my i guess my story i was running this before and i was doing this and i was having a whole bunch of trouble one day when i found

some files and i of course did not read any of the error messages because while they read error messages it just seems to be working so let's just keep on going but then when i went back i saw that you need to run this file using python27 to prevent execution errors and i was like i wonder if that actually impacts me it did so then when you try to run uncompile so there's basically two things and i think i talked about the next slide but basically once you get your python by code you have to convert that back to real python um but i one of the packages that does that is called uncompile but

if you try to run that with the wrong version of python it will get mismatched and you'll get a whole bunch of error messages so if you guys are going to do this at home or as part of your day job let's check out the error messages and make sure our python versions match because as you can see this was python 3 and i was supposed to be using python 2 but when i actually did the right way it worked successfully and everything worked out great and then once we actually deconstruct that and we get back to the python bytecode there's two projects i want to highlight that i really like i did not write any of these so full credit to

those guys but there's one of them called 6 and the other one is decompiled 3. the difference is uncompiled 6 works for your python 2 7 up to python 3 6 versions if you have something written above python 36 i would recommend decompile a 3. the difference is that with the older variants there's actually a difference in the like the byte code of the headers so it again it works better for that one because it's i think a three four three 32-bit works for the older bearings and four 32-bit works for the newer ones um so again just one those little increases that took me about a day so if this can save you a couple

hours then it was well worth it so now that we know how to analyze this i was going to talk about my lovely bypass file so again i started kind of going through this process i was able to kind of get this back to the bytecode and then i got back to python and then this is what i was looking at um it's a little bit blurry but for those of you who are kind of used to the information security stuff you'll notice that this looks a heck of a lot like like powershell and then i got really confused because i'm like i thought i was looking at l files why is your powershell in my l file

and that's when you can kind of start seeing so many things like something went wrong you can't kill av antivirus detected windows persistence time to persist and that's when i was just like oh that's what they're doing this is actually one of those wsl samples that i thought was going to kind of be out there but we didn't really actually find before but when i started looking at this more i'm like if you look at the top part they have a whole bunch of things like windows update they're trying to inject into their strings called can't kill av there's things called windows persistent with the actual run key right in it my thought was okay well this is still just

running power shell i'm totally fine our edr vendors got us and then i looked at the detection rate and it was zero and of course my first thought was what the duck um once again cookie and her lovely duck so that's when i was just kind of like all right well this isn't good we kind of found some of our first um samples and we were just kind of going through there and that's when we actually did a report back in the fall of 2021 but after we kind of did that we were like okay well this was something that was interesting but we're not quite sure what to make of it um so we're going to

kind of take some of those really unique rules we wrote before we're going to kind of why do fuzzy rules so the idea is we're going to kind of open it up a little bit we might take some more false positives but could help us find other activity that's similar to that and then we kind of just let them go for the next couple months because it was thanksgiving it was christmas everyone likes to have time off you know all that fun stuff so after we kept them running we went back to this and i want to say it's january or february-ish sometime around there and we actually found a couple more really interesting samples so we're

going to talk about some of those now uh there was one that i thought was interesting was the key logger we actually found something that was doing show code injection we found the cross-compatible agent that was actually written for alpha like actual linux files and for windows files i thought was really kind of weird i wasn't quite sure what to make of that one and then there was more of like a simple file so i'm gonna start off with the keylogger this was something that i thought was actually kind of interesting because it had like real c2 comps to it it would actually take all of the keyboard or the keyboard input it would put it to a file called logfile.txt and

then it used hard-coded credentials to actually communicate with a gmail address there was a hard-coded password as well which we did not publish for intellectual reasons um and then would actually modify the registry as well so this thing would stay on your system for you know as long as it would be there until someone detected it but this was just one of those things where um it just kind of showed that like this looks to me it looked when i saw this it looked like people were actually starting to use this in the wild and that this is kind of moving off that theoretical realm and into the actual real world realm this was the other one that i really

found to be the most interesting personally um and this is actually the python by code right or the python code to the right for those of you who aren't great python would basically open up a url lib and it would grab a resource um in this case it was a 127 ip address it would then take the size of the actual show code it would allocate memory in the thread and it would take that shell code and put that right into a running thread so anyone who works for a red team knows that that's a great way to start loading some of those more advanced tool kicks so when i saw this again if you're just allocating shell

code into a running thread it'd be very easy to basically put something like a cobalt strike resource on that remote ip address or meterpreter or basically anything you want and that's the thing that kind of got me because once you have something it's a running process in memory it becomes a lot harder to actually detect that than if it was written to disk so again this is one of those things where and of course it had detection rate of zero because whoever looks at l files on windows systems this was the other one that i really don't know what the heck this was if this was somebody's computer science project or if this was somebody who was

just got bored but it was this thing called lee's agent and again it was literally called agent.pi which makes life a lot easier for me um but the interesting thing about this is that it actually would modify linux things so there was file paths for the i will start and then there was also file pass for the current registry um this one was actually kind of interesting too because it looked to be the most i was almost professionally developed it was obviously python so it kind of felt like more proof of concept but there was like allocated commands for upload file download file remove this directory copy this file take a screenshot and it kind of had more of

that functionality to expect for something that was actually being developed professionally and concurrently and not just this quick oh i'm just going to use this to download my cobalt strike and use the same tools that we've always been using and then this was kind of the one where this was just to me felt like a very classical loader so this would again reach out to a remote ip address it would then download the file as an actual python file or as a pi file i'm assuming that was to evade some sort of network based detection and then once it actually got onto the system the script would then rewrite that as an actual executable it would then decrypt that

with a hard-coded key and then it would basically copy that to the startup folder for persistence so again anytime you reboot your system if it was in startup file your operating system automatically kicks that off and again they will just use that and again this will just be able to be used to then develop that wherever it was downloaded so again that could be a cobalt strike it could be meta interpreter it could be a custom payload or it could be something else but that was just something where again when you're downloading files you're decrypting them with the key and you're like changing file sanctions it just causes havoc for all of your like network-based

detections um so those were like more custom samples we also found what looked like open source tools so again these are all on github anyone can go look at these there was like a thing called discord rat that again provides all the standard functionality that you guys were known and like know and love and of course this communicates via discord api there's a second one called token grabber so again it was kind of similar to that and it would basically grab all of your authentication tokens for things like chrome mozilla ie brave and again once you have those authentication tokens or authorization tokens you can basically just authenticate asset end user the other thing is there is also one

that kind of picked my interest called telegram so again it was similar to the discord rat um obviously the main difference is it communicates via the telegram api and to me that kind of indicates that there was more of a broader interest so again if you were hypothetically a u.s company and you saw something like discord you would think oh explain it for our younger employees who's on discord talking to his friends but if you saw telegram it would kind of stand out and say well why are we using telegram that may alert you however if you have a company in some place like ukraine everyone there uses telegram so again that would blend in with the

traffic because everyone has telegram installed on their desktop so again it's kind of a way for them to kind of choose the project that fix the target environment that they're going after and again with these things being on github there is no development time there is no development cost you get cloned maybe modify a couple things for an hour and even without any modifications at all the telegram rat had a detection rate of zero so again you can do this with weight literally no bug bugshit at all and then there was the last thing that kind of sell to me was this credential harvester so this was again kind of based off a public forum website and it

was again trying to basically take the contents of your chrome login database it would then decrypt them and see all that so if you guys have ever gone and used google chrome on your laptop maybe you log into facebook maybe it's your bank maybe it's your gym whatever it is you always get that little pop-up in the corner hey do you want me to remember your password so you don't have to log in next time this would basically grab all of those safe passwords and as you guys know once you have a list of known passwords for a target you can then kind of use that for other accounts because maybe they use the same password for

their work email as their personal email or maybe they use the same password for other things and again this thing didn't have any sort of actual detection rate to it the one kind of unique aspect about this one is that it didn't actually have any sort of command and control channel associated with it so to me this would indicate that someone was already on your machine they already had a show they would download this they would run it they would get the output and then probably delete that and again the thing is because excel files it's kind of hard to sometimes detect this because you don't have all those sysmont tools so for example if you guys are familiar

with sysmond there's a file called save pe upon delete so again if someone were to download a file run it and then delete it and that configuration was set to one it would then actually take a copy of that pe file and put in an archive so you guys can view it later it doesn't have an equivalent for l files on windows systems so again that's kind of a way to sort of get around some of this stuff but as you can see it kind of looked like it was based off the open source one but it was a little bit more narrowly focused on chrome and it didn't have any sort of external channel

so it kind of presenting some of the stuff and then you know of course i was really excited about this i kind of talked to my boss and i was like look at how cool this is this is great and then of course i kind of got the question from her about well this is cool but it's all like python crap like why why are we caring about python crap and then next thing is like we're not talking about campaign we're talking about what looks like a proof of concept you know why are we talking about this um and again i kind of acknowledged that you know some of this stuff still looks like it might

be in development but the truth of the matter is it's still effective it's still able to do its job and it to me almost kind of feels like powershell in 2017 if people remember that before i got hooked by amsi you can type a really simple script in powershell back then it could be 40 lines you would have great functionality and you can get away with using it for 8 10 months before anyone caught again if you can develop a rat in python it's a very easy programming language it's very friendly people can do it without really a whole lot of professional skill and they can still write something that allows them to communicate with an

external api or with external c2 and have a detection rate of zero and if you make that external thing something like discord or telegram well now it's not even going to like a dedicated say domain or a c2 ip address it becomes really hard to try to find that noise or try to find that signal in all the noise the other thing is this is kind of my personal you know circle of life or circle of threat intellections life if you will um so if any of you guys work for an enterprise company you know that typically you start off with monday of you get some sort of message from a boss saying hey you know we saw this report

the other day and i was just wondering if you guys are detecting this brand new charming kitten whatever and then of course that then drives everyone to start writing detection rules for that as they write more detection rules more data gets collected and then as that data gets collected the thing leaks to more reporting which then leads to more questions from all those other people so again this is kind of my thing is by me trying to talk about this in these forms i'm hoping that you know that will then lead to more people starting to write more detection rules they have to have spur cycle to then collect more data to then help do more reporting so

we can kind of keep going this going because i feel like this is a very viable attack surface so monitoring and detection again kind of best practices only install wsl on devices to actually use this so again if you are a kubernetes system administrator and you really need this yeah by all means use it it's a great tool it might make your life easier if you are working in the hr department and you don't know what the terminal is maybe those people don't actually need to have it and then we can kind of reduce the attack surface again because it's kind of running some of this linux stuff you can run things like ip tables you can block outbound

connections so if you're only administering internal databases you don't need to have an external internet connection you can kind of create some of those firewall rules to help do things um the other thing is group policy i was actually looking into that this morning i think there might be ways that you can do that it did it does on the new version okay cool i think it was actually they have like their own like commands like the windows sub stack command or windows yeah so they were talking about how they're integrating it now so i think that's one of the like new features but it's also really cool uh and then like that so i

think there's group policies where you can kind of limit what can and can't be executed um i was trying to kind of play around with that swearing and i don't know the exact configuration but i think if you guys are monitoring that's something i would kind of look into a little bit more um and then the last thing is like previous talk make sure you have proper logging make sure you're saving off all those commands um and then like i said just make sure you're doing that because there's not a lot of pre-built tools so as we mentioned before there was copy upon delete they don't exist for elf so you kind of need to really rely on those other tools to

help things work all right and lastly some parting thoughts um so wsl is not going to be as ubiquitous as things like log4j blog4j was everywhere and anywhere in effect that basically everything from vmware to home routers to anything else um this does not have that same footprint so the thing is it's never going to be that large splashy thing it's going to be on the front of the google news however this is still a very serious threat for corporate networks the reason i think it's a serious threat is because it has such low detection rate and if you couple that with something like a network api to telegram or discord you're not really able to detect down

the host you can't really detect it from dns you can't really detect some of that stuff from the firewalls it kind of allows them to live in that dead nebulous zone so where they can operate and still do a lot of damage the other thing is again because the systems are typically using this are those power users i think dex also makes this a much more attractive target to people who have wsl pre-installed on their corporate environments um as i kind of mentioned before we detected a couple of these samples all the ones i've seen so far were written in python i acknowledge there's a bias because i kind of wrote my ur rules to look for

other python stuff but there could be other variants in things like c c plus pause go pick your favorite programming language and again there might be more stuff out there we're still trying to look for that but again you're trying to find that needle in haystack um again it's used by power users and again the last thing is by hoping to inform all of you guys you guys might find something in your environment um and again i would hope and encourage you guys to kind of report that either through virus solo through twitter whatever mechanisms you have to kind of share some of those samples that way we can all do a better job of defending together

keeping up with us so i work for blackwell's labs we did a blog about we did two blogs on this so if you guys are interested we have a whole bunch of these resources whole bunch of screenshots um you guys can just go there and again you guys don't have to go through grainy screenshots um if you have questions i'm danielle midas that's my twitter handle i'll also be here for a little bit for the happy hour feel free to come say hi get some free stickers um and again there's also a wsl blog on microsoft um so that's actually been really cool microsoft has their own thing where they do it like once every

quarter and they talk about the brand new features that they're implementing um i kind of think it's interesting if you guys are actually doing system admin work it's nice to know like what they're working on and how it might make your life easier um i would encourage you to check that one out as well um and again this is just some of the references if you guys are interested i saw some screenshots this is just where all that was um and then the last thing is just questions so i think we have like three-ish minutes five-ish mix somewhere around there oh a couple more all right yeah what's up um

so it's picked up i think stinks to blog but it's still pretty low i mean it was like 10 or 12 where you know if you look at your standard stuff it's typically around like 30 or 40 um so it's it's getting better but it's still not where i would like it to be personally

so it's similar to like that powershell um and there's actually if you go to i think the second one if you go back to the second block we actually link to a big white paper that talks about this but basically there is uh windows event logging and again it looks very similar to like the powershell blogging again except you see like the bash stuff so again you can probably just grab for bash and pull out all those um you can also do sysmon logging if you're more familiar or if you like that more again obviously throw that to some sort of central repository you can kind of try to recreate what exactly it was that

they're doing and then the other nice thing is if you do happen to find a compromised system and they were doing this you may get lucky and you can just view the bash history which is just another lovely thing if you guys have ever done instant response uh that guy i'm sorry i ain't here

so i just i don't know the implications for um cpu utilization if it will be if it makes sense or if it will kill your box um so again i don't want to say one way but potentially that would be more secure if you are more in line of security that would probably work but again i don't want to tell you something and have it break your system either cool what's up um i don't think we do but again so we don't share the er rules per se but we do share all the samples so if you want to go on our public blackboard labs github you can see all the same files i think most of them are on mileshare and

on viran and on dyrsol um so if you're curious you can just kind of take those hashes download them for free and you can kind of play with this afterwards

as good as i would personally like um but basically what we were trying to do was look for um like i guess if you want you can try find me afterwards i'll show them to you um but basically it was like we're looking for windows systems that we're running things like debian which we don't really see a whole lot since most people you're using like ubuntu or like the routers and embedding systems they're using mips uh we're looking for things like file size um and then i think there's like one or two other features i think we're trying to look for like um imported libraries like c types if you see that that's kind of a

dead giveaway um but again i will acknowledge there a little bit they're still really fuzzy um so again it takes more time to kind of download everything and then i just run a script to just unpack everything and then just kind of look at that it's still sort of manual at this point but it's the best bag solution if that makes sense

yeah you still have to do so when you basically um when you do like python like pi 2 exe or something um it actually kind of converts the python the byte code and then it zips up that pipe that by code and that almost acts like an obvious figure if you will but the nice thing is because it's like a known obfuscation method um it's easier to download that but the problem is again it does take more time and resources yep anyone else no okay cool i guess we can go at the borrowing [Applause] you