All Aboard the Supply Chain!

all right here hey everyone hello hello how are you cool cool that's good to hear hey this is awesome thank you so much uh I'm just honored and flattered to be here with you all this is a very very cool opportunity and I hope we'll have a little bit of fun I try to keep things cool and casual and candid uh so hopefully we'll get into some fun stuff but uh what I wanted to bring to you today was well hey first of all I got to apologize look I don't have any cool Elite sexy zero days or new hot exploits to pull off the shelf or anything that I don't know bring a whole lot of technical

caliber but hopefully we're still gonna get a little bit nerdy we're still gonna get geeky still look into some code and still have some tech aspect to it but at the end of the day I do want to tell you a story and I hope there are components of it and pieces of it at least that might resonate with you a little bit but um hey without all that boilerplate let's kind of dive in I wanted to bring to you this thing that says all aboard the supply chain vulnerabilities and exploits throughout the proverbial Pipeline and I'm going to be chatting a little bit about the supply chain which I realize is super duper vague it's extremely broad

it's extremely General and it's another one of those terms that I feel like we just kind of throw out there and like cast into the wind and sometimes it's really hard to capture what do you actually mean what what what kind of supply chain in what way let me sort of boil and bake it down I think you could have a very very technical concept of supply chain of like hey some CI CD continuous integration continuous deployment blah blah blah pipeline of developing software and code and Tech or even like people of the supply chain like hey you receive some Goods or some service from someone at another party that receives that same good from

something else Upstream so we'll play with that anyway uh look before we get into the weeds please let me say I can't say it enough thank you uh RJ had reached out to me a bit ago said hey do you want to come party with us at the Cayman Islands I thought it was a phishing email um but we're here now and I I can't say enough thank you let me be super duper quick on the boring obligatory introduction the who am I slide establishing credibility I realize no one really cares so I'm going to speed run right through this um hey hello I'm John Hammond uh cut my teeth on some cyber security stuff when

I was with the United States Coast Guard um I also have been trying to grow a little bit of the cheesy YouTube channel to bring cyber security education to get awareness and training and messaging out there it's a lot of fun that is a uh labor of love and a real pet passion project but we're cruising another thing I don't know like half a million folks so it's surreal and very very fulfilling and I'm happy to have a whole lot of those cool opportunities with that again speed running here because I know this is boring and doesn't matter uh I got kind of hey my first gig as a teacher as an instructor with the United

States Department of Defense cyber Training Academy that was kind of fun hey you know got to be up on front on the screen got to do some hanging out by the podium but it is really a matter of keeping students awake and entertained sort of like riding a unicycle and just trying to juggle keep their pulse up so I thought you know what let me go do this for real I want to be on the keyboard I went to go be with the defense threat reduction agency that was a little bit of red tape and some slow moving spots there So eventually I bounced over to where I am now at Huntress which is I love and it's

incredible to be there but I want to tell you some of the fun stories that come from that because I hope I could at least bring a little bit more of hey the front lines or I I try to be in the trenches with everyone on this whole cyber security thing so anyway let's tell the story and I realize we're going to do a little bit of time travel so please bear with me we're going to move backwards in time as to hey when things got started and hey how this all happened and then what's kind of come of it Forward on but let me set the background here with a little bit of context

here's this news article here are these headlines hey here are the journalism reporters screaming and shouting about connect wise finishes rce bug that's remote code execution hey compromising a machine or an endpoint fixes this rce bug exposing thousands of servers two attacks uh I don't know how well you might be able to see the picture timeline here uh this is October of last year October 28th very back in 2022. and they say look it's the problem the weakness the vulnerability is an elements that are used in a downstream component okay there's our first inkling of a supply chain but we'll talk about a little bit more and let me kind of Center us here in October that'll be the

focal point of the story while we do some time travel in just a moment there are more dreaming and shouting and hey hey hey the sky is falling headlines and alerts and all this uh news articles that say Hey patch now yada yada dangerous rce bug lays upon connectwise server backup managers are folks familiar with connectwise I'm super sorry I realize that hey a vendor in the MSP space a managed service provider channel that is one of those spots that will offer services to other businesses and companies and organizations well oftentimes I.T Services hey we'll we'll manage your website we'll make sure your email is up and running we'll get the fire file transfer protocol and all the

stuff on those file shares that make sure that's all working they are one of those managed service providers helping other manage service providers provide for their businesses that they support whether that's it a dental office or a law firm or municipality or I don't know Hospital right it could be anything and that's maybe another inkling of that supply chain here because hey if there's a vulnerability in something that this company a at the top of the tree provides trickles down to B on one branch of the tree and then those potential Roots underneath it could be some damage down there so uh let me bring us backwards in time now let me sort of press rewind and get to

the start of this thing to talk about how this all kind of came to light and I'll introduce you if I may to a fella admittedly on Twitter right hey internet twitterverse uh fricos I know that's a little bit weird to kind of see that name there uh his name is Florian Florian Hauser uh researcher over at code white gmbh honestly think of this as hey Twitter guy keep it easy for your own thoughts here but frico says Hey can anyone get me in touch with the security contact I connect wise take note of that time stamp there that's July 2022. hey I'm trying to reach out I'm trying to get a hold of someone over at this

vendor I want to get a hold of connectwise but can anyone help me because I I'm not having any luck says that in July come to August he says hey look does anyone use cert or like I don't know some computer Emergency Response Team or any acronym right does anyone use these for vulnerability disclosure because of an unresponsive vendor I'm trying to get a hold of connect voice but I just can't get a hold of them I can't reach them they're not they're not getting back to me this is August bear in mind so okay let's think a little bit about the months here that are kind of component of this because now we're getting into September still not seeing

any Buzz still not getting any breakthrough at this point he just kind of kicks it over to the security Community hey fellow nerds other Geeks and hackers are you looking for a zero day like hey look what you want a new hot new vulnerability months ago I tried to contact connectwise for an auth Bypass or authentication bypass in this R1 soft server backend manager no dice couldn't get a hold of them have fun good luck good luck have fun now uh to kind of add some more color here an authentication bypass is like you try to log in to Amazon or you try to log into Google but say you didn't have to log in say you're a threat actor

you're an adversary you're a hacker and you just got to skip right on by you don't need to authenticate you don't need to log in I could open the door to some spooky scary stuff right especially if there's some information there or controls or power and functionality that a threat actor shouldn't have uh so so that's the authentication bypass and this is in September remember we came from July we went down to August now or September okay so months have been passing here and eventually I don't know how well you might be able to read this one but this is us catching it and kind of our internal Slack hey we're working with the team

doing what we do day to day and we have someone that says someone passed this along Someone caught wind of this on the twitterverse an Elon Musk little spot here they say look freikos had tweeted many months past so time for Pock time for the proof of concept here's how to blow this thing up here's the weapon here's the gun here's how you do this authentication bypass because what 90 days have passed it's you try to reach them in July maybe even earlier when he found this and he never got a hold of the vendor to be able to do responsible disclosure so he feels like well now what uh I'm just going to release this I'm just going to

bring it out to the world now these are just some crappy burp sweet screenshots uh forgive me I don't know how well you might be able to see because I know it's blurry super small text but for the fellow nerds and Geeks and hackers burp sweet is one of those web application penetration testing tools this tweet kind of Clues Us in it's like whoa wait a second you look you have the vulnerability you have an authentication bypass and a connect wise product I got to admit that's that's kind of a big deal connectwise is one of those top dogs if you aren't familiar in the MSP space or the managed service provider Channel think like a Microsoft think

like a big business that's running the show so if there's a vulnerability there there might be some stuff that could do some real damage so we start to chat about this we start to Banter uh an individual says look this is how the 2018 cassaya crypto Miner vulnerability and that little certain apocalypse Armageddon came through and then we start to think like look hey John another co-worker of mine a great peer Caleb look guys this might really be worth digging into because it could be so so good for education's sake for getting maybe more of a narrative and to show people the impact that cyber security might actually have and then we're thinking wait a second

can we pull down R1 soft can we play with this can we Tinker with it and where did that tweet go that Twitter post just vanished fell off the face of the internet it was deleted fry ghost our Twitter friend was asked to remove that by probably a vendor we hey connectwise thinking like let's let's clean that up let's remove that let's not show that out to the world here so that's a little bit of a bummer but thankfully slack had cached those images so we could kind of get a chance to poke and play with it anyway start to wonder here why are they trying to respond and connect and reach out to connect wise

when he said this is an R1 soft software he said this is a Backup Manager for another vendor and they say why would they want to report it to connect wise oh oh oh oh sorry okay turns out acquisition R1 soft change of hands now owned and eaten by connectless that's okay totally understood but maybe adds an arm to our supply chain story here because now you've got a product sold to vendors sold to other vendors sold the businesses that came from another vendor that no one works there anymore so this is kind of hard to dig into some of that technical cyber security sense and there's just another uh piece of the puzzle and if folks aren't familiar hey this is

what that server Backup Manager software looks like here's our login page hey so you were to enter your admin credentials your password and it is at its core a Backup Manager so all the other computers all the other hosts that need to have their data backed up would kind of be connecting and communicating and working with this manager the server and the service now once you log in what more can you do if you were an administrator if you were the actual maintainer of the software some curious and interesting things but we started to dig into okay what did Florian what did frycoast what are what did our Twitter friend find for this zero day in this

vulnerability based off of those burp Suite screenshots those are the only pieces of the puzzle that we had but we caught a little bit of an Inkling when we saw this advisory we saw a note we saw a report look there's a vulnerability in this ZK or ZK library and now you might upload a file hey so you wanted to give something to the server to the service but there's a problem there is a weakness and I don't want to drag you through the the mud here too much but they note that this is fixed this is patched they already solved this problem here it's been resolved in these versions 9.6 whatever and if you want to zoom in on maybe that

date here that's what may May of 2022 okay so a couple more months to add to our timeline and now let me get a little bit nerdy here we wanted to dig into it we wanted to see okay can we reach can we access that ZK library in that snippet of code because note this is where our technical aspect of the supply chain comes into play this product the software this application is its own thing it's built and it's made but one of the ingredients that it's baked with has that flaw has that vulnerability and weakness that's that ZK Library so we start to pry this thing open we try to think look is it matching that

weak vulnerable version number is it patched yet and it's not we're still seeing the same vulnerability we're still seeing that weakness in the version number and this is what it kind of looks like when we're Kicking It Up and then we start to dig into it again all we have are these burp Suite screenshots to kind of play with see okay what parameters are you passing to that service what cookie is there what headers are going to be in the mix here I start to sling some gross python code here for folks digging into that we start to see can we decompile parts of this application because it's Java and again not to get too nerdy but

that's something you can kind of take a look at those class files and find the original source code but we ran into a wall we couldn't get past this little roadblock there's like a little ZK error hey a little bit of a note here here's a header whoops it says 200 okay you're good but there's a 410 whatever code for an error message we couldn't recreate the proof of concept and the vulnerability like dang it I'm hitting the same problem that Caleb's hitting we're running up against the wall we're not seeing a whole lot of motion at this point I say uh uh okay well let me see can I just ask like hey I'll reach out I'll just genuinely out

go ask Florian frykos our Twitter friend and say hey uh I know you're doing what you do here I know you're busy fighting fires but if you'd be willing to chat a little bit about this R1 soft vulnerability if you would tell me what you're willing to and what you found here it could really really help us out with our partners because we have folks we support businesses and organizations that use connect wise I try to send in my code I try to send him what I've been tracking and then moments later we got it we say wait a second wait a second wait a second if we weren't using this session ID this token whatever secret

magic sauce value we wouldn't have been able to get past this parts but now we did said yeah I just stole the session ID from the browser session and then we say wait a second here's an ID value redacted here but it's just a zkid value it might come from the requests that's it boom fire check mark that's that's the success reaction in slack I send my code I say look at this we got it we got it I know I'm like a dog with my tail between my legs I'm going to go back to fryco's go back to Florian he says hey John thanks for reaching out big fan of what you do thank you uh and he says I don't

know maybe you downloaded a patch version or something has been fixed and then I say we got it it's working we have recreated the proof of concept for the authentication bypass now what that means is that I don't have to have any credentials I don't need to know Joe schmoe's password I don't even need to know the admin password I can just walk right in the door has been opened and that's that power but now what we've opened this door because of the vulnerability and weakness in that ZK library that higher Upstream only components and piece of the pie but now this whole application is something that we can just beat up right so we say look I've got local file

inclusion I can read some sensitive files I can read configuration files I can see how the server is built but now what because we wanted to think what more could be done here I could literally log in as admin that might give me more power and Caleb okay Wise Guy smart little genius he says look you can make authenticated requests to other endpoints we can crawl around the program and really do anything that we want here if you specify this path zcal or whatever ZK authentication I don't know it'll effectively make a post request and you could pass along any arguments huh that sounds really cool that sounds like something we can work with with a

couple limitations uh for the other folks nerds and Geeks the post request is a little bit limited we can't get or receive info from the server we're only pushing and putting information on the server so we sort of have to drive around in a weird way but we start to think on this and I'm talking with fricos now I've made a new friend hey reached out across the internet and now we're best of buds he says look I appreciate you digging into this I didn't want to disclose a full working proof of concept I just shared those burp Suite screenshots because I didn't want to enable script kitties threat actors maybe some folks that could do real damage have

some malicious intent and make some noise out of this and I didn't try to abuse the authentication bypass even further he didn't try to push the envelope and see what else could be done here I just showed that you're able to reach these endpoints we can see that local file inclusion and I say gotcha gotcha I'm gonna try and weaponize it a little bit further I want to push the envelope here see what more could come from this foreign so with our authentication bypass with this hey moving around the application as a sort of pseudo admin user like a ghost really I can post to other endpoints but now what can I actually do as like a feature and functionality

could I create a new user maybe I I want to act like the hacker I want to I want to put my hacker hat on act as the adversary and let me see can I gain persistence in some way with a new user or I don't know could I could I try and grab their cookies or their sessions or find out the admin password problem is ZK framework this whole Library this application of the way it's written is really weird it's uh it's put together as if it were a desktop interface so it tries to be very very familiar and loving to a mouse and a keyboard right you think user friendly with widgets and

buttons and all these doodads so the way that you might normally think if you're a hey hacker pen test you're a red teamer you think I can beat up this web application maybe I can do some cross-site scripting to steal cookies or session info yada yada well the way that you've done is normally just a single request you say hey I want to change this password here's the username password password confirmation send it over you're filling out the form and in one request you've bundled up and packaged all of that form info ZK doesn't work like that ZK says Hey every time that you move your mouse every time you move into and focus on an input box whether you want

to type in your name whether you want to type in anything it sends another message to the backend server you've moved your mouse over here you clicked on this button you started to type in every single keystroke you sent to the server you press the letter A the letter B the letter c and that's kind of annoying because every time by the end of it you finally click the submit button all you do is tell the server I click the submit button and that's it it didn't have any of the other information because it was already sent to the server what this means is as a hacker as a threat actor as someone who wants to

weaponize what were found here this needs a lot more hand-holding I need to sort of drive around or swing from One widget to another grab identifiers for every element on the page that's annoying all that to say sorry I don't mean to be rambling too long we started to dig into what we could do here decompiling Java code trying to run up against all these different features and functionality and we started to see some like really weird stuff like funny things I don't know if you can dig into this hey here's a bastard class exception from hell I don't know what programmers put in their code right it could be anything but this is for the nerds in the room

jadix JD decompiler stuff that we can use to cut through this code and we start to run into some Easter eggs there's a Konami Code I don't know if any video game friends but if you do an up up down down left right A B thing right hey you got new 30 lives pops up on the screen cutesy dumb but again that's the extent that we were digging into because we desperately wanted there to be a vulnerability here kind of a weird thought right like we want to break into this thing but what if it could be done so we keep running to walls we keep trying to go down every single road that we could trying to hit every rabbit hole

and see what could we take advantage of to do something more than just an authentication bypass could we get code execution could we steal passwords could we change user information ultimately we found this database driver which sounds kind of simple and kind of easy like hey if any nerds or Geeks and CTF players in the room might be thinking oh duh because you could put that in the back end of the server like the structure and the components here now we're not working with the front end anymore those pretty pictures and login buttons and widgets now we're going to the database that gives us a little bit more control so what if we could just upload our own

database driver hardcore right nerdy and giddy if we start to think how do we actually get a database driver to trigger code this is a jdbc Java database connector and we start to think like how can we write some custom Java how can we write our own payload that we'll be able to Fire and do something to prove that we've gained the capability that we want so we start researching we start googling around we start asking Uncle Google turns out hey some individual Airman 604 I think is his handle he says he's already put together this back door database driver I'm like oh thank you someone else has already done all the work so huge Kudos and thank you to to

Airman for that and he has this blog post on medium that says look I've crafted this code I put together this database driver so you can slap it in and run commands operate on the computer as if you were sitting in front of it you have that terminal or that shell access to run system commands got a wolf in sheep's clothing right and that's it we slap that into my python code we've been swinging back and forth between all these widgets things that we can beat up and hit and we got it upload the database driver everything that we were Cruising Into so far and we got new ground push the envelope even further we know we're just beyond the

authentication bypass we have code execution on the back of server cool okay we've done some damage maybe new cve whatever but we want to keep the right mindset here we want we want to keep the right mentality of like hey we we weren't able to do responsible disclosure first with fricos because they couldn't get a hold of connectwise well can can we get a hold of connectwise so we started to think on this but we start to realize that well there could be so much more damage that's done while I might be able to execute code run system commands here and by the way I'm sorry uh this was a very very simple and innocent uh crafting of a small text

file there's no destination or no trigger that's anything crazy we're basically just opening the calculator application saying look I proved that I compromised this but it wasn't just that we could download some files and data that would be used for tech support quote unquote private key files for the server configuration settings other license Keys subscription information things that would just be for the genuine product and that might be a little bit more you didn't even need to use code execution for that you could just pull it right down with the authentication bypass in that ZK Library bear in mind but we're on a backup server which could really be a high value Target for a hacker threat actor adversary apt

nation state right because again keep in mind all of the other computers or hosts that might end up backing up their data will work and synchronize with this Backup Manager so we start to think hang on what if there's a certain amount of domino effect here hey Poison the Well get the tree and then all the branches and root just die here so we started to dig into the API of this application now that we've gained code execution we have full compromise access and we can still act as the administrator what more can we do keep asking ourselves that question this is uh some of the soap services or methods or functions we can call with

the API does anyone see anything really really interesting there are some fun ones delete agent I don't know how well you can see that there's a there's a get agents get default agent ID create agents I see it down there yeah run agent command oh no so what are you thinking what are we thinking look we had an authentication bypass that we just was point and shoot right we didn't need to know credentials we didn't need to know anything about the target of the fact that it's potentially vulnerable and then we just fire away and now we could potentially again code execution and now we could potentially bundle that up to run code execution on all of the

downstream connected servers that use that machine as their backups kind of wild um if I may I think I do have a video here if my friends might be able to help bring that up I'd love to be able to Showcase it because I wanted to bring that to your attention of the impact um I'll see if I can bring it together so here we have a couple machines all put together and I'm just logging into the Backup Manager for the sake of showcasing say hey here are the machines that are connected here's a Windows 10 box Windows 11 Windows 8 Windows Server whatever whatever now around this time this is October so just about a month

ago uh lock bit one of the ransomware gangs was getting into a little bit of trouble because they seem to have a lot of their ransomware tooling released out online it was public and it was released so I thought well you know what some really good fireworks you know what will really blow people's mind what if we just push ransomware down this big giant supply chain attack so there it is here's our run on our exploit running our code here's python to tell every single Downstream host connect back to me and download lockdit and then run and deploy and detonate ransomware across each and every one of these whoa okay I don't know how well hey we might be

cruising through this year but I did want to bring that to your attention because this could be a real big thing this could have been potentially some crazy Shenanigans I don't know if folks are familiar with caseya VSA uh some time ago another MSP kind of spearhead caseya had a little bit of ransomware Armageddon uh and there were a whole lot of businesses compromised with Regal the ransomware operator and it was exactly this using the rmm or the remote monitoring and management solution to push down ransomware to each component thank you for spinning up the video I did want to bring that because I thought wow maybe that shows the impact here but that's that

that's cutesy when I get to put in a video of hey maybe four connected machines for the sake of show and tell but what's the real blast radius here how much damage could this do we fire up Showdown right Internet of Things search engine folks might be familiar with that to see what else is out there on the open internet how many server backup managers are out there in the world and we got what close to five thousand um okay and remember that is the server itself that is the Backup Manager it is not the downstream components of all the hosts that might be connected to it all vulnerable because of some ZK Library Downstream Downstream Downstream

so uh tough to to swallow that pill uh but again we want to do the right thing here we want to bring this to the best attention that we could we want to bring this to connect wise and do the right thing we're not hackers I might be doing the security research to find this stuff out but I'm not a ransomware operator so we start to wonder hey should we wait for connectwise to patch this thing just I don't know like like tricos or Florian or our Twitter friend mentioned and the direct messages that I had back and forth and of course yes the answer there is yes we should wait until they're patched to tell the story

and then some time passes hey there's some communication and they say look connect boys couldn't reproduce it they couldn't recreate it they couldn't actually bring in some of the breadcrumbs that you guys had to build this detonator they asked if we'd be willing to work with them I'm like yes of course absolutely a thousand percent I want to help with all this so we met with connectwise Caleb and I Caleb I gave the other researchers that were digging into this and it went well it was very good it was very very positive to have a communication face-to-face over Zoom to talk about this where we get to show them some of the code show them what we

built show them how this could really impact things and they were receptive that was the best thing that we could hear because it's I know it's really weird when you have someone that says hey I just hacked all your stuff but we want to build the right relationship with that turns out they couldn't directly update to the latest ZK Library they weren't able to patch even though the patch was available in May because they're running Java 7. okay folks that know Java probably understand that Agony uh the patch version of ZK is on Java eight and migrating those is a nightmare so okay look understood couldn't just as easily patch because that version mismatch but then

we chatted about like hey I'd love to keep helping I want I want to bring this to you for all the right reasons so I know if you're getting a micro pack even something from ZK specifically for version seven for that Java version that you're running as part of this product can we help test that patch can we validate can we make sure the fix really really works against our exploit against our proof of concept and they say their understanding and it might be over this they're open to it but they need us to sign an NDA need us on an on-disclosure agreement um two reactions to this and I'm totally understanding of each and you might have

your own immediate opinion knee-jerk reaction as to wait a second huh wait oh actually yes um number one I think yes of course absolutely I understand I know this is your product I know this is your baby I know this is what we're trying to do to make sure that we're doing the formalities correctly we're dotting all the eyes we're crossing the t's legal stuff right opinion number two wait a second what why I'm just trying to validate and help your patch um you of course can conform your own opinion there and I'm curious what that might be but it's always a weird sort of a certain gut punch when a researcher kind of sees an NDA and especially with

the companies involved and you know you know I don't have to fill in those gaps for you but then all of a sudden some days pass time I think it was just Friday maybe that same week they're pinging us out like uh did you guys have anything to do with this do you guys have any more commentary or stuff because connectwise we just published it they brought out in the open and now everyone knows what's going on it's the cat's out of the bag there is the link to their security bulletin there is their uh server Backup Manager advisory and release the patch that they're bringing out kind of threw us off guard not gonna lie and that's on

it again they're their own spot they can do what they'd like to do but it wasn't what we expected because we thought we were kind of hey in This Together synchronizing some of that coordination in collaboration now the cat's out of the back this is when all of those news stories and those headlines and those journalists in the screaming October as we started with a little bit of our talk and presentation here now they're starting to fly hey connect wise patch is this critical flaw that could have affected those 5 000 servers and all their Downstream components Caleb got some time in the Limelight chatting with crn chatting with other reporters and journalists and we started to do our own write-up we

wanted to scramble like oh hang on wait a second I didn't realize we were getting our communication out this fast so I try to put it together um and we wrote up everything the same exact story that I've just told you communicating with frikos hey building this beating our head against the wall trying to find the right solution trying to do this correctly communication The Better Together story that we want here and that's what that is here's the vulnerability disclosure for connectwise or R1 soft server Backup Manager and the supply chain risk that comes with that in both the weird technical aspect the ZK Library Downstream to software and the personal procedural pipeline of businesses sold to

businesses to sell to more businesses what you might find in the grocery store you don't know where that you don't know where that crop grew or that plant was over at the farm sometimes you just don't know how it's all made behind the scenes so this was a really really weird story to tell because especially when we get out some of our information try to do this messaging we want to give the community the right stuff we want to give them indicators of compromise or indicators of attack or some threat intelligence or some info and details and analytics that they can use to make sure that they're safe that they're doing what they need to do if

they can patch or if they can find any signs of exploitation now this is really really weird because um we're taking advantage of normal functionality other than the authentication bypass right the thing that opened the door changing the database driver is something that anyone could do the admin would already naturally do running an agent command still something that was totally okay in the eyes of the API and all those different functions and features here so we tried to show you and look here are the logs I know it's dull and like your eyes glaze over here but there's nothing there there's nothing to really showcase it isn't out of the norm and that's kind of hard to come to terms

with here um but that's just the server perspective right that's that server back at manager itself what about all the downstream agents what about all the hosts the computers that are trying to back up their data well they just have these logs hey a command ran I don't tell you what the command was they didn't tell you I detonated lock but ransomware my host but something happened so it's not super duper helpful um hard to bring that out and weird especially when we're Walking the Tightrope of a potentially real emerging vulnerability so here's the complete timeline hey as we saw in may we got that patch hey July we saw the communications go out August

September October when we start to dig into this when we have communications with connectwise when we start to try to put the puzzle pieces together the best way that we can now one thing to note here if I may along this storyline is that we never shared our proof of concept kind of the same way that Florian frico's Twitter guy was saying look I don't want to enable those script kitties or those threat actors or those real hackers that do bad things the proof of concept our exploit has not been shared and will not be shared outright because that would just be way too risky and way too irresponsible right now um as far as we're aware there has been no

exploitation in the wild there aren't any real threats beating this up right now and then this was when I was putting up that write-up but those are the cornerstones right now as we're again at the center piece of our story after we've unfolded and unraveled all this we've got this new thing to show we took an authentication bypass and drove it towards remote code execution to be like ransomware deployment mechanism across all these hosts but we didn't want to give that out of course as the Arsenal that it was granted we told the story and maybe there are some breadcrumbs in there to make sense of this maybe smart any seasoned hacker in the room could just

as easily create that proof of concept so let's move forward now in our timeline four months later uh wanted to bring you to today the current events where we are right now uh because I know if I weren't already rambling and droning on for way too long the stories not get over Fox I.T another Sweet Security spot releases this article releases this write-up and the timeline here is February February 22nd today is what March 23rd from backup to back door exploitation of this cve the ZK authentication bypass and the R1 soft server Backup Manager they say during a recent incident response case we found traces of the adversary leveraging and taking advantage of the connectwise R1 soft

server back in manager the adversary used it as an initial point of access and as a platform to control those Downstream systems connected by The Backup Manager crap it started to happen makes sense I mean I guess like there's a matter of time I don't know not a matter of if a matter of when you know uh but this is kind of weird and wild because we start to see everyone scream and shout about this even a little bit more so than when we tried to bring it out to proactively patch to preemptively do the right thing to fix this before it got taken advantage of by bad people headlines are back in action I know you

see tweets you see Twitter messages you see folks on Mastodon saying um NSA is tracking us from the United States side uh sizza I don't know if folks are familiar oh this is the uh Center for information or excuse me cyber security infrastructure and Security Agency and they have and maintained this known database this catalog of exploited vulnerabilities some of the big ones like log4j if folks are familiar with that uh and guess what the silly strange vulnerability that we were beaten up got itself a name smack dab in that catalog an envelope for that ZK framework bypass and we thought wow okay uh this is again a little bit of a spotlight in a Strange

New Way um and that's really really weird to stomach if I may from from a personal level digging into it and spending those long nights cutting through the code trying to find how can we push this further for the right reasons if there even is one I don't know it gets super blurry and super duper Gray but Fox I.T appreciate them they say hey you know what October way back to the center of our storyline the security company Huntress hey those nerds and Geeks John and Caleb they were publishing about this and how it can be used how this authentication bypass could lead to remote code execution and it's connected backup agents another video of lock bit ransomware

being pushed down to all those they have a couple stats up here hey how many backdoor servers are already out on the open internet because that is the current situation and that while they found these breadcrumbs in a real incident response case that triggers them to go do their research and to go do their digging what's the blast radius again right how much has this done so there's a graph of hey maybe by country or where there might be some web shells already present because what threat actors would do and I don't know if I may have moved past the slide let me backtrack just a moment uh nope might have just missed it okay

there is a uh traction no no cool there's a slot where they mention look we found about a hundred web shells public on the open internet since January uh January 9th so it's March now and I I wonder and I feel like that number has gone up right so what a threat actor would do what a genuine hacker might end up doing is taking advantages as they would with all the explanation and detail that I've just given you but using the database driver to add a persistent web shell so they can just naturally connect right back to the server anytime they want whenever they want to do more damage uh for nerds in a room that's the

Godzilla web shell another one of those Java code bases that they can use and what they've seen for later post exploitation has been some small simple stuff they throw up uh python simple HTTP server and then just sort of lurk and wait and try to slowly exfiltrate data uh I got in touch with the fox it fellows I tried to reach out and I said look can we talk can we can we have a meeting can I can we share what we've been seeing here and there were again extremely generous when willing to bring me into the fold a little bit because they had to take this blog post down they were momentarily making changes but

they were asked by a third party who was not named to please take it down and again you can probably use your imagination to extrapolate okay what that might mean one way or the other um and I thought thinking like what is what does attribution look like are we thinking okay nation state threat actor hacker whatever normally we don't talk about that because we don't care and I I don't mean to trivialize that but when we see bad stuff when we see malware we just said get that out I don't want malware I don't want that ransomware I don't care what it is what it's from go away but when Pope when people ask when those

cheesy news headlines or journalists come down and say how did this happen where did it come from they asked the attribution question this is skeptical um and complete speculation but I believe they were leaning towards the same actors of hafnium if folks were tracking a whole lot of the Microsoft Exchange vulnerabilities some time ago between proxy logon proxy shell proxy not log on uh those threat actors would China if I if I may be so bold right uh it's funny when you get down to it because when we have these conversations of all these backdoored web shells and what's out in the world they said something that was so pithy to me and it stuck with me and it still

hits like a bullet because it's very very easy to think everyone's going to say it's going to be ransomware and everyone's going to be screaming and shouting worried about ransomware the next ransomware Armageddon maybe something a little bit more Sinister uh and they don't often think about it's it's very likely just going to be information espionage it's it's very likely just going to be sitting waiting lurking seeing as much as they can get a hold of and listen in on and then pulling it out um and that stuck with me I don't know because Ransom words in the news Ransom was in the headlines wrestlers is what we're chatting about but it might not always

be that they might just be taking it slow anyway I'm sorry I didn't mean to go down that Doom and Gloom uh Road here but I did want to show more of the story while we crafted this exploit around October and then timeline uh when we got the messaging out of the very very end of October maybe say near Halloween or so there are new conversations and Chatters of folks posting their own write-ups of this and sharing their own exploits in December so maybe as early as a little more than a month and we've seen these out and about online uh you can still find them obviously this is still something you could dig into but

they go through and if you actually take a look at the GitHub some of the code that's present here this is exactly it taking advantage of the ZK Library and using the exact same trade craft that we used for the remote code execution aspect okay um sorry long story right but and again to pull us out of the really really Doom and Gloom hey thread actor ransomware ransomware getting uh vulnerabilities and exploits I I want to zoom out because I think when we get to tell this story it still comes with the look we're trying to do this for the best reasons that we could and it took every single person in this play through

to make that happen we would not have been able to see this if fricos hadn't flagged it and was chatting about it we would not have been able to get at least a little bit of the preemptive real better defense and security and patching had we not communicated with connect wise what if that never happened at all what if this just suddenly was a real zero day that wasn't already at least some things had the walls boarded up and the doors closed here uh so I wanted to drive that point home because it was a lot of information sharing and knowledge sharing and getting people together because it's really really weird when I come to an event or a

conference and talking to a room full of people it's very easy to wonder why did you come here why did you either come to listen to me why did you come to play the CTF why'd you go to this Workshop why did you come to b-sides it's because you wanted to be part of this community to make things better while sometimes there might have weird fallouts and weird things that you didn't expect or just I don't know some certain amount of inconsequential things that happen uh you're here for the right reason so forgive me long-winded on this I really really do think the community aspect when we get to tell these stories blemishes faults things that go wrong

all in all that's still the teamwork in the building that we're doing together so if I may last bit here because I know I've been going for too long uh what I would offer you is for those nerds and Geeks and hackers in the room or even for the folks that aren't slinging code on the keyboard day in and day out we tried to dig into this thing because we wanted to help fix the problems that could have came up down the line it's software that I don't use every day except for we don't use but Partners people other individuals do and that's something that we should kind of take a weird sense of responsibility

to go fix and go help because that spooky scary supply chain while I keep making light of it that's very very Sinister and like you've got an upstream vulnerability in ZK The Showdown screenshots that I showed you 5 000 servers that's for the server Backup Manager application that is not for all the other programs and all the other software that use the ZK Library I've seen a whole bunch of others that are using for school administration for managing Personnel I've seen even some for visiting tickets to have visiting time for prisons and that's weird but I mean to say that is that there is still more to go fix and we need to go get a messaging and awareness out there

for them to make that better so that's my rally and cry that's my call to action if I may for you all especially as you spent time here at b-sides learning together let's go fix those problems and make stuff better um with that that's uh that's enough for me rambling cool thank you thank you so much for letting me spend time with you all uh if you'd like to chat more anytime please don't hesitate to reach out I have a QR code up here don't scan QR codes your own threat model and risk you choose but you can track me down uh not a stranger by any means please do please do come say hi but I believe we have some time for

questions if that's a okay if anyone wants to chat okay [Music] there are so many lessons from this um uh I could start like companies like involved in in providing these Services I mean I'm no coder I was doing uh doing a masters in in business analytics and I wrote a small python script to just search for script Twitter for vengeance of whatever I mean if I could do this with um a non-coder I mean they could have done they could have done that and see if there were people talking about their product uh second thing command and control um what's your opinion on maybe companies like really being who have those kind of mission critical systems

like just blocking everything except for those IP addresses that they can they can verify they can interact with it they can interact with their software because part of the things that yes you could have the exploit and whatever so forth but just like with one acquiry um how that was stopped was that somebody purchased that the address where the command and control was so something as simple as that but I know it's maybe impractical I I don't know what the opinion is like I'm not I'm retired from being in it for the last 13 years so I really can't tell you like what if that would be practical or anything like that okay so I'm going to try and restate it

just for me to help clarify for myself and then you can gut check if I got that right is that okay uh second question was hey what about some of the capabilities that we might be able to put in place to just block those things that look bad whether it's an IP address whether it's a zone or region or application is that the right thread yes okay and the first question did you was there another first component of that or am I it's just that it's just that okay you shouldn't have taken you I mean directly interacting with connectwise several times for them to respond when they already were contacted from before and we're saying hey ah and

you know so companies can be more proactive in terms in terms of you know searching for dimensions on whatever whatever um sites that are known to how to have these conversations so two threads I'm hearing vulnerability disclosure and I'm hearing proactive defensive blocking malicious known malicious is that correct right cool um so I'll take vendor disclosure first if that's okay um yeah look hard reality to face when you have an individual researcher that's trying to bang on doors scream and making commotions that they can get something done fixed and that doesn't come through uh and then you know the another company understanding and totally falling on our sword look we're a vendor we're in the same pool we

played the same party and maybe that was why we could get their attention I don't know that sort of sucks but you're absolutely right and like can we be more proactive of getting organizations businesses companies vendors whatever to be more accepting of and receptive to vulnerabilities and security weaknesses that need to be brought to their attention by another researcher or someone in the in the hacking fight a couple ways I think there are really good ways to combat this because ultimately I know and I think I heard from frico specifically that said I got the response that said well hey you reached out to us via channels and methods ABC well we weren't going to see that we

don't if you would reach out to us via methods XYZ then we would have been able to get a hold of you like well that's stupid why do these separate things exist can we can we unify that can we standardize that um I don't know if folks are familiar there's a new RFC one of those requests for comments standards thoughts uh for security.txt it's a lot like a stupid robots.text or a DOT well-known uh standard location on websites where you might be able to say if you find a vulnerability in any of our product services or what have you contact us here and it's so easy to find and so easy to get to because it's security.txt and and

that that's how you can go it would really really be a big advocate for that if I may um but yes a thousand percent I don't want to be uh beating a dead horse here but I agree and I hope we can kind of keep getting more on that good fight ladder question I'm sorry what about trying to block the best that we can whether that's some weird geofencing whether that's some hey adding some list of the Deni list Etc um if I may there are I think is a certain new trend that has a certain asterisk and disclaimers and footnotes of a whole zero trust buzzword I said the bad word um I feel like that's really really really

hard to do in in practice um I know from the user experience and from the administrative burden of having everything and this boiling down zero drust into a more digestible thing like let's say if we took security and we said deny everything don't let anything happen ever unless I said so allow by exception sounds awesome sounds really good and I agree that is a cool security concept and that's what it is I think we do that the right way already with access controls and network segregations and the same stuff we've been screaming about for decades the new Marketing sales term makes it a little bit tough to distill that but I'm understanding that it is very

very hard to actually Implement that and when you falter in one way or the other the whole castle comes crumbled down there's a little social engineering play that I do say you have a application you're a threat actor you're a hacker and you're just trying to deceive and fool the victim you decorate and you manipulate uh you create your program with the Google Chrome icon so that when it fires and runs up on the victim computer you have a little pop-up alert thing in the bottom right of their computer screen that says Hey Google Chrome would like to run will you allow add this program to your allow list or deny its execution and you think well it just looks like

Google Chrome I'm good just go run turns out hey the stupid Sans font that Windows displays it in it wasn't Google Chrome it was a capital I it wasn't the real letter L it was like googie Chrome or some weird stupid typo but now what you or user or any person has willingly allowed that malware that ransomware that crypto Miner to run the onus is now on you which might be harder for some conversations with contracts and formal papers uh and that's kind of weird to draw that line so threat actors hackers understanding look allowless denialists they all exist when the threat actor says I don't need to break the entire technology I don't need to I don't need to break zero trust

I just need to break the implementation how have these people managers system administrators Network Engineers developers how the people misconfigured it or made a mistake uh that's what they're going to hunt down I don't know if that answered your question at all cool

yeah yeah I gotta pull up Showdown see uh see how we're looking oh is it working okay so my question is when you went to develop the EK you said you wrote The Python code in Python obviously so Pi odbc would be native for python what made you lean towards jdbc over peyote BC um in relation to developing the EK was that because Java is typically the Swiss cheese for security uh yes with the short answer uh knee-jerk reaction is I want to know what will work I know I want to know what will fit in the environment uh so again when we're sort of operating at the edge of our understanding and like this is new Uncharted Territory I am I'm

truthfully and I'm okay with grasping for has anyone else done this research I'm going to come to that like the same way I reached out to fricos hey I'm really sorry can can you help but that's I think part of it I think that's part of us all growing I think that's all part of us learning whether I could have cranked it out in Python I could have done rust or golang I could have been I could have been hip but I wanted to do what worked so foreign I had a quick question about I was wondering if you can provide more context on security research in general you said um kind of encouraging people to look

for larger Downstream impacts of monitoring software how does that kind of play a role into your target selection do you look for software where the downstream impact might be greater do you look for projects where a lot of people might have been working with the software in general as a security researcher how does things like that kind of play into your methodology super good question um sort of hey where do you go to hunt and how do you how do you go find what you're looking for et cetera what do you choose in that regard really really weird uh and like you you get to ask that question to yourself um because I'm understanding of look hey

people like to do hacking and cyber security because it's very sexy because you can put it out on the internet and he's like wow look I'm the lead hacker um and there's a certain amount of clout to that like I'm not gonna lie I'm not going to turn that down um with that said I look up to the folks that are cranking out windows kernel local privilege escalation zero days um understanding like look that's above my head and I and I want to try to bring an impact toward stuff that I can help with and that I'd know uh and I'll be the first to admit sometimes that means scraping the bottom of the barrel

sometimes that means looking at these old deprecated end-of-life unpatched stuff that hey maybe has been code that some vendor wrote 20 years ago but it's still being used um and I I I have to say the managed service provider Community could use some love right I think there's always room for improvement especially everywhere and everyone but uh we all I think a lot of times the traditional quote-unquote infosec and the cookie cutter Defcon black hat whatever uh we always looked as msps as like the wild west um but there is so much more need and necessity there and their tooling and what they need to fix and what we can fix there because that is genuinely the 99 that is

literally what a whole of our businesses um and economies built on so I I think there is a lot of impact and a lot of fulfillment which might be greater than the sexy clout or lucrative finance that comes from security research just making the difference for the right reasons can be in those areas that need it the most

hey John uh thanks a very cool presentation and it's cool to see behind the curtain how it all works um you mentioned kind of responsible disclosure bug Bounty kind of you must interact with a lot of different companies different approaches and I saw when you you flashed out the NDA it seemed like you had a slight reaction to that um you obviously observing like a level of confidentiality you kind of kept you know writing the information kind of uh the key components of it didn't disclose that publicly but are there any kind of practices what advising companies like ourselves who we're having to enforce our own kind of policies on this stuff what would you say they're like the best

practice what do you like to see as a researcher like what is an impediment to you doing your job what is actually good practice for for the for the purposes of like responsible disclosure super good question hey uh what are some of those best practices or things that we love to see or or help um when there's something that we're a little bit embarrassed to bring to the table of hey there's a new vulnerability or hey we had some problem uh if I may and I realize again kind of tough pill to swallow the absolute greatest thing that you can do is have that absolute honesty and transparency um I can if I may I'll get into it look

look I Huntress vendor and not not trying to point towards or do some silly commercial by any means we got popped one day hey we had our own security incident we had everyone has their own oops and blemish and mistake uh but we sort of had to hold each other accountable the same way that we want to hold other security vendors accountable that we say like look we had some stupid temporary ephemeral AWS instance that left RDP open because it was just like some default Security Group and there was some stupid password like abc123 literally the like top 10 lines of rock UDOT text that sucks that's really embarrassing um but those raw gory details are I think

really uh something that can be commendable because you're willing to say that and say like look okay we've dropped the ball but now this is what we're doing to fix it this is the impact we've contained it the best that we can um I'm scrambling here on the screeching for straws but I do think at the end of the day it is just the transparency and honesty fall on your sword uh as tough it might be to say sometimes one more okay I have a question um we talk about people processes and technology in terms of cyber security posture um what you showed us earlier was obviously a vulnerability in the technology but within the supply chain

so thinking about the platforms leveraged by managed service providers we've got connectwise cassaya solarwinds there's different tools out there almost every managed service provider leverages a platform like that um so we think we think about the industry in in the third party risk the organizations have by engaging managed service providers what are your recommendations for the potential customer and how they consider a value a managed service provider and what can manage service providers do better super good question hey uh how do you understanding that there is a real need to have someone help and Outsource whether you're working with a managed service provider and specifically in that case how do you I if I may I think the first thing is to

just genuinely vet the vendor the new third party someone that you're trying to get to work with something that you're adding to your stack one way or the other uh ask those hard questions when was your last security audit hey have you been doing some reviews of your application hey do you have a software bill of materials look that's a can of worms I know that's a whole other thing to dig into and I won't touch that hot stove but s-bomb real thing when log4j hit the streets it would have been really really nice to know if my vendor actually had that vulnerable version as part of their ingredients list on the recipe when you have vendors

come out of the woodwork one month two months three months later whoops we were compromised or we're vulnerable that sucks uh so it's those hard questions to hold vendors accountable look have you been doing some of those internal red teams I don't even know if you'd go that far but if you do are you willing to show me like can you can you make this and I don't know if that's the transparency level but those hard questions are absolutely the best way to do it and test like test the controls hey I put a new EDR in place hey I'm using some rmm hey I'm going to see what bells and whistles this doodad might

have kick the tires do the weird stuff and you might get a weird phone call that says hey guys what are you doing why are you beating up our product uh it's you can come to them and tell them like we we want to make sure that we are confident in you to provide security for us for the next year for the next three years however long our stupid contract might be uh that is that is how you can kind of hold folks accountable if in my opinion

cool crickets I think thanks so much everyone this is a lot of fun thank you