Ofir Shaty & Sarit Yerushalmi - CrimeOps of the KashmirBlack Botnet

and now we have a really dynamic and sweet duo of speakers who have never presented at b-sides tel aviv before but they have presented at b-sides san antonio texas and besides munich in germany and i'd like to tell you a little bit about our next speakers i would like to invite them to join me on the stage sarit and fear please join me on the stage let's give them a warm welcome and please join me here yes and uh as they get ready they bring up their courage give them all the love for first time besides tel aviv speakers we're going to bring them the chaser shots as is the tradition they each chose a beverage of their liking i want

to tell you that sarit and ophir have some comments cheers they have something in common so ophir likes to take road trips on motorcycles and sarit likes to travel which is a shame because we didn't get a lot of travel the past year but right now they're going to take us on a journey into the world of the cashmere black botnet the stage is yours thank you hi everyone good morning i hope you enjoyed your coffee break so we hear about buttons all the time and in this session we'll we will take you down the rabbit hole into our important investigation we'll show you what we learned and we'll give you a few tips for your

next botnet in investigation i will start by talking about the cashmere black botnet describe its entities and if it will continue and describe the operation and the devops behind it but before we start let me introduce myself and it doesn't work okay my name is sarit and i'm a security researcher at imperva for the last 10 years i mainly focus on application security and i develop algorithms to detect and protect against attacks i'm off here and i'm also security researcher at imperva for the last five years and focus mainly in database and web application security and we are both very excited to be here indeed so before we dive into the bits and bytes of our research

let me introduce you with the kashmir black botnet okay it doesn't work oh again sorry i don't know what happened is that all right that clicker doesn't work

yeah okay i hope it will work now so it all started on november 2018 and last uh seven months which is basically a research period of time we discover a bot net that infects popular cms platforms such as and it doesn't work again my first time sorry for that oh okay so it mainly affects popular cms platforms such as wordpress drupal magento in more than 60 different countries around the world it performs the millions of attacks per day on average and we calculated that there were hundreds of thousands of boats out there participating in the botnet operation those bots utilize dozens of non-vulnerabilities such as remote code execution file upload and many more this session is a journey into the

botnet core from the attacker point of view so let's start security research investigation can sometimes be like a crime scene investigation but our crime scene is spread all over the network with nobody in place so we need to collect the clues and fingerprints to construct the picture of the virtual crime a recent study in imperva we saw nine million attack attempts exploiting php unit remote code execution and we were wondering why is this cv so popular among attackers to understand this hype we started to analyze data from our data lake we saw different ips using the same payload over and over again attacking different customers which remind us abutment behavior so we decided to download the payload

and dive in and we basically started the mapping step we started the button it is updating uh we we ana we analyzed the code and revealed all the entities of the botnet and later we'll talk about it in further detail the next the next step we took was infiltrate we saw that a botnet is updating on a regular basis so we decided to act like a bot and gather these updates for later analysis and finally we played a victim we wanted to understand the post exploitation stage now let's review all those entities that play a role in this massive operation so the kashmir black botnet entities can be split into three groups the first one is the

kashmir black infra the botnet infrastructure the third party services and the botnet actors under the botnet infrastructure we have the cnc and repositories and b under the third party services we have github basement dropbox that the attacker used in order to secure the cnc hide this traffic behind legit cloud services and to um um okay let's move on and under department actors we have uh the victim and two types of bots and later i will elaborate on that so here is the command and control this is the um the entity is responsible for the entire operation it is located in indonesia and has three main roles it supplies attack instructions to bots it receives attack reports from bots

and it supplies the infection script to infect the victim server and here is a snapshot of the infection script we can see that attacker defines a parameter that represents a quantum task and this task is scheduled to run every three minutes and it's written in python now we can see some imports and the attacker was using basic stephon coding to obfuscate its malicious payload now the output of this task will be sent to devenal so no history will be saved in the next code block we can see that the attacker redefines the victim current up tasks and as part of this redefinition he validates to remove all main notifications now one thing to note here is that attack the attacker

was using the combination of pell and python which are both installed out of the box in many linux systems so this is probably uh will um increase the probability of a successful infection now let's move to the repositories the first one repository a as you can see is a printer component shopping site which was attacked by the attacker and was used to store the communication script file between the bots and the cnc now another entity is repository b which is an educational institute that was used to store the bundles of export and payloads we can see an example of the exploit payload bundles actually the attacker files the the hackers file are located under the

css directory among other css files used by this innocent web server and they are actually start with the in-memory prefix and it looks out there css but they actually zip files hidden with the css extension now here is also the modification date and one great thing in this botnet is that the infrastructure is just like plug and play the attacker can decide to add new exploit payload bundle just put it here under the css directory and that's it no infrastructure changes are required just put it and use it whenever you want now another thing i want to add is that every file here represents an exploit the target a specific vulnerability now here is a partial list of cves that

this botnet uses as part of the infection among them we can see remote code execution file upload remote file include and many more this vulnerability this cves targets different widgets plugins and themes and there are even a decade-old cv from 2011 that is botnet uses in the in the infection process now the conclusion here is that it's not necessary to use an exotic exploit in order to expand the botnet now let's talk about the third party services the first one is github which the attacker was used as a repository to store his files when we checked his repository we sell cryptominers and php web shells and we can say that by using github the attacker was achieving a layer

flexibility as he can just upload another file here and add a crypto another web shell whatever and just use it no changes to the infrastructure another entity is spacebin which is a website that allows anonymous user to share a plain text through public posts called pastes endospace was used by the attacker as part of the infection and again the attacker achieved a layer of flexibility it can as it can just add another paste with commands or a web shell or whatever and use it in the infection no changes to the infrastructure now about dropbox or fuel talk later in the presentation it was due to secure the cnc it was used in the upward process and to hide the

traffic behind legit cloud services now let's talk about the two types of bots we saw the first one the spreading bot asap but it constantly communicated with the cnc to receive attack instructions those are comments from the cnc telling him who to attack and how and the bot was used to infect the victim server and in order to expand the botnet now a victim site that was infected by a spreading bot can become one of two a spreading bot or a pending bot so let's talk about the bending bot now a pending bot as i said just now it's a victim site that was infected by a spreading bot and as a result is under the control of the cnc waiting

for the cnc to approach and change its purpose and actually this is why we named it pending bot and i will talk about the purpose in a bit now i just want to add that the difference between these two is that the pending bot does not initiate communication with the cnc now let's talk about the button scope the infiltrate step so the best way to learn for an organization is to be part of it same for learning about botnet and operation we call it the infiltrate step so after we got familiar with all the entities of the botnet we wanted to understand the scope of the botnet the victims the targets the attacks and the

evolution and in order to unders to answer all those questions we had to take a more active approach to the investigation we landed communication particle between the bots and the cnc and we mimicked it and we actually had constant communication with the cnc to receive all those attack instructions we gathered them all together and saved them and for later analysis now here is an example of a request from a spreading bot to the botnet to the cnc story asking to get attack instructions and we can see a special header user agent either ghost 8 that without it the cnc will not return anything back and we can say this is some kind of security mechanism used by the attacker to avoid

unauthorized access someone will get its attack instructions now although this is not a sophisticated security mechanism it is a basic one to secure the cnc now as a response to the request we just saw the spreading bot will get attack instructions in json format the first parameter the script contains the list of commands that will be executed by the spreading bot first it will run the call command to download the exploit payload bundle sorry again you're helping me okay so first of all to align the kill command to download the exploit payload bundle which is located under the css directory the one i just talked about in repository b it will unzip the file as we said it it's not a css file it's

actually a zip file and see that password adelia putri which is remember this name of you will talk about it later in the presentation it will remove the the file and run a python script the next parameter the payload contains the list of victim sites that will be attacked by the spreading bot and the last parameter contains the ip or the hostname that holds all those sites now so as i said now just just now we impersonated a spreading about in the botnet and had constant communication with this with the cnc to receive attack instructions we asked to get them those every three minutes now we gather all these attack instructions and we use showdown api in order to

extract the organization the asn the country the vulnerabilities the open ports and the install components we took all this data and inserted it into a database for later analysis so let's see what we learned so the first step first question we were curious about was about the target origin country we wanted to understand if the botnet has targeted a specific country and here's the distribution of the targets divided by country we can i think it's it's a bit hard to see but there are 60 different countries that is botnet targets and it appears that the majority of the countries are located in the us this is actually the blue part and after we gave it a thought we we come to

the to the conclusion that actually many servers are i are um are hosted by cloud providers so we can say that the region of the target doesn't necessarily reflect on the origin of the victim now another question we were curious about was how the attacker finds its targets so we believe that the attacker had some sort of scanners that scans for potential vulnerable targets and he was probably using showdown or some similar service like binaryage or something like that to search for some open ports and vulnerabilities and then once he has those potential vulnerable targets you may use cms web scanners like cms map or wp scan to check for specific vulnerabilities and once you found a potential vulnerable target

that he may attack we believe that it was initiated inside a queue in the cnc for future infection now when a spreading bot will ask from the cnc to get attack instructions we believe that this target will be taken from this queue now let's talk about the button purposes in order to understand the purpose of the block out of the botnet we had to become a victim ourself we curated the same as honeypot and attacked it with our spreading block from the infiltration step then we reported back to the cnc of a successful infection and by that our honeypot became part of the kashmir black botnet waiting for the cnc to approach and change its

purpose now we saw five purposes for the botnet about the first two we already discussed those are the pending and the spreading bot so let's talk about the others now as exciting purpose we observed was a crypto minute it mines monero coins and while we analyze the code we got access to the hacker's payment address and we could see the balance in real time the next purpose was we observed was a result of our cms honeypot which was converted into a clickbait bot and once we logged into the um to the honeypot we were redirected to one of many clickbait sites and the last purpose was defacement and once we saw the defacement signature we revealed the hacker behind the botnet

we saw that he is part of the indonesian hacker crew phantom ghost and while searching the web we even saw some facebook page and many interesting stuff but we even saw a site that sells the phantom ghost crew t-shirts now that we are familiar with all the entities of we will continue and show you and show you the entire operation in life thank you sorry [Applause] i hope you got my back with the clicks hi everyone so how this botnet works it all starts when i bought exploits php unit remote code execution it causes the victim server to download the infection script from the cnc and execute it now the victim server will approach repository a

every three minutes to download a fresh communication script in this stage we can say that the victim server is part of the kashmir black botnet now the victim server will the new the newly infected bot will approach the cnc will communicate with it to fetch attack instructions describing who to attack and which bundle to use it will address repository b to download the bundle and github and pastebin to download additional payloads github and pastebin so now now the they're both ready to attack the victim server and on successful attack it will become part of the botnet as a last step in the process the bot will report back to the cnc now that we are familiar with operation

let's move on and describe the stages of the botnet throughout the research period and and that's it so when we first met cashmere black it had only 10 exploits and two payloads pending and spreading it concentrated only on the growth and then infrastructure changes started to emerge to make the botnet more stable and scalable and the last stage was the expansion actually this is a an ongoing stage that we saw over the entire research period the attacker adds new exploits and payloads so i'm going to dive in into each one of the stages starting with the growth so when we fur cashmere black actually has an exponential growth i'm going to explain how we came to this conclusion

we found in our traffic 285 bots attacking our customers but this is only a portion of the bots in the botnet since we see only traffic of our customers so for this example i'm going to use 300 bots for simplicity every bot attacks every three minutes so per day it will attack 480 targets our 300 bots together will attack 140 thousands victims per day let's say that only half percent of the attacks are finished with successful infection it means that tomorrow will have 1 000 new bots in addition to our current new 300 bucks so by day number seven we will have almost half million bots but the reality is a little bit different because we know that we have

limited numbers of a potential of vulnerable targets so the exponential growth will stop at some point now let's move on from the growth to the stability stage

i'm going to describe the botnet evolution over the research period and the devops strategy that enable it to carry out its crimes when we so remember that the botnet had only one repository a and b once the botnet size increased so did the load on the repositories and since the repositories were actually legitimate sites they couldn't be considered as reliable and permanent entities the attacker had to take action three changes were implemented in the botnet in order to fix that so adding new entity repository a load balancer expand repository a and expand repository b there were three main reasons behind these changes first to make the botnet more dynamic and scalable add redundancy and load balancing

the following diagram shows the old infrastructure against the new one while in the old infrastructure every bot will address directly repository a in the new one every bot will address the load balancer to get one of many repositories to integrate this change into the bondet operation that an additional change in the bot that was required i'm going to describe it when i will describe the upgrade process of the botnet now let's talk about internal changes that were made in order to secure the botnet and the cnc operation so the cnc is the most important component in the entire operation securing it is critical let me take you back a little bit to the steps where we infiltrated the botnet

and played the victim we created a honeypot attacked it with our spreading bot and reported back to the cnc we think that the attacker goes suspicious as he performed two internal changes in order to avoid interfering with the botnet around two days after we interact with the botnet the changes were a reporting address was changed and i bought ip tracking mechanism was added the first change is related to the reporting address it helps with managing bots and versions but that report to the new address is a new bot the second change is within the botnet's communication script a simple architectural change adds the bot's ip and country while it communicates with the cnc it helps the cnc to monitor and trace

each bot in the botnet there were two main reasons behind this change to secure the botnet operation and to manage the a upgrade process so let's see how it comes to work the uh the changes that we described created the situation where some bots were using the new infrastructure while others are only aware of the old one this diagram described the upgrade process when an old bot communicates with the cnc without the ip tracking header the cnc returns sends back attack instruction that instruct the bot to download the upgrade script from repository b once the bot executes the upgrade script it turns into a new bot that is now aware of the new infrastructure the newly affected the newly the new bot

will a address load repository a load balancer to get in return one of many repositories actually the accurate script changes the contact job that sarit mentioned earlier in the infectant infection stage now let's talk about migrating the migrating the cnc to a cloud disturbance so there are fundamental problems in the botnet architecture since the bots are communicating directly with the cnc and entities their ip is exposed and security controls may block them an interesting infrastructure change has evolved to solve this problem integrating drawbacks into the operation now instead of communicating with the cnc and the entities the bots are now communicating only with dropbox now dropbox api is being used to fetch attack instructions and to upload reports from bots

this is a big step towards camouflaging the botnet traffic securing the cnc and most importantly making it difficult to trace back to the hacker behind operation when we discovered this change we were very excited since we had authorization key of the dropbox account of the attacker we thought to ourselves it will be very interesting to check what's going on so we start with fetching all the files and mapping the the structure of the account the root directory that we found was adelia p we think that the name adelia has some significant for the attacker because we saw it in several places during our analysis we saw it in password as sarit mentioned earlier and we saw it used in urls inside the

cnc actually the full name is adelia putri putri stands for princess in indonesia we think that maybe adelia is the name of the attacker or maybe it is somewhat that he scales for next under the root directory we have the payload directory which used to store attack instructions actually we found 400 thousands of attack instructions in place next we have the loot directory which includes only one subdirectory no sequel no sequel is one of the bundles that used by the attacker to attack no sql databases with no sql injection sorry i need two clicks yeah and inside we found only one report so putting it all together the loot directory meant to store reports from bots

divided by subdirectories for each exploit bundle so why do we have only one report so probably the attacker was in the middle of transition maybe in the development stage or in the testing phase but this is another piece of the puzzle that helps us to see into the attacker's world and understand its operation now by that we close the stability stage and we move to the expansion as i said before the expansion stage is an ongoing process the attacker adds new exploits on a regular basis and so the exploits actually were extended from 10 to 17. we fetched all the files all the bundles and we tried to map them to the affected platform um we found that

most of the vulnerabilities are in generic plugins and components and most of them are in cms platforms we see that more than 80 percent are affecting cms platforms so the vulnerabilities are compatible to multiple cms platforms and actually we found a 13 different cms platforms in this diagram we see that the percentage shows the distribution of vulnerabilities for each cms platform and that's it let's let's move on to some important key features so botnet development is similar to application development process we need to take into consideration some important key features in order to create a stable botnet that is here that is here to stay those are stability flexibility and cicd in order to create a stable botnet we

need to take into consideration load balancing and redundancy enabling scalability while growing but this is not enough the separation of the exploits from the infrastructure enables maximum flexibility as the attacker can add new exploits anytime together those two key features are the basis of the expansion and growth on the other side we have the cicd branch which includes version control and deployment cycles we call it automation behind every massive operation we must have an automatic process to support it expansion and growth cannot exist without a solid cicd process now let's talk about the insider point of view as a security company we have data of hundreds of thousands of customers where we can see attacks in the wild

but it is not good enough since our data is biased by our customers so here are a couple of advantages we got from the insider point of view as an insider we could see the big picture and not just a small portion of the infection we witnessed the botnet evolution from the first row we saw new new exploits and repositories added in real time and by analyzing the code changes we concluded what motivated the attacker to perform such changes we had a unique foothold into the operation that enabled us to analyze the victims from the attack instructions extracting countries domains platforms etc etc the inside intelligence led us to the educated assumption that there is kind of an

automated mechanism that searches for potential vulnerable targets and initiate them inside in queue in a cnc by analyzing the exploit distribution we concluded what types of exploits are being used what is the frequency that they are being used and which are more common than others all of this information is accessible only from the insider point of view and it is critical in order to understand the scope of the botnet and the challenges and motivations of the attacker now let's have some conclusion about botnet development so in a botnet development the attacker wearing multiple heads the attacker is the devops the architect and the developer using of third party services is critical part of the infrastructure

and it is not necessary to use exotic exploits in order to expand now let's sum up sum everything up so you're probably wondering at this stage what is the current state of the kashmir black botnet when we decided that our research has come to an end we collected ip's host names hosting services and every possible piece of information from bots repositories cnc and entities and we notified the owners of the servers and the third part the cloud providers about the malicious activity and today the kashmir black botnet is dead at least as we know it we search our data lake and we couldn't find any trace of new infection

sorry so what we learned from botnet investigation about potential investigation first we need data collection is essential part of the investigation of the research and we need to search for repeating patterns in order to find our next potential botnet we need to then it's time to get our hands dirty and analyze the code it can help us with a mapping all the entities then we should start monitor everything collect all the data and enrich it with a third party services like schoden and bruce total to extract the ip and the countries etc we want to then it's time to analyze all the data and uh search and try to understand if there is some correlation between the

victims and who is the attacker we can use honeypots in order to understand better the motivation of the attacker now for the important part i want to uh for the important part i want to give you some couple of tips for your next botnet investigation so first document every step in the process take screenshots and save the dates to construct a timeline save everything just everything you don't know when the data will not be available for you thank you very much for listening to our talk about customerblack.net [Music] feel free to ask any question and yes go ahead is

so regarding the identity of the attackers we don't really know who they relate for and but we did found that they related to the phantom ghost crew we couldn't find any they are indonesian but we couldn't find any uh if they are walking as part of the government or something like this and for the for your first question and so i don't i i what do you mean by technology so um actually i don't really know to answer this question i believe that a lot but the the the things that have been used there the architecture is a load balancing and stuff like this are exist in everywhere we're using it in the industry so we

just wanted to show how the attackers need to use this technology as well in this architecture to solve same problems

any other questions okay ah sorry you didn't see you wow

it's very hard to hear you yeah do you have a microphone or something

no i really don't hear him if that i can what

oh if the attacker patched the vulnerability after he took [Music] ah [Laughter] [Applause]

uh i don't think that he patched the vulnerability you just use it and oh oh sorry um the nice person here asked if the attacker patched the vulnerability that it was just using of the server that is just uh took control over so we didn't see any evidence for that we were just using it and let's like we said it was you it was um he or she we don't actually know but the attacker was using the php php unit cve which was used as a first initial step into the server and we even saw that the the service was indeed vulnerable like we uh find some of them the repositories that we was uploading the

bundles of excellent payloads were still vulnerable yeah in in one case in our honeypot we we saw that actually he didn't patch the vulnerability he took control over the server and he replaced all the php a code in obfuscated code by his own we analyzed some of it but so for your question he did not test the vulnerabilities yeah yes we don't know we don't know it's one person actually we think there is a crew of people i agree with you yeah okay thank you very much for listening