Ransomware Reaction - Lessons Learned

is this microphone okay this will work all right so our first speaker is Jeffrey Lang he is the director of cyber defense operations at Virginia Tech um Jeff Lang has been with the Virginia Tech security office since August of 2012 and brings with him 16 years of I.T experience before joining the team Jeff was Computing technology manager for the Virginia Tech School of Architecture he has experience with network monitoring and forensics intrusion detection and configuring security appliances Jeff has a ba in Philosophy from UNC Greensboro and a masters of Information Technology from Virginia Tech he is a sand certified intrusion analysis Windows Security administrator Network forensic analysis python coder cyber threat intelligent intelligence analysts and certified Enterprise Defender so Jeff is going to be doing a talk today on ransomware I just had to and I'm gonna let him take over now thank you guys thank you good morning uh thank you for having me today I appreciate the opportunity to uh talk a little bit um we're going to talk today a little bit about an incident that happened at Virginia Tech last year we've had some departments uh get hit by the cassaya ransomware uh and we've been actually very open about what happened uh the steps we walked through and the things that we kind of learned from it and so hopefully uh this will share some information with you and uh be a good talk thank you so again I'm I'm Jeff Lang I mean it had the introduction there uh the responsibilities that we have uh with our defensive operations are our our security operations network monitoring incident response uh we've got computer forensics and we deploy all of our security tools maintain them uh for our monitoring operations uh my email is there if you have any questions feel free to shoot me an email I'm always happy to talk and share the information that we've got so just as kind of a background about Virginia Tech uh our main campus is in Blacksburg Virginia which is in Southwest Virginia and the beautiful mountains um we do have a large presence in Northern Virginia though and we're actually building a uh an innovation campus there it's under construction now and that's in Alexandria Virginia uh when I took the last statistics we had about 30 000 students uh enrollment is actually up so that's probably closer to 42 000. and we're ranked number 48 in research institutions in the United States um you know nine colleges in our graduate school 110 170 programs or fairly large research one University uh but our I.T infrastructure is very distributed we have a few services that we provide centrally things like networking email our courseware various infrastructure mostly is all provided centrally our security office provides services centrally as well but most of the Hands-On I.T stuff is done at the distributed level every department has their own set of I.T staff some of them are very competent very large staff some departments are much smaller and have a smaller set so we have a lot of challenges that we go through working with different levels of people different technical expertise and different availability with them and so those provide a lot of interesting challenges especially uh and this you know as you go through and you talk about best practices for security and you get a lot of uh gasps when people say things like never have RDP open to the internet never have SSH open well many of the research one institutions are open we have no network firewalls except in limited places security is at the host level and that provides a lot of challenges as well we don't have that centralized management of things to try and control and do a lot of the security layer on kind of that outside edge our security office we have 10 people total um four people on our defense team including myself we've got our architecture red team our risk management team we're working on getting a 24x7 security operations center up so we have an associate director who's kind of working with that and then we also have added a software developer to try and help with the projects that we have going on uh so you know that's 10 people total across the board doing all the security operations uh for the University itself so what ended up happening last year uh Friday about lunchtime uh right before the Fourth of July holiday weekend everybody was ready to have Monday off and ready to go and people started coming back from lunch and opened their computers and had these weird files on their desktop they clicked on them and they were encrypted and there were text files saying hey you've been hit by ransomware you need to go to this site to uh check out how much it's going to cost to get your data back um we started getting some phone calls in at the security office from one of our departments and they were like hey we manage this thing called caseya VSA it's a you know Security Administration software package it allows you to push patches install software remote manage and they were like you know we've got all these people who are connected to this device and they're all getting hit by this ransomware so it turned out there were some flaws in that application from cassaya and there were a number of people hit by this so we started scrambling to try and figure out what was going on uh and what was happening and it turns out uh that there was a group that used sodino kiwi and attacked all the machines we had on our Network that were connected to the VSA server uh it turns out it wasn't just us it wasn't a targeted attack at us it was targeted at cassaya the VSA Appliance so this was a supply chain attack they didn't attack us directly but they attacked a vendor that we worked with and the primary targets were actually managed service providers so there were 60 msps that were impacted there were a handful of other uh areas like Virginia Tech that got hit that weren't really an MSP but it was over 1500 clients of those msps it was a huge attack surface and it but it only uh it only attacked their on-premise software so their software is a solution in the cloud wasn't impacted but they did shut that down uh as well because just in case it was something they didn't want that to happen so kind of the details of what it was um it turns out there were seven flaws that were exploited in order to get this attack to happen the cves are all Lister SQL injections cross-site scripting all sorts of you know pretty uh normal and and things expect not to have in a management application like this um what they did is they uploaded an initial payload through a bypass and once they had that they used the SQL injection to execute that file and once they had that they had administrative access on the entire VSA Appliance so once that was they had that footing they were able to push out software to every client that was managed by those VSA servers um they downloaded an agent.crt file uh and they pushed it into the cassaya directories that uh were protected and were trusted so you know this is admin access this can install software and the operating system allows it because you know this is a trusted component uh so they actually were able to run a Powershell script that disabled Windows Defender and then downloaded um a some additional files uh they had the cert.exe which is a standard Windows executable they used it uh to decrypt an agent.exe file so now they had their own agent running on all the boxes that the VSA managed wrong way um so once they had that they actually used it to download vulnerable software and this was an old Microsoft anti-malware that had a side load vulnerability so they downloaded it uh it's Ms MP eng.exe that actually looks like a you know a valid file and for a long time it was a valid Windows File so if somebody happened to see that running you're not going to have any question about that you're like oh yeah that's that's the anti-malware and that's fine um they were able to side load a dll file and then that dll file told the anti-malware to encrypt files and to do it in such a way that you couldn't do things like Shadow recovery or rollback so basically in place you couldn't recover those at Virginia Tech it turned out we had three of these servers running only one of them was compromised it was the one that had its administrative access open to the world so uh Port 443 pretty common pretty normal uh but you know you question maybe why was it there and we have some questions about that as well um now the deed was done and we needed to recover from it uh the department that Managed IT they also allowed six other departments to use it they had already invested in the infrastructure and were paying for it and they said hey why don't you guys use this too so all of those departments were impacted 111 servers and 805 endpoints in those seven departments were encrypted all the files on the machine all the documents were encrypted also it followed any map drives any synchronization drives like uh Google drive or Microsoft OneDrive and we actually had multiple file servers that were encrypted so we had terabytes of data on file servers that were encrypted and from the endpoints themselves so what did we do um well first we shut down all of those VSA servers even the ones that did not get compromised at the initial response we didn't know what that Vector was and so we were like let's shut them all down we also shut down the network portals that were connected to them just in case a VM popped up uh accidentally we knew that no network traffic could get to it at Virginia Tech we have kind of our computer incident Response Team a cross-functional group of people we have some guiding documents I have a link for that later in here uh we activated that team and so we were able to bring a number of people together and start having conversations kind of across the university with the Departments uh and with additional resources that we could bring to try and discuss what our next steps were we notified all of our senior it management which is part of that cert is to get out notifications so that the university knows what's going on we contacted the Virginia Tech police they are a full Police Department so they opened our initial police report which we then were able to report to the FBI and that was in Richmond Virginia that's our local Hub but they didn't have a lot of information because it was being actually run out of the Austin office so they forwarded some information to us as the as their regional office received it from uh from Texas and we kind of went from there uh we also opened a case with the ic3 uh submitted that so that we had kind of our paperwork done and everything going on there uh we did open a ticket with cassaya which was very important uh since this was a now a widespread event across their product they kind of took the lead on incident response we provided data to them that we had and other organizations provided data as well for them to test and to figure out everything that was going on we started having daily Zoom calls we had actually two on that Friday and then one Saturday and one Sunday I remember standing at the grill and then going into the house and sitting down in front of zoom and talking to Senior Management and the Departments to get status updates from that and then to go back out and try and help finish off the Fourth of July party that we had going on um so once we kind of got our our grip on what was going on the department started to identify all those machines that were out there that they needed to basically go to and and figure out what needed to be done next with them uh we decided that basically we're going to restore from backup uh and then you know wipe every machine and replace it so they began that process of identifying things and giving a status with that um and once they did get that restored they started scanning for pii so any any confidential data any Student Records anything like that we needed to know if anything was on those and then we also had to get in touch with the Department of Education or report to them so we had weekly meetings with that update them on our status throughout everything that was going on so the request for the ransom actually came through the r evil group they are pretty widespread ransomware is a service organization um they announced on their happy blog that yes we did this haha look at us we're great um the uh we received permission to negotiate and we got a number from them that was forty four thousand dollars per decryption key there was some confusion uh with them trying to figure out exactly what that meant and ultimately it meant that every one of those machines had its own unique decryption key and it was going to be forty four thousand dollars to pay that um we did some negotiation for about 30 Keys we had some some thoughts that we might need to potentially pay some Ransom and so they dropped that down to 700 000 for those 30 machines a few days later and you can see the announcement uh they decided that they would rather just offer out a universal decryption key for 70 million dollars and of course nobody took them up on that offer uh so as we were going through kind of the review uh one of the Departments realized that they didn't have good backups that failed and so they needed to think about paying the ransom for it and that's how we got to that uh number of 30 they said this would probably do us and we kind of added in some for some other departments in case we needed kind of that over overhead so we reached out to our cyber Risk insurance which at that point we realized we should have done that at the very beginning it's one of our lessons learned that we'll talk about in just a little bit but they would have been able to negotiate on our behalf they have people who that's kind of their job so we potentially could have gotten you know a much better deal on that but fortunately we found out that we didn't actually have any systems that needed to be recovered from the ransomware using a decrypter uh so the the department that was running it they had their VSA server running on a hyper-v M on a Windows server that it was itself managed by the VSA so when the encryption started happening it started happening on those windows hosts that were hosting those VMS and they basically ate themselves so it shut down all the VMS shut down the servers and when they were able to actually get at those VMS they realized that they had encrypted the data was still on them and it hadn't even downloaded the agents uh to those VMS so they were able to go through a process and pull that data and not have to do the ransom so fortunately we decided we were done with that we stopped all communication with them and kind of went from there uh we did have a tabletop uh in earlier this year and we had a lot of Senior Management with the university and our University president Dr Sands said hey any ransomware I have to approve no one else at this University can pay it so that was it was something good to kind of get that perspective and say oh yeah you know any decisions we might have you have to take it to the top you can't have somewhere someone even the CIO of the University make that decision it had to go all the way to the top um as far as decryption keys go uh so uh our evil and their infrastructure disappeared on July 13th they fell off the face of the Earth no one really knows what happened well someone probably knows what happened but we don't know what happened um on the 22nd of July so this was 20 days later cassaya announced that they had a universal decryption key uh so that was great we signed an NDA we actually had a our library had a digital live of scanned images we have a big architecture program so it was a lot of architectural drawings that they had accumulated over the years it was a few terabytes worth of data they had basically decided not to recover that data they were just going to recreate it as people needed it and they were actually able to use the universal decrypter and get access back to those devices uh in September we've all found out that the FBI had had that Universal the crypto key for about a week and a half before they made provided it to caseya so there was a lot of response time and a lot of effort that was going on that you know perhaps could have been avoided if we'd had that decrypter a little uh sooner uh on September 7th uh the our evil group came back uh and in May of this year they started updating the uh ransomware software uh in its under current development now so they kind of went away for a little bit and then came back just as uh as annoying as before so things you know some things went really well uh as far as the response you know of course not having it would have been a better thing but it happened uh and so our cert activation those things went overall very well they allowed us to bring people together and to have communication across the Departments um one of the biggest things from the first call we had was Senior Management uh it was never a blame game it was never fingers it was okay this has happened how do we recover how do we work together to make sure that we get our data back and we ensure the Integrity of the data that we have that made a huge difference because nobody then was blaming someone else so we were really sharing everything there was no impetus to hide things that we found or you know be disingenuous about something and so that made a huge uh benefit to us overall um the departmental response was very quick and effective you know within a few weeks they had gotten most of the systems recovered that they could uh and they were moving uh forward to get things taken care of as they could we did run into some issues uh but that you know that will generally happen we maintained our daily updates for the first two weeks and then we had a weekly call and then we had you know once a month calls and then finally we did get it finished out so that was good everybody attended everyone was there from all sorts of different we had University legal represented our Risk Management Group we had all sorts of people involved um we also during the initial event we had a lot of response from our Central I.T although we it was in a department that this was going on they supported all the work that needed to be done uh cutting those Network portals we had some centralized backups that they immediately changed the data retention periods so that we could roll back and we wouldn't overwrite dials on Central Storage uh they even provided physical Hardware so uh the initial group that managed the VSA server they had you know they needed a large box that was able to host uh those VMS so that they could see what kind of damage was done so our Enterprise systems had a box that would do that and so they lent it to them so that they could stage uh the way that they got things going um and then we also have some staff that supports the division of I.T internally for desktop support they also handle things for our VIP clients their services were offered and they helped reformat reinstall and re-image machines to try and get people back up and running we also had pretty good Network forensics we immediately started looking to try and find out so hey is this an encryption only event or is there also data exfiltration response is a very different different based on which one that is we determined that that we did not believe that there was any data exfiltration going on and caseya later confirmed that that this was an encrypt only attack and it was not a data exfiltration so that worked pretty well but not everything was good and again uh that first thing you know why was that uh administrative interface exposed to the internet um there were some needs for some of the vs8 it would be exposed to the internet but perhaps they had a little too much so you know