Mike Erman & Alex Gonzalez - Stealing the Network via Cisco Anyconnect VPNs

so hi everyone thanks for having me out here at besides it's this actually my first security conference I'll be talking at and it's also my first b-sides event it was Nina Alex Gonzalez did the work for this but alex is expecting his first child so he was not able to make it down a little bit about my background that was a reactor operator on submarines for six years and and then it spent the last 24 25 years doing security engineering systems engineering work forensics incentive response and about the last nine years doing red teaming pentesting navy red team for the navy directly for come up - for doing operational tests on all of their newest systems and then for the

last five years with those of grumman so current currently worked with Northrop Grumman do pentesting internally alex is a he was my coworker at this time that we did the work for this but now he's a more of a security researcher has is doing working in another internal division actually doing exploit development on on various different products and things so a little bit of background about why we looked at this product security testing isn't what we normally do we're not digging into products on you know in any kind of random basis or anything normally stuff is a target of opportunity for us that we that we run across in the middle of a test and we realized we've got an

exploitable condition and there's no current exploit for it so in this case this was a little bit different we were asked to take a look at it I want to talk a little bit about man in the middle attacks I used to think that these were hard there they weren't anywhere near as hard as I thought they were when I actually started doing them and so some of this isn't actually about VPNs it's about other things but it's more so it's more of a public service announcement it's just I would like to never have to put another one of these into a pen test report so if we're gonna talk about how we did man-in-the-middle

attacks on the Cisco VPN the things we had to do to get there what we found when we got there and and then if you're not worried about man-in-the-middle attacks because you're doing all the things correctly then we're gonna sort of talk about what we did to be able to steal your VPN right out from under you - a little bit but I can't release all the yet because Cisco has not finished patching all the problems so we're gonna touch on that and kind of go over it but I'll be updating those slides whenever those get those patches get fixed so why we did this trust but verify in April 20 18 we had heard of a one of the ssl VPN

clients that was storing their session cookie in clear-text in a log file and so what we wanted to do was to find out that cisco doing this because that's what we use and the you know too long didn't read is that cisco is not doing this so we looked at other parts of the client to see if we could find some other vulnerabilities while we were looking at it and you know note I really had to dumb down the settings to be able to get any of the man-in-the-middle attacks to work there's a couple reasons for this that we're gonna into and you know used to think that man-in-the-middle attacks were hard like I said but no they're pretty easy you

know used to think about this of I need to be in the path of the stream somewhere at an ISP or a broadband service provider doing something like that but not not necessarily and to think the physical access was harder and I meant to say easier in this slide but as you as you just heard from Bryce and as I hear from everybody who does any kind of physical access probably one of the easiest things you can do if if I were really going to go after someone in a really offensive manner probably physicals one of the first things that I would look at and we're gonna talk a little bit more about that as well

public access on public Wi-Fi coffee called coffee shops things like that excellent opportunities to do man-in-the-middle attacks also your your home if anybody's ever done any kind of lock-picking or if you have locks from a contractor that built your house installed then you know me first time I had a hold of a lockpick set I went to my front door no knowledge of really how the lock work that had just a you know just a very faint understanding it was through my my deadbolt on my door in under 30 seconds and then I went to the hardware store got a new lock to keep in mind a lot of times our locks are there to keep honest people honest and if

somebody is very very interested in what you do going after your home is very much in play and it's also very very easy and in some cases I don't have to get inside your home a lot of network interface devices for ISPs or on the external side of the home and there's network connectivity there I do it all the time when I'm trying to troubleshoot my my connections so it's something to think about as well and this is just a brief overview the the playing field is very very large so insiders on your network somebody that does is able to get in through a breach like we heard in the in the DVI our report then now they're

already acting internally on your network and so they've got a wide variety of things that they can start to do address Rose resolution protocol attacks so ARP spoofing I can start to become the target that I want to actually get into intercept communications I can intercept authentication something I don't hear talked about a lot but I do see on that works because it's as a pen tester first thing I do is look at the network make sure I'm in scope and and then I'm gonna see what the network is doing and what its gonna give to me many cases I don't have to go very far with my packet captures before I start to find

vulnerabilities that I can use to get in virtual routing redundancy protocol and cisco's proprietary equivalent HSR pr2 things that I see from time to time that do allow me to do main in the middle attacks I can they're broadcasting the password and clear text I can join a group I can become the primary router and now all the traffic goes through me so and there's more and this this is really the public service announcement part of this is that you know if I have access to the broadcast domain in some way length local multicast name resolution and NetBIOS name service is almost on almost every network that I ever see and in some some way fashion

reform and this is a Windows boxes just asking for resources that's if they weren't able to find them in some other way so they broadcast out and what actually auto detect as as part of this as well I don't actually understand why this protocol exists because as a person who builds networks I have never not known where my proxy server was so why these things want to broadcast out to look for proxies it's great for me as an attacker because I'll just send a response and say on the proxy and then I can get credentials which is usually the first thing that I do but also I've set it up so that you know my burp proxy is now the proxy for

the for the network and I'm intercepting creds and things because people they're encrypting but they're not doing it the way that they should and in users would like to get to their things so they will click through the that the SSL invalid certificate pop-ups we'll talk a little bit more about that also so what does this look like this is just a screen shot kind of locked out a lot of the other stuff but the in general things are broadcasting they're looking for stuff I tell them I'm the resource you're looking for it and then it comes to authenticates to me I get these ntlm challenge response hashes that I can crack offline or in some cases I can use

ntlm relay X and just get in SMB client session to the share that I need so I don't actually always have to run exploits to get into stuff that this kind of thing is going on on the network and I just need access to data and shares you give it to me it's I do call it having the network opponent so so why would they do this as we heard in the DB our report predominantly financial gain if you don't think that your data is worth very much than your CPU storage bandwidth those are all things for you know 20 25 years that have been important to attackers for various reasons and something that I have seen in small

businesses in the Richmond area and I'm sure it goes on everywhere is that when they get compromised the attackers are not as much going after their infrastructure as just getting user passwords because the users are using their passwords for Amazon eBay their bank accounts all of that and then they're also doing a lot of that on those corporate networks and so so they're their main focus may not be the organization itself but may be the users and go on a shopping spree so we I have seen that and otherwise you might also be just a target of opportunity someone that is just easy to get into so that I can vector through your network to go

after my real targets and in some cases supply chain type of things so if my target is expecting to do business with you and then you become a point of entry that I can start to send legitimate information to them but the attacks are difficult to detect and in general when I look at physical and I look at these these types of hijack attacks they they go undetected and and they also don't I don't need to expose any of my advanced tools so why use an exploit that's a zero-day that was very expensive to develop against something when you're just going to give the network to me if I ask in in the right way so with the Cisco VPN we we started

this to take a look how does this really work and one of the first things that we noticed is that as soon as the client goes to connect it makes a connection over UDP 443 and at the time I didn't know what it was so I looked it up and it's Datagram TLS they basically TLS over over UDP which makes sense for a for a virtual private network the theory is is that you remove the overhead of TCP and because the tunnel the things that are travelling in the tunnel are handling in replays and other session controls so it reduces overhead increases speed so when I blocked UDP 443 the client then tried to make a

regular TCP TLS connection which is what I would have expected but what I was trying to do is drive it to actually he's a proxy and later I found out that how to configure the profile XML the correct way to get a proxy but in this case it was trying all other options first and finally I got it to to do an actual man-in-the-middle attack and it so we were able to grab from that once I told it yes except an invalid certificate because it was configured not not to do that at first so once I was able to get it to do that then it what you expect is that in there's the username the pin and the one-time

password in plain text I'm pushing it through burp so I'm doing an intercept and it hasn't gone to the client I just log in with open connect somewhere else now I'm connected to the network and so that's what the screenshot show what I found in burp kind of what it looks like once the connection is so I'm connected what's the user do it's if anybody has ever used a VPN is probably no surprise when it's something it doesn't actually connect and so they they just wait and then they try it again and so so generally this is going to be hard to detect and it's more effective in a in larger organizations where you've got many VPN endpoints because in

general with the Cisco VPN s if you do authenticate back to the same VPN endpoint it will invalidate the current session cookie and then move over to the new one so from an attacker perspective for small organization not as effective but for a larger organization that definitely more effective but load balancers and things that are moving the VPNs into and automatically into other endpoints so any connect one time and you know passwords and then just password authentication I can intercept that I can get that but also keep in mind that the what the VPN client does like most SSL VPNs is that it uses a session token to maintain the connection once and once the initial authentication

is done so that becomes something I can go after which was what we started with was that the session tokens were being put into a clear text log file readable by a user so any attacker that gets code execution on the system can read that log file and just take the VPN so I can I can go go after use that at another location and then the user gets disconnected and Cisco just connects to the new one because it's a valid session cookie which was interesting to us so things that we needed to do to make this happen is that first it's got to be configured to accept an invalid certificate the default setting is not

going to do that and then hopefully you never run into an attacker that presents a valid certificate but I know it's got to be happening if you're places like Symantec as a certificate authority abused their power in the ways that they have in the past and in the way that we look at infrastructures sometimes PKI is really hard and so it's not it would not be unfathomable to me if an attacker went after the certificate authority infrastructure to start to create certificates to make re access to the networks easier late later so not as an initial attack but something to follow up and then any connect needs to be configured to go through a proxy at some point for me to

man in the middle of this or as I found out later to I just need to get better at burp for doing a transparent proxy so realize later that it wasn't sending my my burp up to directly do the transparent proxy for this so smart card in smart card authentication with any connect we tried and tried and tried so it will accept an invalid cert it will do it again and it will do it again and it will do it again but it will never actually start the authentication process with an invalid cert now it may not be that we we may just be that we don't know exactly how to make this happen but we

did try it a lot and and even when I had to come back and revisit this this year still cannot make that go so nothing I could really glean from that for that attack vector which is good so the moral is you know definitely protect your certificates and certificate authorities and you know smart cards a pretty good way to go for two-factor authentication so how do you hack the Cisco anyconnect VPN in April 20 19 April this year DHS and Carnegie Mellon released a statement saying that many if not all SSL VPN s are susceptible to having the session cookie stolen and the they didn't exactly disclose how this was done they did talk about the previous ones that

were the session cookies being stored in a clear text user readable file but there are other ways to go after and reliably grab the session cookie and then be able to use it to authenticate to the VPNs that actually cannot be stopped so we started this work in April 2018 and we found this right away we reported to Cisco but we reported it to the internal product team and and I knew this was going to go bad when I was talking to the to the person and I was saying this is how the VPN connection worked this is what we saw this is what we did and then they came back and said the anyconnect VPN only uses IPSec VPNs

and so I really got frustrated with that you know after taking the time to do this and realize this person doesn't actually know how their product works and so we sort of gave up and that's our fault and I apologize for that because we really should have pressed it a different way at that time I kind of thought well maybe they do know what they're talking about and I figured I would just let him patch it and then pick it up later which is what I did in 2019 I was doing this presentation for an internal group and I wanted to see can I still do this attack and it was really really surprised to see that it

still worked absolutely beautifully and so I then opened a case up with the Cisco product security incident response team the P cert and and that was a whole different ballgame so with this case we sort of got the same response at first if you go to that link for the DHS release what Cisco says in that is basically that they don't see that this is an attack vector that the the product was operating as designed that users would an attack would have to do something first to get there what what there and so what they're saying is that an attacker would somehow have to get code execution on the system first but the attacker doesn't need system

access to do it I do not need to have any elevated privileges to be able to get this done and we were able to do it very very easily we do see attacks where users are going to websites to drive-by attacks that are running PowerShell and other things in the context of the user that would be able to take advantage of this so we've had the weaponized proof of concept forward since April 2018 we talked to them about this and then we also said that this is a flaw not only in the client side but in the server side there's no configuration setting that you can change it's going to fix this problem so we gave them the proof

of concept and then they ran it and we got their attention so we've been working with the P cert they have a much better grasp of the implications of what we were talking about and and so they were able to take the the weaponized proof of concept to go further with it and at first they were still fine with us discussing the the actual attack vector they were fine with us releasing the proof of concept code but after a couple weeks there was like well we're not sure yet we're starting to feel more uncomfortable with this and then a couple weeks after that they asked us please don't disclose the vector and don't disclose proof concept code I'm

all Holyoake with this because I don't feel like telling you exactly how to hack my networks and anybody else that's using Cisco anyconnect VPN switch is probably quite a lot of people keep it find though that we looked at this vector we did not test other SSL VPN s but our guess was is that just about all of them are going to be vulnerable to it it's it seems to me like a hard problem to fix and and based on the DHS memo in April of this year it looks like that's actually the case so the patches are coming they will be coming soon we don't know exactly when they've given me a couple of dates

they've asked me not to disclose those because they can't they actually can't don't they don't know if they're gonna be able to adhere to them but it will be soon maybe in the next month or two so so we do so we do get updates on that and then at that time I will update the slides with that and release the proof of concept code so how do we stop some of these things and and I'd like to talk about certification engineering it's frustrating to me that you know we we look at physical attacks and as you've heard from Bryce as you've heard from deviant olam and anybody else that's out there doing talks on physical this is a

very very hard problem especially when you look at a lot of things are actually dictated by code so on you we have to have a fire escape code codes in place and we have to adhere those we've got to have ability for handicapped and others to to be able to get in and out of a building so those are important things that sometimes reduce our security but we have a real hard time working with industrial security folks they really don't want to know about the problems they've done their due diligence as far as they're adhering to what they were told to do and as soon as they know about another problem well now they're sort of liable for that so I get that I

don't agree with it but if you can integrate your industrial security into some of your security processes and your monitoring so if you've got badge access to the building adding those to your logs can kind of help you see that when a user is VPN inning from somewhere else but they've patched into the building that morning so something doesn't line up right and I'm not the first to say it I've been hearing it at conferences for 15 years but you know it's still something that's out there it's hard to integrate they see themselves as wholly separate and that's not necessarily the case also physical inspection of gear and your network connections everything is pretty much a must and I don't think

that anybody really does it most of the ones that are found you hear about them in the news every once in a while somebody has a Wi-Fi access point that they've plugged into a switch in a closet those kinds of things and you know they do happen but generally they're found at accidentally and also like another case not quite the same thing but there was a Chinese citizen that was found in a boardroom in Boston last year they just walked in and sat down at a conference room table like they like they were supposed to be there and they were plugged in the network doing things the reason for them to be there at all but but that but that does

happen so when it comes to two-factor authentication trust but verify understand your weaknesses if you're not enforcing HSTs which is HTTP strict Transport Security and you should be and you should be howling at your vendors especially when you've got management applications for your for your enterprise that you know they they should be enforcing this because it's a server-based cookie that tells the browser you need to have a valid certificate and the the connection needs to be encrypted and if it isn't the user doesn't actually have the option to not to accept the certificate that the browser just won't go through some of these examples are like Facebook Amazon Google all of them implement HSTs so

when you've got an intercepting proxy in place then the user is not able to get through it interrupt service it causes problems but you know what when most organizations that are doing SSL decryption in line they're they're updating their certificate authorities and they're issuing valid search for those services I'm not sure I agree with that if you missed the talk this morning on encrypting things there's a lot you can do with metadata and and those guys did a great job talking about that but you know just be aware that that's that is something there are attacks against it so you can if you have some access as the users you can kind of remove that

HSTs cookie cache and then you can just have it just accept whatever certificates invalid certificates or otherwise you're probably going to have a bad time if you allow this and not protecting your certificates and certificate authorities also a bad time but just because someone says that they're doing encryption doesn't mean that they're doing it correctly or and in some cases it doesn't mean they're doing it at all we've we've opened up cases with with you with vendors that they've done PKI for the most part correctly in a lot of cases but then we find a single protocol that is syncing really sensitive information like SSH keys and shadow files and things like that and clear text across the network

so we've worked with them to get get those fixed and also keep in mind that two-factor only ever gets you into the initial authentication everything after that is using some sort of token or hash or ticket to get in and so you get any kind of code execution in any environment you start to have opportunities to get access to those tokens that then you can then use to keep authenticating as that user we see it all the time and things are getting better but just keep in mind it's a vector you need to test your vendors and make sure that they're actually doing everything that they should be so if you aren't disabling link-local multicast

name resolution and NetBIOS name service and web proxy auto-detect you really really should I may not matter too much with you do with your VPNs if somebody does get some kind of access into the environment then they're gonna own the network eventually because of this if you've got it and you know as a novice already said from Google project zero it's got to be some sort of security problem of some kind that it's almost impossible to disable web proxy auto-detect entirely on a Windows box there's just a wide weird variety of services that will just do it no matter what the system settings are these valid search for any connect tell it don't accept any invalid

certificates you made that may sound like a no-brainer but I do know people security people that know that they're in environments where as intercepting is happening and they accept the certificate anyway because they've got to get work done so don't give them that option they shouldn't be working if they can't validate the path decided disabled proxies also but that also breaks functionality in a lot of environments where they don't allow outbound of HTTP without a to the proxy so does not always feasible and use more card authentication as far as we've been able to tell so far the connections not going to go through if somebody is trying to intercept and do some sort of

man-in-the-middle attack everyone's favorite control to hate monitor your logs you know we've been saying it for decades people do it they meet the letter of whatever the requirement is which is I've got to I've got to review my log so they click on something every week and they say ok I looked at the logs and then they're done so we're not really looking at our logs unless we've got a particular problem that we're trying to troubleshoot and in this case there are ways you can detect it in the logs but you've got to write a custom parser for it because it just looks like it's supposed to be there and unless you know that there's a timetable and an IP

address and you're tracking all of the connections and you know when things disconnect you're not going to be you're not gonna be able to see this and so you you end up having to write some custom code to go through the logs to find it not it not an easy problem to solve which is one of the reasons why we know Cisco is having a hard time actually getting it patched weird locations are always worth looking into but you know that that that could vary as well multiple successful logins in a short period of time which is what you know what we're talking about when you go in to parse the logs and again it's just

hard to do you've got to you've got to track the IP address and the connection and then you've got to be able to say within a minute or two if another connection comes in then maybe that's something to look into and I didn't say it would be easy so the most significant defense in our networks obviously is always our people and not just not just the defenders but also users when you can start to train users to look for weird thing like if their VPN is doing things get them to report it our DoD requirements industry best practices PCI requirements whatever whatever it is that we're being told to do for certification accreditation keep in mind that is the

minimum level of entry is something that having been in this industry for a long time realize that you know twenty years ago but somewhere along the way that this became the if I could get 96% stig compliant then I'm good I'm gonna pass my CCRI but this is was supposed to be meet them meet these minimum requirements and then start your security engineering process from there which is something I just never see you know people just do the minimum they do exactly what they're told they don't do anything beyond what it does is it makes your networks homogeneous so if somebody has to adhere to a certain requirement then I already know how your network is configured I

already know where all your weaknesses are it's going to be very very easy for me as an attacker after that and a lot of other much better people have said this the if you haven't seen it you Bix is talk on attacker ghost attacker ghost stories free and nearly free defenses that give attackers nightmares just little things honeypot type things and other nuances with the network that you as the network admin they're you as the network defender know about can really go a long way to giving you some kind of advance notice that none of your other tools are gonna do and especially if you're worried about more advanced attackers then this is going to be the way you're

going to find them and don't take my word for it as burns and other folks have come out and said it previous storms as I've said you can do this then you're probably gonna catch some of the things that we're doing so use your knowledge of the networks against the attacker and never let the attacker know more about your network than you do so questions

so same thing that everybody else uses Wireshark just you know just go through and analyze the connection look at the cipher Suites see if there's leakages and then kind of play around with it a little bit see if I can get it to do other things but for the most part that's that's it there are some and I'm sorry the question was what what tools and techniques to be used to see if vendors are using proper encryption techniques and and so that's it the we will run some other utilities or some other SSL utilities out there that will look for things see if I can have it you know accept a lower lower service certificate like a SSL version 2 or SSL

version 3 and so and it will over report on that and especially if we can use it in some way

so the Cisco product security incident response team Cisco pisser it was just a generic email that address that we sent it to and then the the right Department for the anyconnect VPN has got in touch with us with so we started the discussions from there and then realize that this is a broader not not just on the client side but the server side they were able to escalate that internally and and then they started to realize it was it was a much bigger problem than what they stated in the in the DHS MMO

we did not test the web portal login because that's not something that we use so we only tested the client-side if I it's hard to say the browser has some protections associated with it and it's possible that the browser may actually be a safer that method but we didn't test it so so I can't say for certain

okay so giveaways wasn't actually expecting to do giveaway so I really don't know what to do here so I guess lose it but some of the others have done so somebody named something that you learned here today that you didn't know yes

[Laughter] absolutely yeah one of the worlds of that story is be persistent if you know that you've got a problem and you're not getting the right level of attention definitely try to try to work with them and see if see if you can't get into a different group like I said we initially were working with a product team not the incident response team and so and so it was just the wrong venue but it was following corporate policy at the time and and then after a revisited this in 2019 I was like well I'm not gonna follow corporate policy I got to get this fixed so we went directly did a piece search but keep keep that in mind

and you know something I'll harp on with with what I do even though we're all in IT to some extent in various areas we don't always speak the same language and in our various industries - we have different terminologies that mean different things so I'll be patient try to explain it different ways and and then you know get permission to to disclose it and and you you know like I said with us at first they said yes and I was like no I was just trying to get you to escalate it but I'm not giving away the ability to hack my VPNs to just anybody so I really like to get this fixed any other questions

[Applause]