Avoiding insidious points of compromise in infrastructure access systems

our next talk is avoiding insidious points of compromise in infrastructure access systems our speaker is sharon goldberg sharon goldberg is the ceo co-founder of bastian xero a startup that is reimagining the tools that engineers use to secure remote access to infrastructure she is also a tenured professor at the community science department at boston university uh if you want to submit questions you can go to slido.com hashtag besides sf2022 thank you take it away hi everyone i'm really honored and happy to be speaking to such a great group of people so um you know please if this what we're talking about is interesting please come see me or my colleague and ming who's sitting right here and tell

us your war stories about your infrastructure access systems we'd love to hear it it's super helpful for us so um this talk is going to be mostly war stories so i'm going to tell five different incidents some of which so i've got not pecha and solar winds up there how many people know in grand detail what happened and not pecha for example in this room how many know like sort of every detail okay um and how many people know every detail of what happened with solarwinds okay so i'm gonna go into like specific pieces of this that are relevant to kind of the point i'm trying to make about how to think about these types of

systems so some of this stuff is a little bit more obscure and some of these are really random like fluffy bunny from 2001 is basically a story that was told to me by andy ellis who's the cso of akamai so this is an akamai hack story from the dinosaur years and andy was actually supposed to be here giving this talk with me but he couldn't make it because of the date change so you just get me instead um so yeah so let's get into it so basically i'm going to have three acts the first one is sort of the the standing credential approach to infrastructure access so let me back up for a second i'm specifically

talking about you have cloud or data center infrastructure you have engineers that need to access that infrastructure that used to be just ssh now it's more it can be coupe control it can be access to databases it can be accessed to windows servers it can be rdp all of these different systems that engineers use to access their systems how do you make sure that that doesn't become a point of compromise right and i have the picture on the right with the the bunch of squares around the cloud right the idea is like we put so much security in to keep outsiders out of our system which is great but then the engineers do need to get in at some point at least in

most places and so how do you avoid having that become a big issue so i'm going to look at all of these incidents and see what they can teach us about how to build these kinds of systems properly um okay so i'm going to start and so in each one i'm going to start with a different sort of approach and um and and talk about sort of issues here so this is a picture of a bastion host i'm obviously very obsessed with bastions as the ceo of bastian zero what a bastion is is a box that sits in front of other boxes and the idea is you don't want your um your people to access the

boxes directly you want to have something in the middle that you can use to aggregate that access to avoid um you know to to avoid them all being exposed to the internet or to even just have a way of like filtering the connections and seeing who's going in and things like this right and so in this picture i've got a very particular kind of bastion architecture where what i have is i have the user has some key that would allow her to log into the bastion and then there are some other keys on this bastion that are allowing you to access other types of targets that are behind it in this picture it's just a bunch of servers

so this is a fluffy bunny incident that you've probably never heard of but this happened to andy ellis and and when i talk about bastions this is the story he tells me so i'll just channel andy um and so in 2001 there was this hacker um and this was like the old days where there was like sort of just like malice for fun you know this was before the ransomware days so what this guy did what this group of people did was they hacked into some users machines and they replaced their ssh client with a new client that was exfiltrating ssh passwords so basically instead of running a proper ssh client you ran this adversary's client

and it would steal your passwords and send it to the adversary and so what they did was they compromised the users and then from there because they had stolen this key they could then log into the bastion and so what they then did was they sat on this bastion which was now owned by them they also ripped out the existing ssh client and put in their adversarial ssh client and then they were starting to exfiltrate all the passwords that were coming in through that client so you can imagine like they were just sitting here in the bastion and just collecting passwords as users were appearing logging in and putting in their passwords to access the next target and

they were just collecting all of those passwords right um and so you know when andy tells the story he actually says that they were not as affected because this adversary luckily did not decide to exfiltrate the s the past phrases that were used to unlock ssh keys so they were not as hacked as others could have been but this was sort of a moment that caused them to change the way they think about credentials and so this is this is like the classic incident that everyone who anyone who runs a bastion host worries about like if an adversary gets into your bastion host what kind of damage can they cause because it's such a point of aggregation

for your infrastructure um so a couple of lessons to take from this um is this this is like sort of the classic story that we've all been hearing and i'm going to talk about this notion of zero trust in a couple slides don't give your users standing credentials um and the issue here right was that when the adversary actually exfiltrated the credentials those credentials were good for a really long time so when you steal them you don't necessarily even have to use them right away you can use them later and the victim doesn't know what happened because these credentials may have been stolen six months before and now the adversary is using them another thing that i think that a lot of

people are familiar with like if you have a bastion host it's a very sensitive place in your infrastructure so it's really important to think about like how do you make sure that this thing is not um you know not unpatched it's not running some old software like you have to sort of stay on top of this because it's a place where if people get in there's a lot of information that can be pulled out of it and the last lesson is is use mfa so i'm a professor by the way so if i want to just turn this into like my infosec 101 class so why would mfa help with this um any thoughts on how mfa

could have potentially helped with this with this attack

they couldn't steal your mfa right so like we're talking about 2001 i'm not even sure that there was mfa in 2001 i'm having trouble remembering i actually i'm fairly old but i wasn't in infosec in 2001. um there was yeah so so this was another way right because stealing ssh passwords stealing ssh keys doesn't have to be so deadly if there's an additional form of authentication that's not the key right so that's another good reason to use mfa so this is a very old example but it gives you a good sense of like why it's important to think about like what credentials are long-lived and what you're doing to actually authenticate your users okay great all right so here's another

architecture that everyone in the room is familiar so in this architecture you know instead of having your targets exposed to the internet you put your your targets behind a vpn so when the user wants to access something they have to first log into the vpn with their vpn credential and then when they do that they can they can go to the specific target okay um so i'll tell you a story of operation aurora how many people are familiar with this incident a little bit okay so this was an early apt um believed to be a chinese apt and what i what i recall happening here was there was a microsoft internet explorer zero day that they

that they found and effectively what happened was when a user and and um when a user clicked something there um there was this vulnerability the zero day an internet explorer that was exploited and then that machine was owned so this is another story i got from andy because akamai was actually affected by this and if you hear it so i'm going to channel andy here if you if you hear him tell the story he actually said that the dwell time of this adversary was so long that they didn't know it was in the network and they didn't actually know when they were compromised and they don't know who was initially compromised so and if you

go back and read the stuff from a decade ago about this incident it's all about the internet explorer vulnerability it's super interesting you don't really see much about what happened next so what happened next was that the end is so relevant because the talk we had before was supply chain attacks right what happened next was that the adversary was moving quoting andy in this the soft sticky inside of this uh of this squishy inside of this network right and sort of moving laterally across across these machines and then what it ended up doing was it ended up going to the perforce software which is the um version control software that they had for for their for their code

and they were trying to sort of get information steal source code from all of these companies um so this was actually an early software supply chain attack that's what they were after they were after the um the source code and the way they did it was through um through lateral movement through this network so um if you hear the way that andy describes this this story he actually did not know that they were even compromised because they were being so quiet about it that the way they found out was because google had this these these targets the ones in red they were being controlled by a command and control server that was i believe in

taiwan and um they didn't know that this was happening so at some point um they were able to take the adversaries command and control server over i believe it was google and then google called akamai and said hey guys like do you realize some of your machines are talking to this command and control server so they didn't even know they were inside because they were doing such a sort of quiet motion around because if you think about what they're trying to do which was just steel code you don't really want to expose that that's happening so this was um this was a this was a sort of very early example of the issue with trusting

perimeter vpns too much so just believing that people are behind the vpn is it good enough or people not people machines or whatever flows of data are behind the vpn is not a good enough reason to let them access certain things and this is where a lot of the movement to like this idea of xero trust like don't trust people based on their network location um you need to trust them you know trust them to you need to actually ask them ask them to log into the exact part of the network that they're that they're needing to logging into um the other thing about this was that at the time andy describes akamai as not being a very segmented

infrastructure so again this is 2009. so there was sort of this whole array of machines that were behind one vpn and you could kind of move across these machines so the vpn itself was not well segmented right so when you think about these types of systems and for any sort of access system it's important to kind of break up your infrastructure into different groups of you know sensitivity or type and make sure that access is not granted from one group to the other if it doesn't need to be right so this was one of the things that that andy describes akamai spending like basically 10 years doing after this incident okay um all right great

okay this one is uh this one is is really interesting so um so has um okay so let me start with the the the the tool first so um the way that um ad administration works it looks a lot like a bastion host what commonly happens is you have it administrators that need to administrate a bunch of servers or laptops or whatever it is and what they typically do is they log into an admin server with their credential and then they get and their admin credential is then used to log into the actual machines that they're trying to um you know they're trying to to manage or do whatever they're trying to do to um

and this is again this is really surprising to me because before i started this company actually a professor like i did never i've never done admin so i'm going to again channel andy here um a lot of organizations even in 2017 were using a single admin credential across a lot of their infrastructure so just one admin construct credential would control access to a lot of their uh of their machines and so um there's an article in wired which everyone should go read it's super super super uh good read like just a great story um the story of what happened at merck um has anyone read that wired article about what happened at mark during not

petcha you should go read it's really it's really interesting so like they describe like people running through the halls like pulling machines out of the wall you know because in the course of an hour or two or three all of their machines were bricked by ransomware um and um the story starts and it's even more relevant today sometimes revisiting these things is so interesting you know later because this was there was one machine this is an early this is a watering hole attack is people familiar with the notion of watering hall attack okay so watering hole attack for those who don't know is when you hack something right in this case it's a ukrainian tax software

and you wait for people to interact with that thing that you've hacked and once you interact with it you get hit by malware right so what happened here was there according to the wired story which talks about merck and how merck was hit by this they had one server that was interacting with this ukrainian accounting software and when it did the software update that software was a malware and what happened what that malware appears to have done is stolen the admin credential okay so now um this one basically they talk about one machine in in ukraine that was compromised and this credential was then stolen and then it was used to basically log into all of

these machines with administrative access and it was all done very very quickly so um so so you get the story of like all the machines getting pulled out of the walls um and people just like throwing their laptops away or turning them off so that this would stop propagating um and it took them like two like two weeks to sort of re reissue computers to everyone in the company because all the computers were basically bricked in this way um and it was because the ad credential is so powerful so this is like um okay so this is this is why i like i really think about this a lot because um we have a tendency in the security

industry to like solve problems that when we worry about our users getting hacked right we worry about our users getting hacked so we solve the problem by putting in some kind of trusted system that's going to eliminate the risk of the users getting hacked right but the system itself may be extremely overprivileged right the system itself may have really really deep privileges into your infrastructure and so if someone actually compromises that system then the impact is really big so that's why i bring this example up specifically like i'm just giving my spin on it is that like it's crazy that there was one admin credential here and it was used across the board to attack

everything but it's something to think about so um so admin credentials with over broad scope danger segment your infrastructure admin is a single point of compromise um and you know we already talked about software supply chain i actually didn't realize i was going to be talking right after a software supply chain talk but that's other another important thing so i think one thing to take out of this is like when you think about what you've built or what your systems look like like try to think if there's one master credential in there somewhere that if hacked like terrible things happen a lot of times these things exist and nobody notices um and this this but not pecha

incident is a good example of like terrible things that can happen if you do that and and so for example with ad administration you want to have you know tiered administration you want to have different credentials for different parts of the network and things like that but that wasn't necessarily done and actually was surprised to see this but this was 2017. so it wasn't really that long ago that people still had architectures like this okay any um okay so i'll keep going great all right so so enter zero trust right so like three incidents to set us up to to look at why these zero trust approaches um are are important how many people are rolling their eyes when i say

zero trust okay so everyone's rolling their eyes it has it has meaning so i'm gonna put my professor hat on it actually has meaning it has real meaning to me and by the way when we started bastion zero and that zero was partly zero trust and partly zero bastions when we started bastion zero we were like talking about zero testing we didn't understand what it meant so i'll tell you what i think it means for access for this specific problem space which is access remote access what does zero trust mean it means don't trust the user with long live credentials don't trust the user because of their ip address that's what it means so it's all about

the user right make sure that you're not assuming that just because someone has a particular network address they should be good to access something make sure you're not giving them a long-lived credential so that it doesn't get stolen like it did with fluffy bunny right that that's what that's what zero trust tends to mean um and that those are like despite you rolling your eyes those are like legitimately worthy goals that i think people could agree to do right and i know that the term is highly used but that's what it means right so this is a picture of a way to build a zero trust system how do we actually implement this there's a lot of different ways um so

you can you can build different kinds of authentication systems that will provide this type of zero trust system right so you can build a certificate authority that would issue short-lived certificates to the user whenever they log in right so they authenticate to the certificate authority imagine the yellow box is a certificate authority they authenticate to that thing they get a short-lived certificate and that lets them access the target that they want to access and then the certificate expires after an hour or five minutes and then they can't get access until they authenticate again so that's a great example of zero trust access with a certificate authority you can do the same thing with a single sign-on right

you can log into your g suite or your octa and then you can get a short-lived token that would give you access to the targets so there's lots of different ways to do this um and um you know this is a picture of uh zero trust with a certificate authority right so so alice over here is um is is getting access from the ca to a particular um to a particular target and and um that's the example of the certificate um if you do it with an sso what you might get is something like a saml token or an open id connect you know jot but anything like this it's a short-lived token a short-lived

credential that the target uses to determine if the user should have access okay so so this is cool so so like i'll just tell a personal story when we started bastion zero um we looked at zero trust and we realized what zero trust means is like basically one of these things that i showed you and then i said wait a minute certificates like here's something that i've been teaching in my class since it happened and i will say that this incident for me is in my list of top five security incidents for me personally and i think it's because it happened in the in the second year that i was a professor so it was like in

my formative formative youth this happened um so how many people know the digi notar incident so not everyone have you heard of it heard of it okay so a lot of people haven't heard of it this is like watershed security incident but it's not in um cloud it's in tls but here's what happened so there was a ca um in the netherlands called diginotar and what it does is it provides access to sites in the netherlands like the dmv and things like this it got hacked and the key was stolen and so the adversary actually had the signing key for the certificate authority and it was able to issue certificates for whatever it wanted what

it decided to do was issue certificates for google and hijack google traffic and it was actually able to introspect on tls traffic for google this incident highlighted a couple of things it highlighted that cas are a single point of compromise if trusted blindly and they get compromised it's super bad it's very similar to the idea of the ad credential and um and this is actually what happened in the tls world as the tls world moved on from cas and started doing things like certificate transparency and certificate pinning and all these other really great things to like lessen the risk from cas um okay solar winds so everyone or wind incident do people know about the saml

part of the solarwinds incident the sample part okay so everyone knows the sample part so just really quickly one of the things that happened with solarwinds after the adversary compromised all the things that they compromised they actually went after the adfs server for their victims and they stole the adfs server signing key so they were able to create their own sso tokens for anything they wanted and once they did that they could log into anything they wanted so they're all the things that happened in solarwinds but this was the ultimate thing that they hacked and from there they owned the entire network for their victims right so again another example of like trusting sso too much dangerous right um

okay question for you in my remaining time so you get my point um question for you in my remaining time if we had mfa would that have stopped the solarwinds incident it depends awesome yes so if we had mfa like in my picture would have stopped the solarwinds incident no right because who cares right because the adversary would have still stolen this key and walked away with the key and made tamil tokens for itself right if mfa goes to the sso provider then it's no good to stop attacks on the sso provider right so here's an alternative architecture and then i'm done right another way if we actually used if they had used mfa in this way then it

wouldn't have it would have actually been able to stop the attack right if the mfa was a separate component in the system and the mfa wasn't going to the sso provider then the adversary would then need to compromise both the sso and the mfa in order to access the targets right do you see the difference right because the sso is you know the targets are validating token from the mfa you can see the orange token and they're also validating the yellow token both of them separately and that's what's required to access the target so if you think about the way you build access systems a better way to do it is if you're going to put something

behind an sso you should have an independent mfa as well that you don't have to risk the sso getting compromised so spoiler this is basically what our architecture looks like right so the idea is when you think about zero trust and you think about access it's important to think about the users getting hacked super important right like the first three stories that i told you but then also think about your access system getting hacked and how do you actually stop that from happening and that's really where our focus is in the company and i'm happy to tell you more but i don't have any more time so this is okay this is a picture basically of how we do it

and i can tell you more about that later but this is the bottom line right so we think about securing the users we don't want the users to have standing credentials we don't want to trust a user just because of their network location those are the two tenants of zero trust segment your infrastructure also arguably a tenant of zero trust but then in building such a system be careful that you don't put a giant credential in there a giant ca or something like this that controls access to everything because if you do that is where the adversary will go and we've seen the adversary go there right so it's another thing to think about and with that i will be done

the slides are available on our website i'm on twitter as well so you know please reach out and let me know your thoughts about this talk i'd love to hear about kind of what you've been thinking about with regards to this problem so yeah thanks [Applause] okay okay we have one minute for questions if anybody has any questions questions questions okay i'll just say recommended reading that article about merck getting hacked everybody read the did you know tar incident everyone needs to know about that yeah sand worm that's so if you don't want to buy the book you can read the article in wired which is an excerpt from the book yeah your question okay awesome thank you everyone

thank you you