BSidesSLC 2020 - Bart Lenaerts - SSH Keys: Security Asset or Liability?

take it away awesome excellent thank you so much guys thank you so much for giving us the opportunity of giving me the opportunity to present before I continue Braz can you just confirm that you can hear me well and that the audio and the video is working fine looks good sounds good all right thank you so much awesome excellent well good afternoon everyone my name is Bart Leonard's part of the Dyna Phi team based in Salt Lake City I'm unfortunately not in Salt Lake City but based on the East Coast here and the next 20 25 minutes I'm gonna talk about SSH fantastic protocol fantastic opportunity I used a lot one of my hobbies is maintaining a website a

couple of website and SSH and SSH keys are essential corner stores or building blocks for doing that work but it has risks and in my case there's only like one or two people that are working on the website that when you get into a lot of people it can become an issue and he'd become a liability and that's what we're gonna talk about we're gonna talk a little bit about it yes it's great too but watch out because there are some issues and how can you prepare what should you do actually around - around Association SSH keys cool here are the topics I'm gonna talk about first of all for those who may not be up to speed

we'll talk a little bit about what is SSH secure shell and what are a cessation machine identities I'm gonna talk about some common risks that we're seeing so common mistakes I will talk briefly on a best practice coldness ir7 966 as a foundation then also I want to share a couple of dashboards that I've seen people implementing when working with SSH keys no one to share them and see exactly how a practice could look like and how these reports could look like and then we'll wrap it up yeah cool so I expect this presentation 2025 minutes and will definitely need some questions at the end alright let's start with just beginning for those who are usually I I ask a question always

like who is using the secure shell and who is using keys and so on and so forth can't do it now so I will I will assume that some of people still think of normal SSH is so a little bit of an introduction SSH is used today by IT administrators as well as automation processes automation processes or service is actually there using this and it can be used in many many ways you can use it for ad-hoc that's what it's mostly known for mostly how it was started actually as an ad-hoc - for connecting to a switch or router or server or Linux box and doing command-line tools and then walking out again but that became sort of boring or

I would say that became sort of people wanted to start automating things and so it's very easy to start scripting out SSH as well and there are many automation tools that starting using this CI CD tools are starting to use this or are using SSH actually but also you can use SSH for data transfer in my case I when I manage the website I use SSH and I use it for instance for uploading and downloading or for moving files and moving pictures from one area to the other area and so on and so forth or from between different sides that I have and so it's no longer an individual IT admin tool it's used by so many

different things by services by automation tools as well as for data transfer now why was it invented why was SSH invented why's it so far well it's actually the replacement of telnet telnet that was very popular in the 90s was D default tool to connecting to a device but it was unencrypted you could see username passwords very easy and that is one of the key things why SSH came to live and got that adoption because when you connect a client to a host now everything is coming is encrypted and the connection cannot be seen anymore and so that's one of the main main reasons why I got so popular now how does this encryption work well I use the

term identities the keys actually the uses when a client connects to a host then a host key is sent to the client and that is used as a first step to do the encryption and there are many forms of encryption that you can start using there are I think the the latest version more of some of these SSH clients they can support may be time or different dozen different time of encryptions now do you have the encryption but there's a second thing that is also very interesting and that is the concept of keys SSH keys and so there is the concept of private keys and public keys that you can deploy basically if you are

once you're logged in you can start creating keys and you keep the private key for yourself but you deploy the whole part of the public key on the host and next time that you want to log on you don't have the full of the key you don't have to do the for authentication access validation one you can just start using that key and off you go and your connections merely sound up so basically it's encryption but especially when we talk about private keys public keys these are the machine announces that enable those automated collection and that is not a very very popular usage for SSH because again when I'm using my websites different websites from develop

loans that I have to protection side I just use the keys and I can go from one to another and it's automated and it's no longer i doesn't require each time username password authentication or an access validation that's one of the great things about SSH again now if you see this some people may say well it's interesting you have this host D and there's private key public key but how do I manage this how what happens when you do this in scale and so we've done a little bit of research on this and what we found out that the operations of SSH keys is actually in the hand of IT admins as well as DevOps

people in the past it used to be purely IT admins but more and more developers are starting to use their own keys as well so they can jump quickly to a box and see how process is working or helping or working out or they want to date they use it actually around their tool to transfer the data and again if you like in my case there's only like two people working on a couple of website no issue but once you start having multiple users all deploying creating deploying and required to rotate the keys because those keys once they are around of course they need to be rotated frequently then you can see how it ends up right you can even see

that you caning you're getting quickly in too strong of of keys actually we did some research on this I found out some research from Palo Alto Networks as opposed to some research from other companies but this was a survey that we did and you can look it up and we found out that in sixty-one percent of organizations the uses themselves the IT admins of the developers we're configuring were both great where were actually allowed to configure their own authorized keys so and actually the administration of those keys was also left over to those users idea tons of DevOps DevOps people no issue with that the issue is although more on the InfoSec site where or even actually on

the admin sites where very few people now have a complete and accurate overview of all the SSH key inventory so keys are being created people leave your organization their keys are still around maybe they're sharing keys maybe duplicating the same key and of course this can lead to specific issues nothing that we see as yes SSH is incredible powerful it's incredible powerful and incredible popular and Palo Alto Networks actually in their unit 42 from Palo Alto day indicated that 76% of the clouds were close to the exposed SSH in one another form so think about this when you're just creating keys and deploying them and you're doing is not effective or you know storing the

private key effective if some of those keys can get into the wrong hands or wall of things can go wrong anyway point I wanted to make is this how is SSH keys at this of today it's usually left up to the individuals now what does it mean what are sort of those risks and mistakes that can happen well let's talk a little bit about those so on the IT side the admin side the DevOps IDE the developer side these are some of the common mistakes the first of all is toxic growth and tuck-in trusted connection I when I watch myself and I see how often I create a new key or you know I I just ignore the previous one

it's amazing and you end up and then I have two laptops it's not anything I have two laptops it's amazing how many kids I have and just because you can quickly create them you create them do that each time remove them likely not right so you end up quickly in a very toxic world sprawl of keys and we've done surveys and leave them assessments and we found out that in some cases actually the host to key ratio was one to hundred so it's if you have one host then quickly you can find around 100 keys around this very important also when you think about SSH SSH keys days it's not like a certificate it does not expire in other

words that key that says installed could be there forever the other thing that was seeing and everyone will agree with me with this is excessive privileges I'm likely one of the one who violates the principle of least privilege access a lot I like always to have somewhere way to get in and and we're seeing it again is that people are often deploying root keys and not just fruit keys and removing them that leaving them permanent on and we even seen that some of those keys are being used by employees but not only employees but also they've been shared with employees and contractors who may walk out with them so another thing that we're seeing and then the third thing that we're

seeing is also is wikis sharing keys between ethnics putting them on a USB Drive moving them from one client or a client copying the same identical keys to assets and it can be both the private public key can be one of those but also the hosts key we seen for instance and some automation or even some some deployment pipelines that service provider was continually deploying the same virtual machine with the same SSH host key on which again if someone gets access to this could of now start decrypting a lot of the traffic anyway this is happening on the young on one side right so toxic growth excessive privileges and we key now I come from a

security operations background and here are the other things that we're seeing especially on the InfoSec side so first of all there's a lack of keys no one has the time for doing this it's I mean everyone knows that the SSH keys are in use but very few people have access to and knowing where those keys are and I mean it becomes a nightmare to start installing this or to start monitoring is if you really think about this or what a lot of people think about the second thing is intelligence right is that most InfoSec security person have no clue of which keys exposed some risks for instance are they being shared how old are they are they

has it become week actually and then the last thing is that if something would happen right then a lot of the InfoSec team a lot of the security operations team would not have a capability to the thatõs what they usually have they can say oh yes I can go in and I can start removing the keys yes that what you also are doing it's now disabling some of the connections and you could bring your company IT services an IT environment actually to hold to stop so I just want to show the two worlds one the worlds of the developers and the IT admins the other side the world of the InfoSec team another thing that we're seeing is that

a lot of people say yes but wait a minute it's okay SSA encrypted no one will know about this well actually no that is absolutely the wrong attitude the wrong attitude is thinking is is thinking that if it's encrypted it must be okay no you're playing with fire because those keys when they become weak they can be stolen by an adversary or mine insider threats they can then open the door to a back door or give another fast access and not only that if you have multiple of those keys people could start pivoting around with those keys from one host other source and by the way SSH allows forth mirroring so they could extract data it could do SCP to

SFTP things like this extract the data and you don't have a capability to respond out of this so this is what also is known often as the sleeping dragon and some organizations talk it like that it's a sleeping dragon it's over it's it's not managed and it can have big big consequences and that has happened actually so here's a couple of examples this is from the last I would say eight months there have been vendors who have issued critical warnings about their data center switches because what they shift the same default as as HK Pierre everywhere right and of course there were specific malware campaign to go malware that's that's malware that actually was doing

nothing else is trying to use those keys those those common keys that were known that we're found on some of those devices to go in and in get access and then a very very big one actually still going on is the trick bots malware and malware really that's hard to detect and this has the object is really to steal credential as well as SSH private keys from 30 and the goal again is to steal it out to take access to get those keys and then use it as a backdoor later and or salad online via the dark web for people to go and execute the full of that there's others like skip maps get map

it's very nice so it the difference between go malware trick bot and skip map is so simple go malware is just fishing actually or using keys that are known like from vulnerabilities trig bodies trying to steal it and skip mop skip map is doing the opposite skip map is doing nothing else as installing keys knowing that info SEC teams don't have a inventory and say oh I will just install new keys now enable a backdoor and because no one is monitoring this this dis maker well and I have a full time back beggarly that can start using this so these are real real threats and again like I mentioned trick bots was one of the most popular or prevalent I would

say malware in 2019 another comment that I want to make around this and I won't speed it up all but is that when you think about SSH it's not just about info SEC also auditors are very hearing about this so others get influenced by peer reviews they hearing about those risks to hearing about the path of least resistance and the internal audits now here also about this and they want to validate attestation of control and so a lot of people are failing their SSH although today no is everything lost no you can prepare and I will give you some very specific guidelines on how you can prepare the first one is think about best practices framework

you'd have don't have to do to create your own your own policies if you go to nist ir7 966 you can find in the appendix i think about 40 different controls of the old NIST 853 based and you can select them and you can view them and so it's it's very very comprehensive comprehensive paper actually from list on how you should how you can manage actually Association what you should do about it we'll talk about in a moment as well the second thing that you can do is try to assess you can there multiple ways how you can there are multiple tools that you can use to discover your environment try it and see well how many keys are there things like

that also that brings usually up some very interesting insights and then use that as a map of to start governing the lifecycle saying look guys something is wrong we have way too many keys we're at risk things like that now we need to start implementing a lifecycle pacifically about communicating out of those controls what they should do what they shouldn't do also try to put in a process where you can request them and hand actually keys out or deploy them for them automatically another very interesting thing that's a thirst that you can do is around creating an inventory so discover but that building inventory built in country about who is talking to who which keys are being used

and maybe the purpose why they are using for is it just in interactive protocol and get insights on which are active and which are non active and those that are old and haven't been used for maybe over a year erase them throw them out it will really limit immediately your risk and reducer is substantially a fourth thing you can do is of course policies right use those policies apply them against your inventory see now exactly like which keys are Aldens to be rotated which keys have been duplicated which are where is risk which other root keys make sure you take them a little bit faster things like that so do that and the last one is when you mitigate

try to do it in an automated way manually is very very time-consuming try to do an automated way there are may be some public domain or open source scripts but you also can start using tools for this and so that's the next thing that I want to talk about so again I don't want to enforce a tool here but what I want to say is start with the framework right start with the framework that NIST ir7 nine six six and then work out up to a way that you can mitigate this now if you are looking for two if you wanted to here's a good checklist that you could use and missed ir7 966 has a very very

comprehensive checklist of things that you should look for there multiple tools in the industry you can therefore the source tools there are commercial tools things like that but this is a the nest ir7 nine cities that appendix even overview of what should do you need to make sure that you can discover that you can build an inventory that you can analyze that you can provision that you can rotate those out that you have a portal through which people can request it or that you can further request you li can implement it monitor contentiously which one are active which one are non active and teittleman's and also integrate it also with your Security Operations workflow things like

alerting also all of it frequently so there I'm not going to go into details I'm just saying that be a little bit limited on time for that but it's an excellent - an excellent [Music] publication around how to make tool selection it what it actually is saying is there forth three big groups dislike a group that talks about the visibility a group that talks about building out intelligence and a group that talks about how to automate it so to simplify to think about when you wanted to the tool that gives a visibility in thousands of automation all right this is a quick use case as you will quickly go over it but it's a use case

where of course identify we have to but we did a we had a customer and major US airline and they had one point four million T's and years it's quite a lot and they failed and although they got to have big big security actually so they had that sleeping dragon was found by an orbiter and IT said there's no way I can remediate this this is this is this is gonna take a lot of time and effort and so what they did is they discovered that they eliminated started limiting unused keys they start splitting and rotating and then they stopped building in an auto provisioning process and so the end state after this is that reduced number

of keys initially it grew back but in a much controlled way but now they have a continually view of all the keys the accounts the systems and so on and so forth alright I have a couple of minutes left I will go a little bit faster but I want to show you a couple of dashboards it could be very very instrumental so the first dashboard is around knowing your assets and your key usage so I'm showing a couple of dashboard here like knowing how many hosts you have how many keys here how many unused keator our distribution this is basically the number of connections for clients so when you see like more than 20 thrusts

its means that actually it's per host is how many clients can be connected are have deployed keys to a specific host key size information do you still have all key sizes 512 and also very important what is the trend is the SSH users going up is it strongly going up how much is it going up is it even am i working on this yes no when you see a strong spike watch out it may be that one of your automation tools is going nuts right another good tool this is really very important is about risk surfacing risks and so capability to say here are specific mix control numbers filter out now the keys that may violate

through those specific controls and so finding out route orphans basically the route keys they're the public keys they're the private key is completely unknown no one knows where it is right that seems to be like a high risk to me private shared keys when when the USB Drive goes around all the keys try to find those right another sample reported around in the rotation and mitigation like how many are you rotating on a weekly basis and what's coming up try also to find those that have failed maybe the communication is not working fine things like that and then another set of report is around the lifecycle like how many keys have been requested deployed either waiting for approval or

assigned to me to others and and of course managing that is very important because admins ideate means hate waiting for an approval so make it as easy and as simple where they have requested key or it gets immediately automated and delivered to them but they can start using this immediately and don't have to wait for any approval all right I'm going fast but I mean this is the last slide here SSH keys and the automated connections they enable are an essential building block but there is a risk there's a risk that I was talking about earlier you are not you don't have to stand and and build your own controls for this there is the NIST IR 7 and 6/6

that list it's very very detailed and yeah there are tools and when you wanted to think about visibility building out visibility into those keys think about building up the intelligence about this and it automating this and of course the benefit for this is first of all yes you're going to avoid those expensive audit failures or last-minute requests when someone is not if you do I need all the key so you're going to avoid those that can be very very expensive it also can benchmark your policies and your practice against industry best practices you can set your own quality level actually and last but not least is what you're doing is you're protecting your most critical assets all right this

was my last slide if you have any questions this is my email address Bartlett Leonard's at benefiber to come we have a web page as well but if there are any questions I will I will try to watch now I'm going to stop sharing and I can see if there are any questions for this specific topic