Unstick your PAM Program - Lessons from the Field

so my name is Lance Peterman just a little bit about me you may have seen this slide before okay cuz you're not gonna get it okay I was gonna say cuz i can talk to this i have a voice that's not kind of small rooms so been an IT for about 23 years now spent the large part of it in and around the information security area particularly focused for really the last 10 on identity and access management and I kind of relished the fact that I'm a little bit of a odd duck at most security cons because identity in my experience doesn't seem to get a lot of play at your traditional security cons and you know as John

pointed out a lot of the emphasis it b-sides and most security cons tends to be on the offensive side which is great I kind of like opening up because I'm the defense so and I'm not the only defense but obviously I have some opinions that make me think that this defense is important so in addition to my current job at merck I'm also an adjunct professor at UNC Charlotte and the main campus I teach software architecture and design safe harbor I of course all of these opinions that i'm about to share our my own some of them will be war stories from either my job or from talking to other people and you can always reach out to me on twitter

linkedin I'm not hard to find because there's just not that many Lance Peterman's in the world I guess that's a good thing and I will be publishing these slides on SlideShare turn it on okay so this was actually two days ago which of course was Cinco de Mayo but how many of you also know that it was another special day and being in security you should all know this it was world password day so I'm going to attempt to do this if the audio doesn't come out right I'll kill it but we have a special message from someone I

you guys might have cheated and got a preview of that at my Twitter feed but I just couldn't resist I mean any light although how do I follow that so there's obviously a link to the talk with passwords main Liam's for entertainment value but credentials are a key part of the hacking world the security kill chain and so one of the pieces of this and it's been my emphasis and I gave a talk two years ago at this conference kind of doing an introduction to privileged access management and given the newness of the audience I'll go over some of those concepts again but the real goal of this talk was too kind of well let's be honest in two years I've

learned a lot more about privileged access management and I'd like to share some of those pieces of actual implementations as opposed to you know let's take a look at this vendor or let's take a look at this vendor it's going to be more of a vendor agnostic talk and it's going to be more about the how a privileged access management which surprisingly enough doesn't necessarily involve the technology so in doing so we'll talk about what Pam is just to do as a level set we'll go briefly through a couple of breeches specifically tied to privileged access and then we'll talk about Pam as a program and then finally kind of the money section of the talk

we'll talk about some of the keys to success as part of adopting it and then also some challenges and final thoughts so what is Pam this is what it's not obviously privileged access management these are kind of dyed-in-the-wool definitions so you know while they are a little textbook II don't consider them to be you know the Bible of privileged access because as I especially as I get to some of the breeches what can constitute privilege may not necessarily neatly fit in this definition but this is the conventional definition of any feature in any level of access to a system that basically gets you that kind of route administrator or higher order access that can override application

trolls privileged accounts or identities are obviously those credentials that allow you to create those to leverage that access and that necessarily doesn't imply a human behind it could be also be a system account or device account and then the use of privileged accounts should be managed and the password monitored when they're stored digitally and this is kind of my policy statement that comes out of this and you'll hear me talk about policy a lot because these types of programs don't succeed in your company unless there's policy behind it but this is it basically in essence the you know the essence of privileged access management so why is privileged access management important well part of it is because yeah there's been this

within my field when I go to identity conferences you know one of the favorite sayings they have right now is that identity is the new perimeter and of course they want to say that because you know we're passionate about digital identity and it is important I kind of disagree with the statement though because I don't think identity is the new perimeter actually think the perimeter is gone the old idea that you have this castle and moat mentality and you can control who gets in and who doesn't get in as a little bit gone by the wayside is you guys have worked at your various jobs or had experience with other companies date is moving everywhere so you have an environment

that's much more like this where you know based on a device that has access to dropbox it was in my enterprise five minutes ago it's now out in the wild five minutes later or you have a software as a service that you're leveraging for curricula functionality that's sitting in a data center that you have no control over the security controls in that organization and that's not necessarily a bad thing because those guys may be on the hook for doing tighter security than even you're doing but the point being is that data is moving everywhere now identity is critical still to determining what you're authorized to do with that data and that's obviously what we're going to

talk about here but I just want to kind of put that mythology to rest so we're going to talk about a couple of breaches you saw the old talk some of these may be familiar but there's relevant because I like them because they talk specific to the vector within Pam the first one is service accounts you guys are probably familiar with the breach of the department of revenue in south carolina since they're our neighbors about 15 minutes away from here 3.4 million individual taxpayers and businesses had the records compromised as part of that breach and one of the things i did love about the you know nikki Haley and she's you know obviously a politician so they're going

to try and paint this and the best terms possible so they they painted it as these really sophisticated hackers in this really sophisticated attack when probably everybody in this room knows it really wasn't that hard given the security infrastructure that they had at the time I suspect anybody in this room could have gotten into their network and the reality was and this is again why I call out the definition of privileged access this was not a high level you know system admin account that was compromised it's a backup account and a lot of people don't think of backup accounts as being necessarily privileged identities but the reality was because this account was responsible for backing

up the taxpayer database which happened to not be encrypted they were able to exfiltrate that data so that's one aspect of this the other one that a lot of people won't look at with respect to privileged access management it's probably the most critical one is local admin accounts whether it is the user themselves who has local i I've been access to their workstation or just the administrator account that the user may not necessarily have access to in the case of the Saudi Aramco breach this was an insider attack where somebody got a little disgruntled but because this company in making fun of Saudi Aramco but a lot of companies unfortunately do this they had the same password or

similar passwords for all of their local admin accounts on all of their workstations and so this guy knew what that one password was and was able to erase just running a simple script erase 30,000 hard drives now the good news was Saudi Aramco actually had some decent SEC ops from a backup perspective and they were able to get back up and running relatively quickly you know but to say that this was a disruptive attack would be an understatement and they did lose some data out of this in spite of the backups so in looking at that you know how does your organization manage those credentials does your primary user do you have admin access to your

workstation and of course everybody here says yes because I'm not going to get caught by one of those attacks right I'd love to say I'm one of those people but I got hit by ransomware about two years ago I still to this day have no idea what I clicked that created the ransomware attack so probably something that got past my bad blocker but so it happens so one of the things you can talk about and I'll talk to this later with respect to the best practices is if you can get users to give up their local admin privileges obviously giving them access to those privileges and an alternative fashion you can stop a lot of attacks I was talking with Dave

Kennedy who was keynoting in a conference that i was at two days ago and i just asked him point-blank I said if we revoked admin access to all of these users a lot of your attacks get harder don't they and he said absolutely so that's one of the things to consider and then of course briefly ebay typical spear phishing attack one of their key IT people got hit so it even happens to the IT folks their security aware too but they're not security perfect nobody is really and one of the things that I call out here is your primary account and by that I mean the same account that you use to send emails does that have

elevated access to other systems in your company it's a good chance the answer to that is yes depending upon your job role and it's a matter of convenience but one of the things you may want to consider is segregating that access whether you create one admin account that you can use against multiple systems or the real pain in the butt approaches you create multiple admin accounts but the important thing is you separate that access if I click on an email that has malware attachment in it if I've lost my admin access they can't execute what they need to do to implement the malware most likely not teed and more importantly they can't do anything with the credential they just

stole and then finally this is one of my favorite examples default passwords is the bane of any enterprises existence in this particular scenario the two Canadian teenagers found an old ATM manual and discovered that all of the ATMs built with this have a default password well guess what they did they went and found somewhere that had this older version of the ATM and admittedly it was a very old ATM relative to the standards that are out there today but they managed to find one and guess what still had the default password on it so needless to say they were able to do a little damage with that so what does that tell us obviously the threat

landscape is changing daily but a lot of the patterns are kind of the same compromised and privileged access is the key stage in one hundred percent of all advanced text now I'm not saying all of them involve privileged credentials because I think that would be a kind of a biased viewpoint but all of them include privileged elevation how you achieve that may involve credentials that's the easier target or you may be using exploits and things at that but that's obviously not the focus of this talk this is the critical attack vector for internal and external threats recent report from psychotic said forty five percent of hackers directly target privileged credentials now I think they said thirty-three percent are going

after the basic user credentials now you guys are probably all familiar with the verizon data breach investigator report ninety-seven percent of all breaches are preventable through basic and intermediate controls this is obviously just one of them in 2012 forty-three percent of respondents to a survey said they either didn't have a pam practice of any kind or they weren't sure if they did and if they aren't sure if they did one of two things is true either a they don't or b it's not a very good one so we're going to go briefly through the practice of privileged access management it's basically designed to answer some very basic questions who or what has that level of privileged access when was

it used where was it used from because if you're in an organization where you may outsource or manage service some of your access that's especially important to understand what youre managed providers are doing on your network because you not only have the potential for insider attacks from the here but they could be naughty as well and sometimes it's not even a question of being naughty you look at the target breach it was the third party vendor who got compromised and that's how they were able to get in so you could put in the best controls over your individual users and create you know a magical security awareness program you can't control the guys that are part of

your third party vendor and one thing I do emphasize and you may hear this a couple more times technology is only one part of the equation people in process and my previous talks actually started off with a horse story because I one of my passions in life is working with my horses and one of my horses got out because I didn't close the gate problem and had all the great controls in the world but if you don't close that gate and make sure that it's stuck horse gets out so and then finally this has to be a part of your governance process if you don't know what your governance process is at your company that's where you

start but one of the things that frankly I have some scars from even though I preach this from the outset when you're working with the business on this this cannot be considered kind of a one-off enrollment like okay you guys are knocking out my door i understand i need to get compliant what do I get it to make you go away and the answer is a yes we want to enroll you in our privileged access management program but be you have to continually demonstrate compliance and we demonstrating compliance includes monitoring how you're privileged accounts are managed and what's done with them this slide I won't go into a whole lot of detail on but one of the things that I want to

call out is a lot of the larger companies have some relatively sophisticated IT Enterprise models whether it is ITIL sdlc DevOps or itsm one of the critical ways to gain success in privileged access management is to get integrated with those programs what we do at Merck is when we're on boarding new application they go through that process of onboarding and then for those applications that may not necessarily be on boarded yet when they come to the change review board and they say hey I need to schedule an outage for next weekend first question we ask them is have your privileged accounts been enrolled and if the answer is no we may not tell them you know if they're

looking for an outage next week and we may not tell me what you can't do that then but that's our way of getting engaged with those guys to say you need to get those accounts on boarded so that we can manage other being accessed the other thing that I'll call out here is one of the toughest things about privileged access management is you can install a firewall on the users don't blink unless of course you block port 80 and 443 but generally a lot of security practices are designed to be frictionless to the user to be successful Pam is not one of those experiences we're going to make the lives of the people that operate these

privileged accounts at least a little bit harder but what you have to sell them on and hopefully this talk will help feed this a little bit is the why of that and so that's why those business relationships are so critical and it's not just for privileged access management it's for security in general if you can have partnerships with your business customers and constantly communicate with them but when you're getting ready to do a project you can help explain them the why of it successfully more often than not you're not going to get as much friction and enabling that and I'm not going to say that that's a perfect story because I still get a lot of friction with it but

we've also had a lot more success because of it so that's one of the things you know educate and engage your business owners whenever possible so briefly this is more for future reference I am going to be making these slides available on SlideShare but this is my attempt at a Pam reference architecture and obviously it starts with the pam piece so you have your password vault you have session management recording Pam policy management because are going to have policies surrounding some of those accounts you may have a segregation of duty policy depending upon you know what type of systems you may be enabling you're going to have discovery and policy enforcement discovery is great sometimes because you can actually

discover when new accounts come up in your business and you can automatically enroll them into your Pam product for most products and then in addition to assessing management and recording and I'll talk to the use case in a second have some function set up to do session review because doesn't you know if you record it but you never look at it what was the point of recording it and then of course you're going to overlay that or integrate that with your identity and access management layer that includes your core user identities but it also in core includes your non-person credentials and I separate this out ordinarily this would be considered part of your identity management program but

I wanted to call this out because these are their system accounts that are on all these devices that are out there that typically have a lot higher access than your average user account will but you also want to account for what those accounts are capable of doing and then finally you won't have access certification if I have root access or high level system access to a database at least within a year my manager should it be able to attest that yes Lance still needs access to that system so having that access certification process and that may not necessarily be intrinsic to the Pam tooling that you're buying so you're going to have to work with your identity management group to

hopefully create that access certification that's a critical piece of this process and then you achieve that by integrating that with your various IT resources your policy store your SRM our ticketing whether it's remedy service now or whatever workflow could be outside of your product some of the products also support their own workflow functions you need logging and audit obviously and then ideally you should integrate this with your cmdb and change management process that's really where you get the teeth behind this program and then finally there's some encouraging work taking place in this area it's already taking place and relatively mature in the saem space but in the analytic space it's even more interesting and I'll get to that in just

a minute but trying to create identity or what we call user behavior analytics out of this is an interesting potential so now we're going to talk through a couple of use cases these are the big use cases and the top one is what you would call your your classic vaulting scenario where the user is going to go to your privileged access management tool they're going to check out the password and they're going to go to the system paste in that password and then after they check the password back in ideally you would change that password that's the ideal pattern when you have to vault the more ideal scenario is what we call the session management scenario where the user

ideally doesn't even have awareness of what the privileged password is or the credential for that matter although obviously they may be able to figure some of it out but the same scenario they're going to go to the privileged access tool they're going to check that credential out but instead of going to whatever tool they're going to use you instantiate the tool from within the Pam tool and whether it's you know doing SSH to a linux server or whether it's you know remote desktop to windows server that session instantiates from within the Pam tool what that allows you to do is record what takes place and that's where you get some real bang from the buck but that's also where you get the

most secure solution because a the administrator doesn't know or whoever the user is doesn't know the credentials being used particularly the password and be from a change management perspective and this is where you sell it to your change management board they can go back through the recordings and figure out what took place in a particular session so that when you do have an incident even if it's not a security incident they may be able to figure out what took place and so this could be a real asset in their arsenal other use cases you can most of the Pam tools nowadays can use api's to manage script and patch management so instead of storing the

credentials in those scripts you know how many of us work at companies that do that today unfortunately way too many and those are typically higher level credentials to because they have to have the ability to manipulate data I've already talked about local workstations other use cases involve cloud infrastructure and SAS accounts are working with your amazon accounts also any sass products that are out there oftentimes you can use api's to change those virtualization platforms don't forget your VMware infrastructure you know the route accounts on those systems have a lot of power to them and I'll give you a little bit of a joke about that in a minute and more most importantly look at all of your hardware

platforms including the industrial systems not don't just think of it in terms of the client-server hierarchy look at your networking devices look at your skata systems even in some instances this is still kind of growing look at your internet of things the devices that are coming into your enterprise where's your risk at that's ultimately what's going to drive this and I'm gonna skip through the adoption approach here because I know we're running a little short on time the one thing I will say with respect to the adoption approach verify that they're using your privileged access management tool and I say this somewhat jokingly because should be obvious right but one of the things we had early on when we

had a managed service provider that was helping onboard our customers and occasionally they would get friction about the onboarding not the fact that we created an entry for this account but part of that process for obvious reasons has to involve changing that password and then we go oh wait a minute we've got an outage coming up in three weeks we really can't afford to change it right now so if you can just give us a little bit of time we'll get it changed and you know what we'll take care of them more like okay okay we don't want to you know disrupt you guys too much five months later they still aren't using the product so this is where you

run into challenges because pan isn't just about compliance ultimately it's about security as my friend Fernando would say and I'm not going to attempt to do his voice so what are some key to success here the first is fault tolerance obviously if you're storing all your privileged identities in one place it has to be fault tolerant that kind of goes without saying if you're a global companies should be architect or even if you're not if it's just a United States company you should architect for performance and geography so you're west coast and your East Coast should have the same type of latency or if your global same rules apply but probably even more so support for this must have

senior leadership adoption I can't stress that enough this is one of those programs that cannot start from the ground up and you walk around and go hey guys I got this really cool tool will help you manage privileged identities you're going to get the yeah whatever it's just not something that they want to focus on focus on the process first then the tooling I've already talked about consider integrating it with your cmdb if you're cmdb is up to date that can often be a challenge in some bigger companies it was for us we invested a major effort for the last two years in cleaning up our cmdb and it's a lot more useful to us now be creative with this

one size does not fit all so when you go back to that reference architecture don't feel like you have to adopt all of those processes even if you're just starting out with a privilege of a password vault that could be moving you light years ahead from a security perspective when you're selecting a vendor consider cloud implications can they talk to your various cloud vendors that you work with your SAS providers can you change passwords at that level if not you may stay with that vendor but you might consider an alternative vendor to support that piece don't virtualize your vault the why here's why now in defense of cyber Ark which is one of the major Pam vendors out there and a very

good vendor I will also act because Kevin also said several tweets later they he still considers them one of the top vendors in the market the reason they were complaining about it wasn't the tool itself it was how it was implemented it was configured using ESX so the vault itself was running as a virtual machine on a server that was shared with 50 other hosts and that was a vector for attack and not surprisingly he went after it eat your own dog food first that kind of makes standard since I would hope get used to the Pam tooling yourself start securing your own accounts with it you're going to get experience with the tooling that way you

can share it with your customer base and then finally don't think you're too small for this even if you can't afford to negotiate with a company like a cyber Ark or quest or Lieberman or who ever think about getting a vault you know you can even there's some open source vaults out there that can get you guys started but just because you're a 50-person company a doesn't mean you don't have privileged accounts you do and B doesn't mean you shouldn't be trying to do something to protect them so final thoughts here clash with teams on tooling and process is going to be common database administrators in particular but this can be true of any IT organization they have certain tools

that they want to use to access their databases with and sometimes your Pam tooling may not necessarily support them so you've got to work out a compromise on that make sure you're monitoring for out of band accounts particularly with domain administrators if you have a domain administrator guess what you can also create other domain administrators so you want to be monitoring for that and the good news is most of the Pam tools out there today will monitor the creation of new accounts so you can automatic you could either sandbox those accounts or you could auto enroll them into the pam tool make sure your priorities or where they're supposed to be if you're not if you're partnering

with the business but they're like you know what I've got a day job and you know which is true just about everybody you know they're going to say hey I really don't have time for this if their boss is calling on the phone and says oh by the way you have time for this that's how you get success on this and you hate to use that stick approach ideally you want to start with the carrot and sometimes you can if you sell them on the priority for it sometimes that happens sometimes you do have to bring the stick one way of doing that sometimes for certain professionals is making a KPI tie into their bonus

they'll guarantee you he didn't forget about us that'll guarantee their attention the cloud can mess this all up except for it doesn't like I said there's kind of a transition time in getting some of the cloud vendors on board with this it's getting a lot better taking a look at API is one of the key challenges when you are looking at an API strategy if that happens to be part of your focus is your win is access of an API privileged access when does it rise to that level and you're going to manage that based on risk and then finally with the respect to the role of analytics this is kind of a burgeoning area it's still a little immature in the

identity space so when you look at user behavior analytics you're going to look at it a much more from a more conventional security context you but you can still monitor using those analytics tools what you're privileged identities are and when things are happening that are out of the norm with those privileged identities you can use analytics and some of your orchestration tools to maybe manage what's happening with those so if there's an application that is doing things that it ordinarily doesn't do maybe you disable that service account or delete that service account and insert a new service account immediately after that hopefully minimizing disruption but it might be a way of throwing an attack and with that any

questions probably the easiest metric I look at is understanding who's actually using the tool you know so how are you you all boarded this account two weeks ago has that administrator logged into this tool now sometimes some of these accounts are arcane so they may only go in once a quarter to use this council obviously you have to have some context for that but you can look at your adoption rate from that and the other metric and it's a little bit dumber metric but unfortunately some senior execs may buy into this one is you know how many accounts you have what percentage of them have been on boarded if you're successful in getting in them

properly adopting the tool that becomes a much more effective metric but it's the one will probably pay the most attention to any other questions

huh well I was strictly talking with respect to the password vaulting I to be honest there may be some open source tools out there I am and I'm not personally aware of them yes sir it's not necessarily issuing passwords for services and that type of thing but a lot of the services now gets more into like providing privileges to like systems or other things so that the design of it is a little bit different to eliminate passwords and things like that or integrating into those environments what yeah I mean there are some issues with it I think it's a great question because that's you know it kind of comes from two vectors one is those

type of accounts are the most dangerous ones from that perspective because they can do a lot of damage that they're compromised but by the same token the more that you can get away from the password model now admittedly from a security perspective sometimes that can open up new vectors to attack like pass the hash things of that nature but if you can get to a password list solution that you know all the better you're obviously looking at hopefully some more sophisticated ways of authenticating those accounts so they can execute those functions the vendors today aren't as caught up with that yet because in spite of identity vendors telling you how much they're going to kill the password they

haven't quite achieved that yet and frankly I don't think they're going to anytime soon but we are getting some good inroads into coming up with alternative authentication mechanisms that I think will blow up support that in the future because somebody back here yes sir

ships I have not

oh okay cool oh yes sir I always try and avoid this here's what i will say that there are a lot of great vendors out that space and i will list some of them but this list is by no means all inclusive lieberman software makes a really good product cyber Ark obviously you know there was a joke at their expense here but they also make a very good product it used to be quest now it's dell dell has a TDM product that's pretty solid there are a couple of cloud-based vendors and one of them just shot right out of my head I can't remember they those are some of the big ones are out there I know some of the

other vendors a lot of people are trying to get in this space Oracle has been trying to get in the space and I'm not necessarily going to advocate for them but you know they're there are recognizing if you guys are a huge Oracle shop they might be worth taking a look at because they might have some unique advantages working within your Oracle product suite to manage that privileged access but a lot of the other vendors will will give you assistance in that space as well but those are some of the big ones hope that helps anybody else yes sir

I appreciate you mentioning Centrify because that was the one that shot right out of my head I could not remember their name Centrify is definitely in this space their value proposition is managing a lot of the cloud-based privileged identities they are trying to focus a little bit more on you know your enterprise infrastructure privileged access as well so they're definitely a vendor in that space and they're very similar I mean similar vaulting and session management things of that nature anybody else well thank you guys very much for your attention and your questions