What’s The Big Secret? - Graeme Simpson

good afternoon everyone I'm going to um make a start hope you're having a nice day so far um my name's Graeme Simpson I work for IBM um specifically I am in the IBM Cloud division in the SEO team so it's part of my job along with a great deal of others to make sure that IBM Cloud's kept secure um my talk today I'm going to be talking to you about Secrets Management on the scale of a cloud service provider so what is a secret a secret is a small amount of protected material which unlocks something more uh interesting so more precious but actually it's the secret itself then becomes as valuable as a thing that you were trying to

protect and that can take the form of all sorts of things as it says up there API Keys encryption Keys things like kek and kex key encryption Keys before we go much further into that I'd like to tell you a quick story that my primary school days so at primary school they sent us a maths challenge a maths puzzle to try and solve and that was the number of handshakes between an arbitrary number of people so these two fine stock image gentlemen are shaking hands two people one handshake if you want to try that between yourselves and count you can make sure that that's true two people and one handshake so I'm going to show you another slide in a

second yeah one handshake and I'd like you to try and work out how many handshakes I know you all expecting to be able to snooze You' got to do a little bit of thinking so answers shouted out please how many handshakes between five people I can see some C Sing I've heard 10 and 25 right answer is 10 hold on that was very fast so back in primary school I came up with this equation for doing that X2 - x over 2 so 25 - 5/ 2 is 10 I thought at that age that I was clearly a genius my teachers may or may not have thought I was a prodigy but in reality I think I

probably just peaked very early so how can we use that how many people how many handshakes when you've got this many people the answer of course is I have no idea I am not going to count all of them but it's a lot so why have I told you that story it's to give you a sense of scale when you're talking about IBM cloud or any cloud service provider you're talking about an enormous number of things connecting to other things hence handshakes but my story is not quite accurate because when we're thinking about handshakes like in that mass puzzle everything shake hands with everything else and reality that's not actually true that's not how it works

with cloud services you get clusters of things together depending on the scale or size of particular service or thing that's running in the cloud so typically for a smaller thing you might be looking at say 20 or 30 Secrets covering encryption and interconnections and things like that for a larger service you might be talking about 500 or even a thousand secrets that you're managing just to run a single but large secret so of course what we've got is secret handshakes hope you like my joke there so IBM then what sort of scale are we talking about exactly so IBM are the sixth largest service provider in the globe sounds impressive Amazon are number one they've got 30 something per

of all of the cloud power or in terms of money at least IBM we're a mere 2% so you know a much smaller player really but even 2% 2% of the global Cloud that's an enormous amount of cloud infrastructure and Cloud power we've got and that map represents all of the data centers we've got around the globe the ones which are particularly bad for anybody that's color block are the multis Zone regions the sort of double bubble ones and those ones are actually three or even in one case four different data centers all within a fairly close geographical area and they're connected with super fast pipes in between all of them that's to our handshake

problem and that means you've got the speed of a single data center but the resilience redundancy of uh of the split so one of them can burn to the ground and the rest of them can keep keep going so yeah IBM huge numbers of cloud uh Cloud power here's a lovely picture of the data center I don't know how many people have been around a data center I've been around a few I've never seen one that looks like that color coordinated lights lovely it's easy when we're thinking about this sort of thing to consider the data center quite an abstract thing okay so the data center you know the secret goes wrong well it burns to the ground

in the greatest Calamity but it's over there okay it's you might lose Facebook or Netflix or something for a while but how much of that is an actual real world impact and you have to remember that they're actually connected to things in the outside world as well so it actually has a real life impact and this picture I've chosen specifically relates to IBM very well because IBM while we're a lot smaller we specialize in uh those Industries which uh require higher security or higher regulations depending on the industry so that might be uh financial services that could be um the airline industry or Healthcare and each one of these has a different regulation so in America uh it's hipper for for

healthcare so what uses secrets well of course there's lots of different things and it tends to sort of build up in layers so right down at the bottom or actually at the top of my list we've got bare metal so bare metal represents an actual server or some tin which you know through IBM Cloud you can you can purchase or for any cloud and then building on that you've got hypervisors and then VMS or containers storage could be native storage it could be databases uh that sort of thing Cloud object storage and of course you'll see TLS everywhere stringing the whole lot together backed by certificates to make it secure and then right on top of that

typically you've got your application layer so Secrets right the way through so onto the what do we do about the problem now typically you would use a Secrets manager IBM has one all the other Cloud providers have one you can also buy open source packages which you can use yourselves and what what does this actually do for you well it's somewhere safe it's a safe place to go and put all of your keys so that you can manage them and it's not just a secure environment they provide lots of other things as well so for example if you put a certificate in there the system's clever enough to recognize that it's a certificate and it has an expiry date so

once it gets to within 30 days of the expiry you can get an alert so you don't miss something and of course these things come in various different options so at a very basic level you get multi-tenant environments so lots of different customers sharing a single platform you get a dedicated space to yourself but it's still built on top of a shared platform if that's insufficient for you if you need some dedicated space you can of course get your own you can get ones which have redundancy built in uh you know including globally and uh the top end you've got things which are backed with something called an HSM our Hardware security module and that is a

dedicated bit of Hardware that bolts onto the back to keep those Secrets extra safe so you get this nice management interface this is sort of um this isn't a screenshot this is just nice picturesque version but it gives you full life cycle management and of any of the secrets that you've got through obviously use right through to um deactivation and destruction of the keys uh at the end you can also do something called keep your own key or bring your own key which is where if the customer feeling particularly uh paranoid they can generate the keys locally within their own infrastructure and import them directly into the secrets manager rather than rely on local generation uh so then

obviously you're entropy injection that sort of thing so that's uh quite useful um and then we get into things like notification I've already mentioned about certificates but you can be not notified about all sorts of actions whether it's just adding key or even accessing it and of course depending on the industry you're in you'll probably uh require some sort of audit capability and the audit capabilities configured such that um the customer themselves even cannot go in and delete audit audit records the whole thing is stored so that anytime anyone accesses a key deletes a key Etc it's all uh it's all audited for anyone else then to go and look at these things aren't just passive they

don't just sit there and uh store the keys for you and then give them out on occasions they also can do some pretty nifty tricks so the first one we've got here is rotation so some Industries require you to rotate your keys it's good practice anyway at a minimum you want to R rotate Keys every 12 months some Industries specify 90 and even seen 45 days if you went back to the example I gave at the beginning where you've got a system with 500 to 1,000 Keys you imagine trying to manually rotate those every 4 5 days it would be an absolute nightmare it would be like uh painting the fourth Bridge you you'd just be um

starting again just as soon as you finished and the chances are extremely high that it would go wrong the chances of you managing to complete that successfully every time are very low whereas if it's built in in an automated fashion not only is it likely to be extremely error free but it's extremely fast the whole thing can go out and rotate the keys very quickly seamlessly and no human has to get involved so no one actually has to see the keys much safer and of course it's not even just on a routine basis that you might want to rotate your keys if unfortunately you discover that one of your keys has been compromised you can slap that big red

button and all of the keys you've got under your command get rotated in a very short space of time therefore hopefully sorting yourself out for that particular potential vulnerability and then lastly my favorite is checkin checkout historically if you had I don't know five administrators all needed legitimate root access to a server then they would all have their own username and password they'd have dedicated accounts so that you could track reasonably who is doing what but that means you've got five permanently enabled privileged access accounts and that's a risk five risks what you can do with these key uh Secrets manager systems is you can check out a username and password when you want to go and

administrate a server you use that username and password you go into the server you do what you need to do legitimately you've been given access to that particular key when you finished you come back to the secret manager you check it back in again at that point the secrets manager go then goes off to the server and changes the password so the password you were using two minutes ago is no longer valid valid so there's no permanent privileged access available to that so what can go wrong if you don't do it properly I thought I'd go through a few examples that have been uh in the news so most people will have heard of the Equifax one because it was so big

some 145 million customer records something like that were exposed and at the root course of this was bad certificate management they were hacked they were hacked on internal system which um had vulnerability monitoring on it and the vulnerability monitoring was unable to spot the problem because a certificate had expired it had expired 10 months prior to the hack that meant their vulnerability monitoring was unable to monitor the traffic you not you can't say for sure it would have spotted the attack but it certainly couldn't when it couldn't view any of the traffic BMW they used Secrets management storage Unfortunately they used a COS bucket it was of their own design it didn't have any of the nice automated features and

somebody managed to make it public which meant all of the keys them were accessible for them they were luckier a researcher found that probably before anybody else did so they were able to sh that one up Microsoft did something similar sorry Microsoft and know you're here today they did it the same but they stored their keys in GitHub as part of a larger deployment I think it was something like 32 or 38 terabytes of AI training data was what they intended to put out there Unfortunately they threw in a few keys for good measure they were made public and again for them uh they were saved by research of finding it but my favorite I think has to be Toyota

again they were lucky they didn't have any particularly ill effects from this but again their keys same as Microsoft soft was stored in GitHub but theirs were stored for an amazing 5 years they had an active key there ready for anyone to discover for 5 years so not only were they dared with their key storage but simple uh rotation would have uh reduced the problem by a good four

years Turtles all the way down now this is a bit of an odd title this one and this is from a presentation I think from 2015 or 16 where they originally proposed this um it's not a very good description I'll come to that so I've got a database I have a secret I need to access the database VI my application so I've got a secret so I need to protect it so I'm very good and I encrypt it so now I've got an encryption key to the secret to the databas and I'm good so I put my encryption key and I put it in a Secrets manager safe base now I need to get into

my secrets manager so I've got an API key to get into the secrets manager I need to look after that so I wrap it in encryption key now I've got an encryption key I need to look after so I put it in the secrets manager you can see we've got a bit of a loop so what do we do about that well that's the nice thing about Cloud infrastructure it's it's not just an anonymous box there is something at various levels depending on where you are in the stack that the cloud provider is able to um provide some recognition or sorry a cryptographic proof of the identity identity of that particular thing so even though it's a container

it's ESS blank that container is actually able to prove its identity so when it reaches out to the secrets manager and says I am who I say I am can I have that first secret please then you um then it can do so in a way that it's it's confident it's okay so you found your your bottom Turtle if you haven't figured that one out that's the Earth is on a giant turtle what's under the turtle another turtle so on yeah so you need something like that to greatfully and find that bottom turtle so if you haven't already how do you go about um using or deploying some sort of Secrets management well the first thing

you have to start with is is assets why are your assets and then for how you protect what keys because unless you go and find all those keys you can't possibly go and protect them so um so yeah you need to go and find all your assets and then these are the sort of headline attributes of a fairly decent Secrets manager I won't go reading through them but it will pick count the one from the bottom cuz bizarrely enough I have seen this go wrong with other systems never log your secrets so if you're auditing you say who's done what when what have they done and you don't actually put the secret in the audit log you'd be amazed how many

people count that one wrong so where does IBM come into all of this well like any cloud service provider we eat our own dog food so we provide these tools for our customers to use but we also use them ourselves because it's good practice to do so so these secrets managers propop up you know an enormous amount of our Cloud infrastructure and that is the end of my talk I hope you found out of interest there anybody good any questions

yes yes

um that's a good question even though you as a cloud you kind of move around potentially your compute unit as it were is is movable it's still on some sort of underlying device and we do have devices that use things like State attestation to confirm that absolutely nothing has happened to the v a sort of BIOS level and then you just build up from that so even though it's moving around you've still got a a chain of trust back to the route which you which you can trust so yeah it is both mobile but there's still a chain there

[Music]

um so the answer is is is well it's kuna twofold one is there's an element of trust uh but you don't have to entirely trust us or trust anyone else because we are audited by Third parties so you can go out and you can see that IBM have been checked we don't have access to that particularly if you use things like keep your own key we simply can't get access but you're right at an underlying level you know someone has to have a key somewhere that you can do something with so there is an element of trust but that trust is backed by third party verification um uh an audit and we are constantly being audited uh myself in my

job I review uh the parts of the clouds that that go up to you know go that make up IBM's Cloud so I work with the development teams everything thing I do pretty much has to be auditable quality even if that's a a WebEx call is video recorded for later audit purposes and that is shown to thirdparty Auditors so that they go through and they say yes we can see that you track everything yes we can see you're doing everything properly and you are doing what you say you will do including making sure that we can't get into access things that we shouldn't

um yes obviously that's you know we come into IBM bias here hashy C's very good we just bought it yeah absolutely yeah like anything yeah you can still get it wrong yeah hashi's very good we've been using it for years and uh this was just stated we bought it about a month or two months ago yeah but you can still go and buy the open source or can use the open source version

okay uh oh I have no idea you can email me that one I'll uh yeah I can go back to you on that yeah okay all right thank you very much