Hardening HSMs For Banking-Grade Crypto Wallets

I do um so who in the room has ever used an HSM no okay who recognizes this room that the red curtains the floor no you're too young for twin pigs I guess uh all right so who who am I uh quickly I'm JP hello I've been doing crypto security for quite a bit uh my specialty is cryptography not cryptocurrency I had cryptocurrency even though my company is kind of in that space um I don't like to brag but this is a very good book if you like cryptography uh I'm in a company called tus co-founded in 2018 what we do is technology for banks to issue and manage tokenize security so stable coins cryptocurrency and all that stuff um so

I'm here with a colleague of mine B now was here uh and we are not in the crazy unregulated criminal cryptocurrency business we are regulated Financial firm audit every other week so you know we good guys okay so let's talk about hsms what's an HSM in the first place is it a Hardware security module or is it a Hardware security module it's a bit of both as you'll see so agenda and so on um small disclaimer there's many different hsms on the market we haven't used every single one actually we only use hsms from one manufacturer so if you know you know you will see you know not dropping names but what we say may or may not

apply to your favorite HSM but if you you haven't used hsms apparently you don't have any favorite HSM so we already did that and now let's move to yeah definition of an HSM so if you go to a vendor's website they might say it's a dedicated crypto processor that is specifically designed for the protection of the crypto key life cycle oh my God you're even more confused now right uh crypto processor so crypto is not cryptocurrency again it's crypto photography encryption signature all that stuff processor A system that processes stuff that you know has an input and gives you an output and also stores stuff specifically design we understand the production production in means confidentiality and integrity and

availability to some extent and the crypto key life cycle is the life cycle from the you know like your life from the moment we born to the moment we ultimately die so the keys is the same they're just you know small life beings they're generated they're stored somewhere they leave they do stuff they do good stuff they do bad stuff maybe they rotated they're removed because they're not doing a good job maybe they're getting too old so they retire and then ultimately they're deleted and goodbye so so here's how it looks typically you might um see hsms as you know one rack unit boxes like that uh a big box and people will call that an HSM even though

technically what's the Hardware security module is the small piece inside that is doing that stuff I mentioned you have oh my God what did I do I killed oh no okay uh on the right hand it's a PCI Express from Factor um HSM um so it's kind of equivalent from Factor these one are connected to the network and this one just to DPC Express and you also have small USB form factor HSM as well so why would you use an HSM why would you worry about buying such a box which is quite expensive admittedly uh so the whole point is to store secret stuff and to keep that stuff secret so secret we mean secret Keys private Keys what do you do

with private keys and secret keys well you sign stuff you decrypt stuff and you might encrypt stuff if you're in the symmetric key world uh here ma is message authentication code it's kind of signature where you have the same key on the two sides so you can admire my drawing skills here uh it can work as follow it can be very basic you send a request to the HSM hello HSM you have this key K I have this message I send you the message and you send me the signature of the message with the keyk I don't receive the key the key stays in the Box it stays in the secure box the HSM is Computing the signature and I get

to learn the the signature so here you might say oh that's cool but if if everybody can do that if everybody can use the key then what's the point so you need some Access Control will get there um the whole point of an HSM if your company is if all your computers mobile phones doors fridges are compromised then the attacker will not be able to extract the key which is super secret and if you lose the key you die so you put it in a very secure box and even if all of you know Russia mosad CIA is in new organization well these guys might be able to do it but in different ways uh but if an purely remote Network

attacker has compromised all your virtual machine hypervisors kubernetes they should not be able to get their hand on that key that's Theory you know and you know insecurity is always complicated so why we we do that when do we have keys of high value well blockchain of course you know how it is you lose the key you lose the money they steal the key they steal the money if you're an individual with like 100 Franks dollars Euros you don't want an HSM you don't need that stuff but if you're a bank and manage literally billions worth of bitcoin or whatever is your favorite currency then you want to be very careful you want to buy one of

these boxes uh code signing as well okay you probably some of you probably know the Microsoft world whereby you create an application if you want it to be signed you want it to be recognized by Microsoft Windows without having the popup manware you need an HSM to sign your binaries to sign your installer and whatnot sometimes database encryption um even though what you have in the HSM is not the actual encryption key but a GE key encryption key and lastly last but not least pki public infrastructure um certificate authorities where the private key could allow you to compromise you know every single website on the planet like Google IBM and whatnot so you want this private key to

remain private because you don't want to be able to force certificate to for signing certificate and and so on okay how do you talk to an HSM you know do you just you know send a letter no it's works with computers Network PCI Express USB CP I as usual nothing new here I will skip okay security better for me I guess I let you bra as well so the s in HSM is for security so we do have security feature inside and you know bear with me there's a lot of stuff not saying everything is good but there a lot of stuff uh first of all you can share your HSM with your friends well if you have friends and if you have

an HSM um you have this concept of partitions think of it like user accounts with different isolated m spaces to store your stuff to do whatever you want uh but sharing the hardware you know spectr mown this is sometimes called partitions and so you have a partition that is for you so you have your credentials for each partition has their partition security officer so for each partition you may have different types of users with different types of capabilities uh airb back Ro based access control so you have different roles you know ad me users and other stuff that have different capabilities like this guy might be able to update the firmware this account might be able

to change the password of this of this other guy this guy might be able to add new cryptographic objects in the HSM so like other a key this other guy might be able to create a certificate and declare it as trusted so I'm not going to go into the details it's quite hairy but you know it's quite hairy there's a lot of stuff that get go wrong as well if you don't configure it correctly of course that's the problem with security pkcs 11 who already heard of that okay trust me you don't want to you don't no don't go to internet don't open the specification you will hate me for that it's terrible it's horrendous it's

C like syntax but it's extremely important is the way uh you communicate where with hardware token with Hardware stuff that those crypto is the way you format messages where you put each bit what this bit means how it's interpreted so when you talk to the HSM you have your client code and you can use the same client code for every single HSM because they all talk the same lingua FKA which is called pkcs 11 so PS means public key crypto something yeah but yeah if you don't need that you can get on with real life you don't need it last but not least uh fips so the federal uh us uh line of standards and 140-2 and

now 140 three is about physical security temper evidence temper protection and to some extent temper resistance uh what it just means is that you get some guarantee that if someone you know let's say that's an HSM if you get your hand in the HSM you can just open the box and you know put a sensor and just dump the memory you have a bunch of security controls like if you have your system and you kick it it will not like it it will reset and zeroize all your keys if you plug the power uh in some some ways I don't remember what we did but sometimes we're not really nice with the HSM and it didn't like it

and it reset and if you try to open the enclosure likewise they will it will detect it and say oops no sorry I forget everything um that's interesting way of operating and it also guarantees you that the crypto the buil-in cryptography is correct uh correct according to the nist FIB standard like as cdsa and so on however so if you been doing software security exploitation you know like aslr aslr is everywhere oh your exploit Works without aslr who cares well welcome to the '90s the hsms often don't have any aslr or or dep you know so no exploit mitigation maybe the reasoning is yeah people won't try to exploit this in the first place you know the attack surface

is so strong so who cares about um just there lot of Legacy code and um yeah there might not be remote attestation so remote attestation means you get the response but you get a cryptographic guarantee that the response actually comes from the certain program so it's quite difficult to put in place it's the strongest Guaranty you can have um because if you receive a response you say okay but how I'm sure that the the code they run was this exact piece of code which I've been verifying with my you know Suite of tools you can do it as I will show later but it needs some some hacking uh now what's inside uh you might find a well a board with

electronic component um some power PC chip there might be some arm there might be some some other stuff but the ones we work with they tend to have power PC so you big indan and whatnot uh some crypto accelerators so crypto accelerator what it means it means on the system own ship you have some uh Hardware silicon so not fpg but really hardcoded but you might have fpg but the whole point of this crypto accelerator is to accelerate crypto to make it faster sometimes I wonder if it's really faster because it's super slow but um no the interesting thing is that when it's in Silicon you can't patch it if it's in fbga you can patch it if in in software

you can patch it too uh there's typically some Linux I've seen some stuff with BSD but most of the time it's uh it's Linux and you have a bootloader which may or may not be secure you know how bootloaders are and there's crypto software libraries when I was young and naive and and even more stupid than now I saw that in Hardware cryp Hardware security modules all the crypto was in Hardware I was like ah so cool strong crypto you know Hardware defenses and whatnot well if you look up on the open source credits they use op SSL they use the Linus kernel they use um P the same stuff that everybody uses so the how

Ware security doesn't mean that everything is done in Hardware so don't think like me and there's a firmware so the operating system of the of the appliance it can be updated the updated the updates are signed sometimes they might even be encrypted but if you see that's once encrypted you can maybe look in the past and see if two years ago it was also encrypted and you can do other stuff um yeah what crypto is it doing so all the boring crypto that everybody does itic curve signature tic curve def man encryption decryption a all the fips sweet essentially and very important there's a true random number generator uh so true just means it's nothing fancy it just

means that it's not deterministic it takes entropy from the analog world for the temperature of the room from the I don't know whatever you can predict and feeds it into a drbg deterministic s random generator that gets you unpredict table random bits that you can use to generate keys and if this sucks then you're dead uh it doesn't matter if heuse an HSM or not if the randomness sucks then all your crypto is dead and breakable so single point on failure and so on um very important thing and for the rest of this talk so you buy an HSM and you can send its request to sign and decrypt but you can't really customize you can't run

your own you can run you cannot run your own games you cannot run doom on the HSM you cannot add your email server but some hsms allow you to extend the firmware with some custom software module so when you will send a message to the HSM it will first be received by your custom module and then you can run your own code in the HSM which is pretty cool because you get a guarantee of Integrity that other people will not be able to modify your code because it's been signed it's in the HSM it will not be modified and guarantee of confidentiality because whatever happens in the HSM stays in the HSM so it will replace the the init

function of the HSM you can add your own stuff it has to be developed in C the C language boring c99 um people don't like C but I love see there's the SDK of the vendor but you cannot run you know your huge uh I don't know PES cluster or SQL database it's limited to 8 megabytes but it can do essentially pretty much everything that the firmware itself can do it can talk to pcs1 objects it can access all the crypto Primitives so it's pretty cool okay um now how can you make the HSM a bit smarter a bit more resistant to attacks you know defense in depth and that kind of stuff uh because the

obvious problem the elephant in the room is that oh you might use an HSM but if someone compromises your laptop and gets the keys the pins the password to use the HSM and they can sign whatever they want uh and then it's just security theater so how do you deal with that and also PK cs1 I told you it's a piece of something uh and there's some implementation flaws occasionally and there's also some design limitations that you know some G shots that you really need to know uh and you can shoot yourself in the feet um so you can look at this advisory about the UB UB Keys uh there was some news about Keys recently

there was a paper by The Ledger guys in 2019 about um about HSM and they found some pretty scary stuff um so there's tons of things that can go wrong you know now you know the last few days everybody's like oh supply chain attacks and uh s bombs and all that stuff and you saw before that there's Linux there's a bootloader an open source bootloader there's open SSL libraries there protocol buffers all all the stand out stock so to speak but you know firware updates they're like every every other I don't know every two months every six months and it takes a while to get it f certified again they might not have the little

version they might actually have very old versions uh very old like 5 years old you know um so you can look at the internet see okay you have this open assess version from 3 years old how many how many uh High sity CVS do we have here uh and also what worries me well not worries me what I know will happen someday uh publicly is what I call cross slot attack cross partitions because if you share the same Hardware by two software isolated components inevitably by the laws of physics you will have leaks depending on what you do so you will find a way to extract Keys If The Enclave or the the parti one is doing

one thing the parti do is doing another thing they will of course be S Le so you have to be careful to what you do you have to be careful to what you do on different slots and that's why many of our clients are like uh no we prefer our own HSM we don't want to share an HSM with other guys we don't know uh soain issues I told you you can run your own code in the HSM yeah very cool but if a janta or some random people can back door your code then they have root access on your HSM and that's not a good news well not for you and of course the

random number generator issues so I'm always scared when I see a black box RNG when I see a box saying us oh don't don't worry we have a good RNG but you know Roa couple years ago this infin on chips they were found to be vulnerable because they tried to optimize the r key generation it was optimize it was much faster but it was much less secure uh now what about custom modules tons of things can go wrong we found lot of weird bugs and you know you have a f system and um if you put a slash in a name you know for whatever reason at some point it would crash like okay interesting uh we had the case I don't

know if you mention it yeah if you lck too much does logs but if you L too fast it will not like it maybe the io is freaking out so it might crash also what we find quite early on um if you log a line that was more than 80 characters it would also crash you know but not gracefully you know s fault um and also the crypto interface something that works on one version may not work on the next version of the HSM because maybe they made some de different design choices that they did not document and sometimes you have documentation you're like yeah you read the documentation you do exactly what's in the documentation it doesn't work you

have to be more creative and find something else but I guess it's the same for many lines of product um okay hard running how much time we have like 10 minutes 15 okay that'll be fine okay hardening that that's really important um if you've been using HSM all your life this might be boring to you but if it's not the case that will be potentially very useful I will talk a bit about Cloud hsms at the very end of the talk so I told you that pkcs1 can be you know Annoying but you can filter the request you don't instead of directly exposing the pkss 11 you can you know sit in the middle receive the input and

add your own validation passsing White thing whatever you want to do or just block it all together so if you don't use an API function why expose it in the first place just ditch it and it saved it saved us I think couple years ago there wasn't bug in one of these functions and we didn't use it we blocked it and it was not exploitable um oh alsoo vendors can be created the the other own extensions to this pement interface of course because you can um what I found really cool so in an HSM you have a lot of attributes um when you add a private key when you add a secret in the HSM you can tell the

HSM okay this key can be used for signing this key can be used for encryption this key can be used for decryption this key can be used for whatever else so you want to put the permissions to give the least privilege if this key is only to be used for signing you only authorize it for signing and like likewise you can authorize keys to be extracted exported wrapped so you want to be as tight as possible so on the right hand you see these attributes that that's the um pkcs syntax cka something value sensitive private modifiable blah blah blah blah blah and to do that you have to understand what each of these mean um

it's not always straightforward so I mean that's basic it hygiene but sometimes people don't have time or don't know that they can do it so if you have a cust module inside you can say no I refuse to work if I'm not in a safe space if I'm not with the right attributes I want if I don't if I don't have the right label on this key and so on and so on um okay told you hsmr for crypto but you can run other stuff you can put your software it's sing complete you can do what the hell you want so if you want to put business Logics ad just or I will only sign uh messages that

start with 0xab 3 you can do it um if you say I received a request but I want it to be signed by a guy with this public key um you can also do it you can say actually what we do when we receive a request to avoid a single point of failure say Okay um we only accept to sign this blockchain transaction request if your request if your message is approved signed by three out of five of these public keys that I know and I have in my HSM it may not be clear I I repeat so it's this Corum out three out of five for example validation whereby you receive a request please create a blockchain

transaction to send two Bitcoin to this address so it's not yet the blockchain transaction it's just the request I want to do this but this request has to be cryptographically signed by number of parties and the their public is identity are hard in the HSM so you kind of simulate you know what blockchain people call multi signatures or multi party something or social signatures but purely in the secure environment of the HSM and what is great is that it's complete agnostic of the blockchain because you receive a request and you don't care whether it's for salana Bitcoin ether or whatnot and people who see the as a public database they don't know that this happen in

otherhood so you know privacy and so on um but of course if you use a custom module if you get a remote code execution in the HSM then you win but you cannot get an an LP previl escalation can you guess

why you cannot escalate to root if you are already already root yeah it turns a rot yeah that's not fun um so this leads us to World base Access Control you can Implement your own you know layers of roles you can have different layers of fun uh for example what we do we receive a request to sign something and it has to follow certain patterns certain roles like for example the the amount is limited to that the address has to be wi listed and you create a number of rules but the rules might have to change so you cannot hardcode the rules in the HSM so you're going to receive roles signed by admins and you will verify the roles using the

public keys of the admins which are in the HSM so there lot of keys lot of parties it's uh kind of becoming a headache but the upshot is that you can do quite complex stuff in this small 8 megabytes okay boom um H something that I like um so you're familiar with the concept of end when encryption so n2n encryption is from the consumer to well from to the from the producer to the consumer so the textbook example is you know messaging so signal WhatsApp end to end because I send you a message it goes through number of servers of WhatsApp and whatnot it ends up in your device you device the Crypt it I want you to

read my message I don't want you I don't want anyone else to read it so only you can decrypt it that's producer to Consumer if you look at um Telegram you send your server you send your message to bav durov to his servers in Dubai you know everybody and their dog can see it you know you know who and then telegram say okay I'm going to forward it to the recipient but they have this huge database of all the messages of everyone yes but the ux is so cool okay so you can do N2 encryption in your HSM because typically you may not talk directly you talk via Gateway Bast and horse so you really want to implement

producer to Consumer encryption in both ways and you can do that five okay sure um one important gu line is at least from My Philosophy is I don't want to use software components that I cannot see inspect analyze um very don't like black boxes because if you do a black box you know if you have no accountability then you can do what the hell you want and put back doors and there's been a lot so you may want to minimize the usage of the blackbox components from the HSM firmware and instead use your own of software yeah especially well for the prng you don't have much choice for the RNG for the entropy but you can add on top of it

your own drbg okay I will skip that part because it's quite complicated it's very short anyway uh at black hat we present that um although the point is that you know we have multiple Rand hsms they are stateless but sometimes you would like some statefulness you know for you know anti downgrade for certain stuff uh but you can store L of data so what do we use you know when you have this kind of problem you use trees Merle trees and we actually use something called red black trees for our specific problem so look it up online it's on my website um okay to conclude the best practice best practice Yeah update your software uh tighten the pks1 attributes like I

mentioned before uh there's something called secure Channel which is not enabled by default so sec channel is some kind of secure Channel between the HSM and you and you has the first degree connect connection not necessar the the consumer it may or may not be completely secure it used to be basic Anonymous def man so not regret but it's better than nothing I guess uh yeah minimize dependency if you add custom code and if you have the dependencies you need to you know hard code them vendor them in your GitHub project don't pull dependencies from some random people on internet I mean we're not we're not PM um okay access control key management blah blah blah blah blah um Cloud hsms

okay you have a question can we do all this stuff with Cloud HSM the answer is no sorry because the cloud HSM they're managed by the cloud provider you don't have rout access on it um you typically share the HSM with other people you don't know uh you cannot run your custom software you can custom software module on the cloud hsms there's one provider they all you to bring your own HSM in the DAT data centers but then it's still your HSM so go back to to square one um and also it depends what you're up to of course but it's hard to get a guarantee that the HSM you talk to the cloud HSM

you talk to is an actual HSM so there are some ways some vendors give you some way you know pki y see why to do it but I still find it a bit bit scary and a bit like okay I just want I use an HSM I use a cloud HSM okay so let let me conclude with that so hsms and in HSM software is very powerful but you know more power more time to do stuff more responsibilities and uh more code to review and maybe more headaches at the end of the day so you need people to maintain all that stuff you need to be very very careful with the sdlc so what we do for example

we have when we build our custom module okay that's very important that's the last thing I will say uh when we build the the software we put in the HSM we don't build in a laptop well just you know by myself we have an auditor in the room we have a specific laptop for that we have a specific doer container for that that was created with the auditor with only the minimal stuff and we make reprodu reproducible bills and we get a certificate that this binary was built from a given version of the code and we give it to client and the certificate is done by an external auditor and this certificate is also temper proof because

it's signed by by the Swiss Federal Confederation Authority um so it's not completely perfect but it's putting the B pretty high and that's all I will say and I thank you very much for your attention [Applause]