Guarding Against Protocol Subversion at Coinbase

welcome everyone and thank you for attending my talk guarding against protocol subversion of coinbase

one second hmm sorry about this my name is mark Nesbit and I'm an application security engineer at coinbase coinbase is a digital WAP digital currency wallet and platform where merchants and consumers can transact with new digital currencies like Bitcoin and aetherium some of my recent priorities have been security support for systems that integrate with cryptocurrency networks and also security assessments and mitigations for assets that coin based supports my talk is going to have two main parts first what this protis protocol subversion mean and second to live real examples of attacks that we caught in the wild the first is a 51% double spend attack and the second our ERC 20 administrative privilege takeover so first what does protocol subversion

mean most everyone here is likely already familiar with the CIA framework for security C for confidentiality I for integrity and a for availability examining a system for these three properties gives a great start into understanding the security and threats against the system I'll describe how cryptocurrencies work in a bit more detail but for the time being it's important to realize that a cryptocurrency is a network of nodes that communicate to one another according to a protocol the nodes on the network store a copy of the blockchain which is a public shared database and the network protocol allows nodes to communicate state information about the blockchain nearly every blockchain has an authorization model based on public

key cryptography the state of data in the blockchain can usually only be updated when the proper digital signature is provided an example of this would be sending bitcoins from one person to another the sender must authorize this transaction and which is a blockchain state change by signing this send a wallet is an example of software that is built on top of a blockchain wallets hold private keys and could submit transactions to the blockchain as you could probably imagine the the blockchain is a shared public database anyone can choose to build a wallet application on top of a blockchain we'll take a quick look at the CIA for wallet applications confidentiality a loss of private keys

is a violation of confidentiality that you often hear about in the media when you hear about hacks in the cryptocurrency space if the private keys held in the wallet or leaked anyone can authorize transactions for actions controlled by those keys integrity manipulating the recipient of a transaction another interesting wallet failure mode if an attacker can manipulate the recipient of a transaction prior to that transaction being signed there's no need for the attacker to have access to the private keys a large component of my day-to-day work is ensuring confidentiality and integrity in our wallet systems much of this is traditional security architecture and back-end security engineering work for completeness I've also given two examples of availability failure that

you might have heard about both of these for getting the encryption password and not backing up the hard drive will result in a loss of keys it's not really an attack vector as the attacker doesn't gain control of these funds but they are availability and therefore security failures this talk is about subversion of protocols not wallets so let's apply the CIA method to the blockchain itself confidentiality it doesn't exist I defined the blockchain as a shared database thus it's entirely transparent and there's no confidentiality jumping to availability this is not a huge concern for anyone except for the protocol designers concerns about the availability of block chains have driven much of what's known as the scaling

debate in cryptocurrencies if protocol design makes the resources required to run a network node too expensive it may impact the availability of the blockchain information which could have many negative impacts on the network after all everyone needs access to this data and if it isn't available it's kind of a deal-breaker violations of the integrity of a blockchain is what I mean when I say protocol subversion the integrity of the blockchain has recently become a bigger focus across the industry as I said before the block chain contains shared public data and is often called unhackable in popular media descriptions whether or not you're a blockchain expert I think nearly anyone in the room and a security conference

can laugh a little bit at the word unhackable just like every other system of watching a certain properties to it and I'm going to spend time describing ways blockchains and cryptocurrencies can have their integrity attacked as I mentioned before I work for coinbase a major cryptocurrency exchange exchange is make an ideal target for all kinds of attacks including and especially protocol subversion you may not recognize this guy but you've probably heard an apocryphal story about him this is Slick Willy Sutton a bank robber and when asked why do you rob banks he replied that's where the cryptocurrency is I think that's how the story goes anyway exchanges hold a lot of cryptocurrency on behalf of their

customers that's an avi enough reason obvious enough reason for them to be good targets slick really hit the nail on the head with that but beyond simply being where the money is there are a lot of other characteristics of an exchange that are attractive to an attacker liquidity and volume being able to trade one currency into a different currency can be very advantageous to an attacker speed exchange is often credit funds to deposit to to anyone depositing on the exchange in a relatively short timeframe and allow for nearly instant sends an attack could therefore happened very quickly the ability to get in and out fast is obviously a good thing for an attacker remote interaction an attacker

can execute an attack from across the ocean maybe even from North Korea and lastly in some cases anonymity I want to take a second to talk about this many popular media descriptions of cryptocurrency seem to ascribe it with some quasi magical anonymity which it doesn't have this is especially true if you have an authenticated session with an exchange such as coinbase coinbase strives to be the most trusted exchange in the entire cryptocurrency industry as part of that we're heavily regulated and a large part of that regulation involves the lengths we go to ensure that every customer on our platform has gone through a rigorous kyc AML process kyc know your customer so that means knowing

their identities this is important for AML anti money laundering any exchange that doesn't have these strict requirements would obviously be more attractive to potential attackers so I'm going to talk about two major examples of protocol subversion the first 51% double spend attacks as I mentioned before a block taint is a shared database stored by all nodes on the network and accessible to anyone for this database to be useful there must be a way to update it block chains are append-only databases and are updated in batches of transactions each batch of transactions that is added to the blockchain is known as a block so we could visualize the blockchain as above with block n being the most recently

added block on down n minus 1 to the very first block we could expect that a block n plus 1 would shortly be added but who defines this block the database is shared it's distributed so there must be some way of reaching consensus among the network participants about what constitutes the block the answer is that it depends on the crypto currency this is one of the major defining characteristics of different cryptocurrencies and a lot of newer cryptocurrencies have innovative methods for adding to the blockchain ripple and stellar have validator nodes which use a voting consensus protocol to determine which transactions will be in the next block EOS has regularly elected nodes known as block producers and they

defined the block Tesla's and Cardno work with proof of stake where a node is chosen based on the proportion of the network funds that it controls which is known as its stake and lastly the ones people are most familiar with Bitcoin and aetherium the node that first successfully solves a cryptographic puzzle defines a block this is known as proof of work it's known as proof of work because the solution to this cryptographic puzzle has to be brute forced which takes considerable computational effort this is called mining mining a block is when a node discovers the solution to the proof-of-work puzzle if there ever any competing versions of the blockchain these competitions are resolved by deeming the version with the most

accumulated work to be the canonical blockchain I'll explain what I mean by that this diagram shows the blockchain tilted 90 degrees with the blocks separated block and plus 1 will be added on top of the other blocks as before each block contains some number of chances the green arrow indicates the canonical blockchain suppose some node solves the cryptographic puzzle it broadcasts the block that solves that puzzle to the network and all the transactions in the block are added to the canonical history of transactions that is - the blockchain but suppose the second block is found simultaneously by another node how do we decide which block contains the transactions that are to be added the rule is that the nodes on the network

define the series of blocks with the most most work as the canonical history so if either of those two blocks gets another block extending on top of it there will be more accumulated work on that branch which makes it the canonical blockchain the other block is not added this rule means that there's never a case where a block is truly finalized in the chain if enough work decides to extend from a different block once that branch has out work the rest of the chain it will be the canonical history this situation on the slide is called a reorder short for reorganization and the grayed out blocks are known as orphaned blocks let me repeat a key fad any actor that

can outwork the rest of the network is the sole arbiter of which among all possible valid transactions are the ones that are actually added to the history so if there was some kind of network instability where blocks were not always immediately shared with the network after they were found or if some actor was deliberately holding black blocks they that had they had discovered we could see something like this where the blocks on the left are hidden but if they are shared with the network the network will switch over to these blocks as the canonical chain because they have more accumulated work orphan abox that were previously the most recent additions to the chain because of this

potential for instability in the most recent blocks anyone receiving a transaction should wait for several blocks to be found after the block that contained the transaction to lower the chance that the block containing their transaction will wind up being orphaned an analogy that I found interesting is that the most recent blocks are like are like fallen leaves in the fall they can blow around and change and shift after a while they might get waterlogged and don't move nearly as much and even longer they decompose into mud and after a while they become soil and potentially eventually sedimentary rock you can adjust your risk by adjusting the number of blocks that you wait until you consider the transaction finalized this

is known as the confirmation requirement and each recipient of a transaction decides for themselves what they want their confirmation requirement to be so imagine we had the following situation where coin Bates supports a fictional coin McCoy abbreviated muh suppose the confirmation requirement for muh is three blocks coinbase also supports Bitcoin BTC and muh trading any customer of coin Bates could have the following intention create a transaction T that sends a coin the coin from the customers wallet a coin base wait for three blocks after which the confirmation limit is achieved and coin bits will credit the funds from transaction T to the customers coin base account the customer can then sell the muh for BTC and send the BTC wherever

they like this is a completely normal pattern of behavior for a customer to take let's imagine however that this customer is actually an attacker an attacker with the ability to outwork the entire rest of the muh network the attacker will create transaction t sending some amount of McCoy onto coinbase suppose t as quickly included in a block by some miner on the network simultaneously the attacker will create t prime a second transaction notice that T prime sends the same funds that were sent in t funds from address a1 T and T prime could never exist in the same blockchain together as soon as one is included the other would be an invalid transaction because the money had

already been spent however the attacker is mining T prime in secret remember we've assumed that the attacker can outwork the rest of the network meaning the attacker can produce blocks faster than the rest of the network in order to do anything with the muh on coinbase it first needs to have three confirmations the attacker does not sit idly by and continues to secretly produce blocks the network also produces blocks but unknown to anyone is not keeping pace with the secret blocks produced by the attacker now there are two confirmations for transaction t3 confirmations are reached the attacker is now credit with the muh and can sell it for BTC which could then be sent off

the coin based platform so the BTC has left its off the coin based platform it's any attackers control remember nothing seen publicly thus far as anything out of the ordinary however if the attacker reveals secret mining activity by broadcasting blocks in the network the blocks have more accumulated work than the existing top three blocks so a reorg will occur with the attackers blocks now representing the canonical chain the top three blocks that were previously seen publicly now become orphan blocks and noticed that T was in these blocks T is the transaction that the attacker used to fund the deposit to coinbase there is now no longer a transaction funding the attackers coinbase account in the

blockchain anymore but the Bitcoin the BTC has already been withdrawn this would be an example of a successful 51% attack if this were to occur the ability to do this is directly related to how difficult it is for an attacker to overpower the network the more work being put into solving the proof of work puzzles on the network the more difficult it will be to overwhelm the network note also that the danger of this attack comes when you accept a deposit directly from the attacker in this example Bitcoin was provided in exchange for muh the attacker if the attacker can't get something irrevocable in exchange for the vulnerable coin this attack isn't viable this is one of the

reasons that an exchange is such a good target for this attack liquidity 51-percent attacks are easy to spot if you're watching one each block is each block is identified by its hash which provides a unique fingerprint to the block to observing hash changes will help you detect reorg if the hash of some block at height n changes from what it was before there must have been some kind of reorder small reorg ZAR normal they happen on a regular basis primarily driven by the fact that many nodes are attempting to find blocks and there is some amount of latency within the network so multiple blocks can be found simultaneously and eventually only one will be in the blockchain

any reorg deeper than the confirmation limit of the recipient will allow for the possibility of a successful 51% attack you can examine the hash of the blocks at Heights n minus 1 and minus 2 etc and get a sense of how deep or severe a reorg was you can expect inspect the contents of a block to look for the presence of T and T Prime you would rather you would inspect the contents of all the blocks involved in the reorg that's the smoking gun that a reorg is mrs. money that was sent to one place originally is effectively clawed back by the appearance of the new blocks and the reorg recently etherium classic a cryptocurrency was successfully 51%

attacked because this is an asset coinbase supports we had monitoring systems in place which alerted in real-time to the attack allowing coinbase to pause interaction with the e.t.c blockchain I'll talk about how this attack unfolded the et set the EPC network is minding its own business mining box as usual adding transactions to the blockchain blocks continue to get produced as the cryptographic puzzle continues to be solved BAM all of a sudden seven new blocks show up out of nowhere and these seven blocks don't extend from the most recent block but dig down five blocks back we're running four blocks twelve hours later it happens again six new blocks show up this time Orphanage five previously found blocks

I'm calling both of these incidents practice attacks because neither of them contained a pair of transactions T and T prime which is the smoking gun that an attack is underway these are just reorg however we had never reserved observed reorg of this depth on etc' even though it would have been premature to label these as attacks on their own since there were no double spends it still got our attention then three hours after the second practice attack there was another reorg this time 74 new blocks showed up all at once orphan in 57 blocks and within those blocks we observed a tea and a tea prime that's shown here this happened on a Saturday night our on-call

engineers responded validated this alert and paused our interaction with the e.t.c blockchain it's important to realize that pausing interaction with the blockchain is sufficient to protect you from this attack because the attack requires that you credit a deposit that happened on the chain

twelve additional attacks happened after we had paused interaction with the e.t.c blockchain larger amounts of e.t.c were stolen in each of the attacks the total amount was over 1.1 million dollars we estimate the cost at no more than $250,000 it was wait is much more likely that it was closer to 50,000 several exchanges later identified themselves as the victims of the attack and we posted a blog post with more details about it on the coinbase blog if you're interested in learning more now we're going to move on to ERC 20 administrative privilege takeover smart contract a year ce 20s are a type of smart contract and you can think of a smart contract as a protocol

of an in and of itself it sets up a system with rules interfaces and interaction logic smart contracts are our group arbitrary code deployed on a blockchain most tracks state within whatever system they establish ERC 20 is a smart contract standard to create a token this means that an IRC 20 token tracks balances and allows for sending and receiving of coins many ERC 20 contracts have superuser privileges that means there is one or more administrator roles that exercise control over core functioning of the contract well dive into an example of this here are two snippets of code from a smart contract they're both what are known as modifiers which must evaluate to true before a

function can be executed on the left we see that the sender of the transaction must be equal to whatever the owner function returns on the right we can see that the value of the boolean variable paused must be set to false now we have a function pause that has both of these two modifiers as you can see this means that only the owner can call pause and only if the boolean variable pause is false the pause function will flip that boolean to true now we have the function transfer a fundamental ability of any type of token system notice it how it has the modifier when not paused meaning if the boolean paused is set to true no

one can transfer anything so we can see how there's the ability for whoever is the owner to entirely shut down all the transfer functionality for this token subversion of an e rc 20 contract can be accomplished if control over these super user privileges is gained by an attacker I want to take a second to clarify this is a very different type of protocol subversion than a 51% attack for 51% attacks the protocol is the blockchain itself and how things are appended to it for thus a 51% attack is a blockchain level integrity violation in the case of ERC 20s the blockchain it's functioning entirely as it's designed but the integrity of the smart contract system

is under attack subversion of the smart contract is the act of including a transaction in the blockchain that causes integrity failures of the original system I just gave an example of the pause super user power but some super user privileges provide nearly unlimited authority over the smart contract system this might sound like a crazy thing to add to your smart contract I'm going to talk a little bit about the motivation for why such authority exists smart contracts are meant to be immutable building software is an iterative process this is a fairly fundamental issue for smart contracts to grapple with how do we resolve this introducing the proxy structure it's a smart contract that works according to

the following setup one attempt to use your own internal logic to execute incoming transactions and change state if you can't do that fall back on a contract that is specified in some variable that you hold this fall back contract can have all the logic of a system in the case of a token where you have a transfer and yet balance function all that the proxy needs to do is maintain the pointer to the token contract along with the authorization configuration required to update that pointer if you want to update the system because you may have decided the transfer and get bounced don't provide enough functionality you can deploy some other contract that has additional token

functionality included in it such as pause and mint the proxy contract still points to the old token contract but it maintains the author and functionality to change this pointer we can see here a function upgrade to if admin is a modifier and if that's true this function can be called and specify a new token contract if admin calls upgrade to we have now upgraded our system to use the logic in the dead beef contract on the right not the center contract without changing anything except for a variable in the proxy contract integrations with the proxy aren't disrupted so our customers aren't disrupted by this upgrade however you can see here that the variable holding that contract address with the tokens

logic is extremely important and I'm sure most of you can see where this is going it sure would be a bad thing if a malicious actor could set this variable if there was some way to update the proxy to point to the bad contract attackers would have the full functionality of the bad contract at their disposal in this threat model the attacker could choose which contract the proxy relies on so we can assume that the attacker would have written and deployed whatever code is necessary to execute this attack in this case perhaps the attacker maintains all the properties of the existing token system with the single difference that the attacker can invoke drain balance which

perhaps allows the attacker to steal anyone's tokens this completely breaks the protocol defined by the original deployed smart contract allowing the attacker arbitrary control over the system coinbase contracts have been probed in this way we can monitor for these similarly to how we monitor fit for 51% attacks by parsing the state of the blockchain and observing whether any of the criteria we've defined as subversive or dangerous has occurred two transactions probe the u.s. DC GRC 20 contract on Saturday u.s. DC is a stable coin that coin base operates these contracts attempted to take administrative control over the proxy contract these transactions both failed when they are evaluated by the etherium Virtual Machine because they didn't have

the proper authorization to make these updates this means they didn't result in any state changes to the US DC smart contract however if these invocations hadn't failed the entire nature of the US DC smart contract system could have changed these were the two functions that were attempted to be called and evaluated as a failed transactions change admin which would change the owner role from the previous example who can specify the proxy contract and upgrade to actually changing the proxy realized this attack can only work at the administrator of the proxy contract has lost the Keefe's of the contract or if there's a failure in the proxy contract that allows a non administrator to take administrative

actions coinbase takes this responsibility very seriously and has strong controls over the keys and reviews the code very carefully in order to guard against it this type of probing is normal and to be expected and this isn't the first time we've observed this there was a smart contract system where attackers were able to successfully abuse administrative privileges for a kick coin here's the super user privilege that allows the owner to mint tokens to any arbitrary address here's the super user privilege that allows the owner to destroy tokens at any address the owner keys were compromised and the attacker called destroy admit many times effectively causing a transfer from any address to an attacker address without

changing the overall supply of kick my slides will be available so here are some sources for anyone who wants to look more closely the point here is that we can clearly see that as was the case with 51% attacks actors exist that are capable of successfully executing these attacks so to recap I defined protocol subversion as integrity violations of the underlying systems and I talked about the threat model unique to exchanges I discuss 51 percent devil spend attacks with an example of one in the wild with aetherium classic I discussed ERC 28 administrative privilege take over with an example of one in the wild with kick coin does anyone have any questions there's a microphone if you have

questions and we can bring it to you there's a question

hi do you monitor like the risk of a specific coin to be like hijacked by a 51% attack so like for example there's a high risk for a theorem classic then four bits going correct is that something that you tracked absolutely yep question there so to my knowledge it seems like coinbase is planning on adding more and more assets how do you actually scale how you're actually kind of monitoring all these assets and looking to different unique risks that they face right we have to have a system that will allow us to do that based on based on some rules based system that we can monitor and determine the risk as you pointed out of each individual coin

and we have invested the time and effort to build the tools that can do that so that we can properly support coin bases ambition to support more and more tokens on the platform two questions related one of them is is there any such thing as a privilege escalation attack in which someone manages to gain administrative privileges or something equivalent or equivalent to access but without using the key of the admin account to get there and the follow-up question is have there been any implementations and smart contracts of a permissions model that has more than two levels okay for your first question yes you don't need to use there's there's more than one way to get administrative

controls there are basically two ways one is to pack the administrator and masquerade as the administrator that's the example where you steal the private keys the other is simply if there's a authorization logic failure in the contract some of these contracts can be complicated and it's quite possible that the code written doesn't check they make the proper authorization checks in all cases in which case maybe an administrative action would be exposed to a user in that way

not necessarily it I see yeah it's it wouldn't be like a privilege escalation vulnerability within Linux or something like that but it would it completely depends it's going to be very specific to the authorization model within that smart contract and so it depends on the nature of the bug I can't think of any off the top of my head but I would be pretty surprised that there weren't many you had a second question

yeah yeah there have been so the question to repeat for anyone who couldn't hear because of Mike have there been other authorization models other than just privilege and non privilege and yes there have been there are different smart contracts that provide different roles for almost anything you could imagine so you could have a positive role instead of owner you kind of owner pause or mint or any of these any of any of these roles that that you just define as an author of the smart contract and you can write a modifier that checks that and then you apply it to whatever functions you want those authorizations to be scoped to probably time for one more question

they're like public repositories of vulnerabilities associated to these different code bases yeah vulnerabilities are cataloged I'm not sure if I can think of a best place to recommend where they would be catalog off the top of my head but a lot of times when these vulnerabilities are discovered because smart contracts are immutable the contract may have to be redeployed and the entire system may have to upgrade to the new deployed contract so it is actually a serious issue when something like this happens okay I think that's it thank you everybody for coming appreciate your attention