Deploying Two-Factor Authentication to Millions of Users

welcome everyone this is the 210 presentation on deploying two-factor to millions of users please join me in invite in welcoming Manuel here to have his presentation I'm really looking forward to this one so take it away thank you and hello everyone is anyone here like building or planning two-factor authentication oh wow uh more hands than I thought all right so one of my first project at Pinterest was to build two-factor authentication and you know between design implementation deployment I think I've been spending quite a few months just thinking about this space and during this month I learned about fiber - I got really really excited about hardware security keys and then something magic happened like one on one

side I had this experience this inspiration from my job that deploying to a PHA and fillion power to protect so many users and on the other hand I sort of realized there was no open source alternative to hardware security keys and so together with a group of friends we co-founded Salukis where we are making solo solo is this little security key it's open source and it's the first one to implement 502 and today I think there is another exciting theme solo is probably the biggest key you have ever seen I am back to the talk so today today what I would like to do is to share my journey at Pinterest with you and what we learn on deploying

two-factor authentication and so this is going to be our agenda for today after an introduction and a design we are going to focus on the two main flows that are logging and enable two-factor authentication and then we're gonna and with some juicy juicy details so I always like to start with a definition so two-factor authentication is an authentication method where the user is granted access after she presents two or more pieces of evidence also known as the factor so why do we want to factor authentication I think there are two kind of users are the good users for which 2fa is a way to protect their account and then we also have the bad user those that you know if

I use the password there can't get pwned compromised and then they complain to us to the company and so for us as a company to a phase of safeguard in a way we can go back to those user and say hey there is an instrument please use it the next time of course 2fa is also some cons user can be locked out and so while you build two-factor authentication you definitely want to engage with your growth and support team because those are going to be the team that will handle bugs and user complaints so authentic Asian factors they are organized in three something you know something you have something you are and I think there is a

little bit of mismatch between what we learn in a school and what we have in practice so the first factor is usually a password or a social login or maybe a link sent via email that prove that the user owns something the second factor so we have codes codes can be sent via SMS TOTP or even backup codes and the reason I put them towards something you know is that code are short and they are short for a reason because the user need to remember those for even for a little amount of time and because of that this code are fishable so if we really want something strong that proves the user has something we

should probably use push notification or security keys even better and as you can see there is no something you are here and this is because fingerprint and recognition today our client only verifications as there is no server challenge and so we cannot really use them for implementing two-factor authentication and by the way please absolutely don't send biometrics data to any server because if you do so you go to hell so let's see the design so how do we build two-factor authentication well we mentioned already we have login and enabled flows and enable is typical Aberdeen's in the setting so only some user can find it so less problem for us the critical point is that typically at

least in a bigger organization this two flows can be owned by different teams and so you will have to engage and to keep both of them synchronized both require implementation on the backend and API luckily well the login of course you must have it on every single platform your support luckily the enable you can choose for example at Pinterest we choose web only someone like someone else like for example Instagram I think they did mobile only so it's really up to you for us so when whenever there is a product feature that is really visible to user we are to engage also with the design team and in this case because it's international also the

internationalization team and finally my recommendation is given the size of this project you should also try to engage with the marketing and PR such that you can get the best out of your project this is going to be the high level architecture we are good we are going to use so we have clients API and users service or database and this all need some sort of minor modification to support 2fa and then there is going to be service either it's a micro service or a complete service that implements the to a very specific logic for example generation of the codes and verification at Pinterest in terms of clients we have i/o Android I and web well our API is an API and the

user service typically we have three Java services internally and backed by my sequel database in terms of two-factor authentication service we chose to partner with an external service which is of--they by Twilio okay so if I were to go back and build 2fa again I would definitely definitely start from logging and so the idea being I could fake some data and pretend some user already enabled 2fa and then I can implement logging and the reason seems pretty straightforward but it wasn't evident to me when we build it the login should be it's a must on every single platform and on some platform especially on mobile the deployment cycle can be slower than web and API in addition we

made a choice to only support codes and because of that we can only implement a single login screen and so that simplifies a little bit both the user experience and the implementation this is our experience today and thanks to RT we can have essentially two experiences in one on the left side yeah on your left side yeah you can see the SMS on the right side you see the user that has installed the OT app we receive a push notification and we grab the code from the RT up this was particularly important I think for us because we use a lot push notification for let's say social reasons and so mixing social and security is not really a great idea for

user experience and it's not show in this representation but also the same screen supports backup codes so if the code is not seven digit which is the office standard then it's a backup code and we support us or that one so in order to blend two-factor authentication we need on one end to modify existing login in points and on the other end to create a new one to verify the second factor so this is an example of a modifying an existing end point so the user the clients re will send the credential to the API the API will verify the credential at that point if the user passes the first authentication then we can check with

the use of service whether he has two factor authentication enabled and then imperil s under code through the two a face service and return an exception are for one two factor authentication required at that point the client will show the two a face screen and after the user inputs the client will send the code code is verified and an ok is returned to the user so in our experience the backend is relatively smooth in the sense yeah you have to implement API endpoint but that's what you do basically every single day the Fernan is where we are the issues the issue so in principle it seems easy because you have to catch an exception and show that to a face cream the

problem is this exception is typically caught very deep in the network stack and so based on how many engineers have been working on their on your network stack how old is that Network stack and especially who is working now on two-factor authentication you may encounter difficulties so in order to hold out properly what we did was so that's what we do always we get all new code behind experiments we have a solid experiment framework that used to haul out fergus of lis every single single new feature we typically deploy to employee first so that we can have some internal testing before release to the public and finally we were monitoring for unexpected regressions and that's

unfortunately happen in our case so in our case Android went out and despite there were no user with two-factor authentication the new code was causing issues with the facebook login on Android and yeah so because of that we had to close the experiment wait for a new release and one month passes very quickly so this is why I highly recommend focusing on having a solid login experience as soon as you can so next we are going to talk about the enable two-factor authentication the idea is to have this option in the settings where the user can enable two-factor authentication so my tip here is it's a relatively smooth process it's a flow but many Engineer in your company

are used to implement flows don't waste time in little details for example there are chances to spend hours and hours deciding what's the best number formatting what are the country calls that you have to support so if you are the one that implements the flow that's probably something you can focus on but if you are relying on an external resource a highly recommend to provide very detailed specification so that they don't waste their time and again the choice is to choose your favorite platform so that you have a single point where you will roll out to a file feature we have two flows actually in the enable they enable and disable I leave you as an exercise to design them

on the architecture should be pretty straightforward but they enable we want to absolutely verify the password before we enable to have a we want to input and possibly verify a phone number or scan a QR code whatever you decide to implement and then most importantly you need to force the user into a fallback in case they don't have their second device so the fallback the the typical solution is to provide randomly generate and backup codes and we know that any nobody is actually printing them but it's again a safeguard in case the user lost their phone we can say hey we give you the backup code why did you write so on the disabled we typically in the industry

verify the password I don't know why technically I think we should verify for 2fa but nobody does that and so we don't do it and finally you should never provide an override function not even possibly to your support team because then they tend to use it very easily so disable into FA if you no longer have access to your account should be painful process by design where you verify explicitly the identity and so I highly recommend if you have to build it yourself to go through the Google on the Facebook or the orthey experience and for example we rely completely on Aarthi for that some little details about enable flow so it's zero it's really

important to send always security image to the user to announce that they to FA has been enabled or disabled in our case we also made a choice to implement a unique phone number in the sense that if a user has a phone number and a second user uses the same phone number this phone number is going to be removed from the first user ok this is because we cannot assume that no phone number don't move around and this was by the way at least when we built it the same experience that Facebook was doing I don't know if they change it recently but this is I would say a common practice for the rollout well we had an

international rollout so I highly recommend to go through the experience in a language language that is not English this is because the from a user perspective the flow is pretty complex and so you want to make sure they can follow all the steps and they don't have we are messages in - air language - English and finally my recommendation even if you do a big launch maybe we press maybe with a big marketing push roll out very slowly because any possible error in a security feature can cause a big blow up and so if you look not just for Pinterest but also for Instagram Facebook when they launch two-factor authentication you see that the message is always something

like we are starting to haul out to FA or we're starting to deploy to a PHA in the coming weeks or months okay so I highly recommend to go slow and make it and do it well and now the juicy details so the first question often we have is what do we have to protect with two-factor like of course we want to protect the user login with username and password but many other things are kind of a question mark and so we kind of try to give a universal answer think into two FA as a contract that we do with the user and in this contract we tell the user if you enable - a PHA will never

get you a tea or the game and because of that essentially we want to protect with - a PHA every single point in our stack where we generate a new assistant and so literally what I did was to grab generate as a stock and or the equivalent function in our code base and figure out what were this endpoint and for example I found some API endpoint of all login like for example with Twitter that were not in use and so this rule will allow us to very to protect consistently across the board one important one is the password reset so to reset the password the user should verify to a PHA and another big one I

think is social login so in Pinterest if you use fail if you login with Facebook or Google you are going to be requested two-factor authentication again the motivation we mentioned before there is a somewhat user experience motivation if you'd like so imagine the user that enables two-factor authentication and then they want to test it so they log out and they try to log in again and so you cannot expect in that moment that they are gonna use username and password maybe they just use Facebook because it's more convenient and so in that specific time they are waiting for to a Fae that they want they are eager to test yeah so one possible optimization would be nice in my opinion if we could

know if the user has two-factor authentication enabled on the social on Google or Facebook but unfortunately the hoth specification doesn't support that so yeah maybe in a near future security keys and so today you are already about security key that's amazing Fido too also protects against phishing not only a county cover and also as you know now there are those open source security kills so how cool is that as you heard today web bottom is a standard so that's really exciting almost every major browser to their support so it's really a good time to build support for security keys and on mobile the situation is a little bit different so I can require a little bit more work on

your side especially on iOS Android is moving forward pretty fast so let's assume we built two-factor authentication on with the codes only and now we want to implement security keys so what we have to do is a new screen where we also allow the user to change the to a ferry method and that's for sure what we need on web on mobile that's a big question again it requires more involvement and more resources and so it turns out that to the best of my knowledge at least only Google supports security keys of mobile and also companies like Facebook prefer to force you in push notification or or typical and finally for the enable it's really

important to support multiple kiss per user so not unlike AWS and again to support of to force of fold-back method and so if you have to choose I highly recommend to look for Facebook experience because I think it's really well done so let's predict the future for our second you're gonna have bugs where are these bugs so I think the enable you are not gonna likely change it a lot and so you can expect essentially its bug free which is great on the login side instead login is gonna change a lot because new feature new login method that offer even better growth experiences and so the problem is that those people in the company might be newer and and they may

not know everything about two-factor authentication in our experience enforcing that every new endpoints is protected from two-factor authentication was relatively easy because security could monitor for new endpoints the vice versa assuming that every new screen is gonna properly work so it's going to properly display and it's going to properly send the code and verify was not really the case and so we had a bunch of bugs in in this experience and so I think you should keep an eye on those and in my opinion there is not really a very good solution is just about constantly educate your developer and especially the new one about the fact that 2fa exists and yeah we see in

the next slide another thing so marketing and PR it seems that this is a I think you all agree this is a pretty solid and big project especially for the security team that typically doesn't interact a lot in the in the product and so I think it's really important to market internally by showcasing for example we do brown bags and we do we also have a one minute demo where we can showcase whatever we build in front of the entire company and then it's highly recommended to employees to enable to assay and again if if you think to the newer employees and they go theme they can at least know that two FA exist and by the way I'm

super excited to share the tower CEO band suburban has to a fail in able on his Pinterest account so it's pretty cool and then market is eternally so certainly right attack blog post that's pretty easy that's a new company blog post if you have one you can connect to these people they are typically really excited to work with technical people and to sort of bright something geeky between a less technical voice and in our case for example we release the blog post in five languages that for me was especially cool because there was also Italian and I'm from Italy and finally PR PR might be harder because 2fa is no longer a news but you can probably

combine it with other security features and so if you search for Pinterest the fact about indication you see that the PR message was actually a combination of multiple things that we released together and this one finally is a slide that I copy and pasted from one of this internal showcases that I mentioned this is just to show you how many people collaborate and in first release and of course many more came after that and so in conclusion well to factor I think two-factor authentication is great but it's a big project so put some effort in it log in is gonna give you addicts but enable will be pretty smooth and so my invite for you is to go and build it and

before I close I just want to let you know that corner my co-founder is so lucky this year and is going to give you away some free security keys for you to try it out and if you want more of course you can get as many as you want thank you very much [Applause] yeah if you have any question me yeah and by the way if you have more questions later on I'll be at the Pinterest booth like in the sponsor area so feel free to okay so the question is how many what's the adoption rate and how many people decided to use of--they so unfortunately we don't share publicly the adoption rate what I can tell you is

that in general there are two people it's two sets of people one is the people that have decided to enable 2fa and the other one is the one that we recommended because for example their account was taken over or we proactively protected it and so the second part is the the big bulk so people tend to not enable 2fa by themself at least of interest which is sort of a personal use experience and they prefer when they get burned essentially so why do we decide to build vasu salty yeah so I think the decision was mostly well on one end cost in the sense that cost of maintenance I would say and also the experience that

he adds in terms of fighting for us against 2fa and so like in a certain sense it would be so the maintenance part of 2fa can be more costly the technical side I think is doable relatively easily especially because there are so many open libraries that you can use so I think for us and literally was support and yeah maintenance in the future this entire process of the presentation about six months including the release and marketing yeah yeah so the question is whether we have different set of user some user high risk or high profile and whether we do something different for them in terms of two-factor authentication so yes we have advertisers so advertisers spend money

in Pinterest it's a it's a double-edged sword because they are higher profile in the sense that they are more valuable but they are also need to be we need to be more careful because if you protect a Pinterest user and you lock in the account there is no financial impact but if you protect an advertiser and you block the account then you also lose money so you want to trade those two things or at least use very careful measures in terms of 2fa they both have access to to FA and we haven't done any specific campaign to enable to a PHA but I know that that's a request that we get through the sales channel and so

essentially they are more aware generally speaking than the regular peanuts thank you hi you mentioned you wanted to make it painful for support to disable this sort of like once it is enabled for an account and not just to be like a flag they can turn off could you expand on what that looks like in general like what kind of process you have someone go through once they've lost their Authenticator to turn it off yes so the let's say if you have of--they you can try a sense early the spirits too haughty and it's basically very similar to Google where you say I forgot my phone pulled a website and then they go through identity verification if the same things

come like through the Pinterest support then we essentially ask the user to enable Aarthi so we tell them to go through the Artie process one thing that happened in a singles isolated case was a partner like one of those advertiser where someone essentially set up two FA and then they left the company so the new person didn't have process in that case because we have a direct sales channel we essentially verified our SADS and so we kind of bypassed but it was really are sort of more like an exception because we knew all the steps otherwise they go through RT yeah do we have time do it we are at time now so thank you all for attending just

remember that there are still I think they are still selling t-shirts um it goes to charity so you can even pick which charity you want hack for hack the hood hackers for charity or EF f thank you very much for it and I'm the behalf of besides SF we have a gift for you and everybody thank you very much thank you [Applause]