Access Control with Concierge: One Tool to Rule Them All

dr. Gregg origin from autocar access control with concierge before we start another reminder to thank our sponsors Fitbit and hacker one thank you to fit fit and hacker one and all the other sponsors whose name show up on your badge so without further ado car tech run garage ugh thank you hello everybody hopefully you're having a good monday afternoon i'm here to talk about concierge and how we do access control with it to introduce myself I am Karthik around rajan i am the lead security engineer at a tapar I'm a huge proponent of the info ops framework we've spoken about it before where we're running develop security and I deoxys a single unit with security at the forefront I

have hosted broadcasts in the past I used to be a daily host of the infosec daily podcast I've hosted the think of the infosec hot button which had an early death unfortunately but most importantly I am a dreamer and idealist I think about grand solutions sometimes will succeed at building them sometimes I don't but today we're going to talk about concierge and my I am dream and how we're trying to live the im3 we're going to talk about the motivations for building concierge the different design considerations that went into it as well as some of the implementation details of how we did things and why we did them that way as well as a recorded demo

because live demos always go amazingly well cool so a bullet in the beginning of age 100,000 years ago we had 101 credential we had users walking by openldap single sign-on with Google Salesforce Expensify SSH github sly our own SAS product that we have for customers users would log into that as well and it was a nightmare to manage them we had so many credentials we didn't really have a single source of truth we didn't know their users were what they were doing things like that which obviously meant if you were a new employee you had to spend the next two days onboarding yourself there wasn't an easy way to figure out what you had to get access to

you would have to file a jira ticket and somebody would have respond to the jury to get to prove it somebody in IT ops creates you and like five different things somebody in DevOps creates you and four more things somebody in business operations and three more things and so on and so forth and there wasn't like a single place where you could go to and say hey I need access to these 10 things to do my job effectively and please give me access to these things so which essentially man people were really confused IIT was really confused I was really confused about where people who are getting access to what and it was just tedious off

boarding was in fun either we had as i said 101 credentials and when you have to offer for somebody from all of it within 30 minutes on the day they leave it's not the most fun time we had google forms that tracked all the services that we had in created three different generate ages for different ID people and develops people and sales people and finance people to delete users from all these things it was painful it was really painful all the thing wasn't fun either the auditors would come and they would be like hey show me all the things you've done with your users and make sure that your forward your control processes and i would spend the next

three days pulling up these logs it was a lot of fun it was the best part of my life obviously we didn't need a solution for this at all but the biggest part of it all was that there was no single source of truth we didn't know what users should have access to we were really know who their managers were we don't know what department to belong to we will see them and open out app and we'll be like all right that's their name let's go search them in on a chart directory although she has their managers aren't the same anymore let's make sure we change them or their role has changed well let's retro actively

remove access to things and stuff like that which is an ideal that's not how you should be running access management you shouldn't be catching up you should be proactive you should know exactly what's happening with everything else so I've been down the spot I was like is single sign on the way to do all this maybe we should just have one login or auto or pink identity or any of the thousand solutions in that space just just throw money at it and solve it for us and it seemed an appealing option all of these disks all the whole bunch of problems for us it let us manage users according to certain roles it let us manage a

whole bunch of apps you could centralized stuff like Google and Salesforce and stuff like that without having to worry the provided sort of the same security controls you know they would provide sings to factor they would let his IP whitelist and stuff like that but the biggest downside is that we would how to manage active directory this is the true story Consuela is the devil mascot she's her dog that's always watching over everything the moment i mentioned ad in the meeting that we all had she threw up immediately as soon as i said we had a setup Active Directory Consuela threw up in the corner of the room and I could have taken that as a

sign that I was in the wrong path but i did not i console is just sick it's okay she's just scared of things so me we've been forwarded ad but before we set up a tea before we did all that we had to think about a system to actually manage these different moving parts and there are a lot of moving parts when you set up something like this so that's where concierge can man concierge main goal was to sink users automatically with the source of truth and whatever the source of truth was managed 80 users and groups so that you don't have to actually go to the Windows machine to do it manage access applications depending on a

user's role or department or whatever else it might be centralized SSH access and other types of temporary access that users might have provide a centralized location for user access others so that I don't spend three wonderful days going through lots of various different things most important to fall never ever already be into a window sir that was actually the biggest goal for concierge I don't want to already be two things so like I'm just going to build something that prevents me from already being into things there were a few design considerations that went into building something like concierge mostly it has to be extensible if i'm building the solution if we're doing destroyed then anybody should be able to go in and

change it to mark to their environment I didn't want to build something specific for a tapar I wanted to build something that anybody in the bird with the same kind of problems could use it had to be modular what if you didn't use some of the same page our systems what if you did not want to use one login what if you don't use salt or any of the same things that we use well it still has to work for you it shouldn't just break because you don't use some of the same things that we do it should provide a restful api I mean great we have a web app that we can manage users send but if

it can't integrate into anything else it's completely useless we have to have it integrate into other systems that we have in place and it has to perform the same way across all the different users we have for it most importantly add in wanna reinvent the wheel a lot of the problems lot of the problems in the space have been solved by somebody else and I wanted to look at what problems have already been solved and just do the things that other people haven't done or I'm not confident to the solutions that somebody else can provide so the biggest thing we started with was a single source of truth who works for you at at affordable used bamboo HR for our HR

management system meters a HR directory and it is updated by our HR folks and they manage all of it however using this there were some kinks if our IT systems depended on it we had to make sure we told her that there were now more dependencies on the system that they use for finance and other HR audit purposes so if the change somebody's department of the change water department is called if the change somebody's named stuff like that we had to know we had to account for that and we had to make sure the new there was an interdependence II we couldn't work in a vacuum we also had to have a single source of

truth for who can access our systems we had to track who had access to what and what groups that were in and stuff like that and that's where Active Directory came in how much ever it made me miserable Active Directory track all of our users or upper 80 groups that made sure we knew who had access what the other thing is how did it log into systems there is few systems that integrate with AD but there is a lot of systems that don't we had google apps we had Salesforce we had a whole bunch of things and that's where we have one login comment to do some of the single sign-on and salmon authentication for us

so singles one login provided single sign-on for Google Salesforce expensify bamboo which are a whole bunch of other things that we used and put a lot of other things we just used google for single sign-on so one log in kind of became our centralized place to manage users and systems and things like that so what does it all look like what does concierge architecture actually look like so we have bamboo nature which is their source of truth for who works for us we have active directory which is our source of truth for who has access to what we have one login which lets people access things so what concierge does is it full employee details from bamboo HR

and it gets who they work for what the department is what their first and last name is what their email should be things like that it pushes that that active directory based on what department there in it creates them to certain groups it creates it allows our ID ops people to create new users and manage them within groups if somebody's role changes if somebody needs access to something that we automatically don't random access to then it lets them manage groups and stuff like that with an ad so that you don't have to actually manage 81 login then just sinks with Eddie if there's a connector that post information from ad the roles and mappings and whatnot button one login

that tells one large and who has access to what and they automatically get access to those things on day one without ID having to go in and manually do things but one of the things we run into was we use linux servers people is a search into it based on their public keys we don't use ed for this how are we going to manage this centrally and leo's ec2 instances but ideally it should work no matter what cloud platform or local platform that we use that's where concierge own so we already have the system we have easy two instances or open stock or a rat space or whatever else you have these salt stock with a

nada para salt is our configuration management system we use salt and pepper along with Walt in some cases to manage all of our configuration management secret set of systems things like that so what we did was we joked assault module assault state to actually manage users within concierge so salt actually pulls Chaudhary trees memberships usernames public key stuff like that from concierge the vase or guest public keys is as part of our standard user onboarding to have users that need SSH access to upload public keys into concierge and so also treats all of these different information and manages them within all of the easy two instances that it is supposed to monitor its creates delete or update users based

on the server role so say we have a server that is for our data team to use if somebody is within the ad group for data it adds all of them as well as the recipes say somebody leaves somebody's gets off boarded whatever then if they're removed from the data group it automatically removes them from all of these instances we don't actually have to go in and manage this little salt separately we don't have to say oh let's make a pull request to remove their public key from salt it is all automatic and it runs every 30 minutes or something like that one of the biggest things we have to deal with as a FinTech company is that

we have to limit who has access to what we can't give everyone access to production we can't give everyone access to staging or you know software gap we have very sensitive data or customers here aboard who has access to that data so we need to have ephemeral SSH access of ephemeral access to a lot of things not just as a setch which means that we have to build something within concierge and because this access services within concierge you want to access to 82 instances okay concierge will do that for you you want access to AWS itself sure we can set that up with an concierge you want to access to individual customer instances with an

adder particle do support or help them on board or whatever it might be we will manage that within concierge we even built something called comet which is automated environments that last for a period of two days that developers can run code on do performance testing on stuff like that and all that is managed within concierge the way we did that is by using Python medical offices to sort of meet all kind of extensible and modular so there is something called the government service interface which any service has to extend and implement a whole bunch of methods so each service will have to define how to create a user deleted user list all users within each

axis object which is you know if you have ssh it would be a dot at a bar kombi data for work on stuff like that so list users within each access object list the different access objects that people can't get access to it also defines sub through will process if your request access to a server or if you request access to an instance do you need manager approval do you need secondary approval from somebody in dev ops stuff like that each of the services will have to implement that the whole goal set was that if somebody is adding a new service they don't have to think about how the backend works they don't think about how the UI works they should

be able to extend an interface and it should just be magic it should just work without them having to think about maintaining users without having to think about deleting them once their access time expires stuff like that the one of the baby manage that is with celery so celery's an async job runner which can run line either a RabbitMQ back end or read it back in so we use celery to kind of do a lot of the background tasks for us so listing access objects expiring user access whenever their time expires creating new users we don't want to user to sit and wait when their taxes gets approved so creating users back in the back end

without having the API wait for it stuff like that is managed with celery so let's see how it works I have a demo for you guys if I can get out of this things which I'm failing to do I stop oh there that worked

ok

my computer is not behaving like I expect it to

cool so this is concierge we both sign in with Google so we use doors with Google to log in the basic functionality which I talk about is managing the different employees so I anonymized everybody so that you don't know who every single employee is but this was this would be real name so i just created a user like these are all new users i created a user but before i created users i want to show you this department mapping that i spoke aboard say somebody works in dev ops or InfoSec you can say these are all the groups that they should be automatically added to based on their department so if you go into user management and you actually

create the user then they should be automatically added to these departments so some of the stuff is automatically populated so you click create so if you go back to user management and look at groups one of the groups that they should be automatically added to is hacker one so they you're automatically added so let me add one more user just to show what happens when the user gets deleted so if you go back in somebody gets off boarded whatever it is you need to delete that user that's great just click on delete it'll make sure you want to actually do that and they'll lead the employee so now if you go back and get the list of groups again and you look at

hiker one one more time the person is gone so there's only my user which I added earlier so straight onboarding of wording super easy you don't have to think about it too much you don't have to manually do a lot of this work other thing is can do is you can create API keys to access concierge through any client that you want we ship a Python point that is pre-configured does a whole bunch of things you can get your own API key API secret you can also manage your own sht so if you want to search access to servers and you want to get added things you can get your own as a search key you added

the concierge there and next time somebody asks you by assault it should automatically take this key put it on your authorized keys file it should just work like magic the other thing we added is audit logs you can see what every user did and what even the system did so somebody got added to a department store Active Directory group by a department mapping it will show you that somebody created an API key it will show you that so the other thing I spoke about was access requests so you saw our assassin census there which was on production line staging what if I want to add a new service like it's a sec service so all I

had to do was import a new thing called its asset service and I'm going to refresh the object the doorman notebook I ran the wrong command so I'm going to run it again with the right command and that will get the new objects that all of these access services are available so when you create a new access request you know once i refresh this I guess you should be able to see as a church service for a departure vers so if you click on that you can then get a list of all the servers that you can request access to I'm going to choose I have to reload Apache to make sure everything works correctly but I'm going to choose

salt alibaba.com which is our saltstack for this set of servers I'm going to give a reason because why not and we request access for one hour so when it is less access it does two things it sends me an email saying that my request has been submitted for approval and what I requested access to and it sends my manager an email in this case I am my own manager because demo so it sends me an email saying please approve it then once they go there i can either approve this request or rejected i approve this request and now i should be able to go in and as attached into salt on our forum com and it should be magic as it is cool the

other thing it does as soon as a grant access is it sends an email through the dev ops group saying hey this person got access to this if you feel it's unnecessary you should go in and revoke it so if they want to their work with extreme prejudice and if I try to attach again I can't do that anymore so that's the demo there's a few things remaining before we can make concierge open source and that is if you saw the UI i use google materialized I'm not a designer I don't really write your wise so it looks like skittles so it's not the best you I but you know it works there are very few

deaths there's definitely deaths but they don't really cover ninety percent of the things so we need more tests for the back-end and front-end and once that's ready we're going to open source concierge and the logic for that is if we don't have tests then we can't have people meaningfully contribute they can't know whether something is broken never there everything works stuff like that so one of the things we want to do before we open source is at least write tests for the basic functionality that we don't want to break important announcement aircar is hiring you want to come work on concierge we're hiring security engineers developed engineers i see oc's folks talk to me talk to

anybody in the airport hurry talk to that tall redhead guy that's wearing that for shirt for happy to talk to you if you're looking for a new job or if you're not looking will convince you that this is the right move all right that's concierge make access control great again [Applause] what

absolutely so the system does a low all like so if you are an admin or if you're a manager it allows you to see all the requests that somebody made so I can absolutely see all the different requests somebody made I'm building functionality to approve them within the UI as well yes you can see everything

mm-hmm so so yeah so one of the questions was why not so concierge pool data from the HR system why not push and have an event that reports it so the main reason we don't push is so the only thing we pushed today is maybe their email address but we've had issues where for some reason their names change magically and we don't want to get into those inconsistencies so exactly marriage and stuff like that so that's why we don't do it today but it's absolutely in the verge like it's something we want to do

right so the question was when you're off boarding one of one of the api's you're accessing is down so the vague volunteers deals with that is it looks at the celery exceptions or pushed and keep it keeps track of it and it retry said at a later time so it does look at that and the other thing you can do with concierge you speaking you can tell concierge wat services to awkward them from and it keeps track of that as well and you can then go in and say okay this they were an awkward it correctly let me try this again other questions

[Music]

sure sure absolutely the question was howdy what QA process do you have before you had this fully take over everything so honestly speaking our IT department couldn't wait for this so our QA process was simply try it out and let me know if it breaks and that's how we tested that is so they it was in place and go like when we launched it there were a few bugs I put myself I put myself on page duty to page me when something went wrong and yeah like our QA process was honestly our IT people looking at it and stuff like that

so one log in octave didn't really play a velvet ldap like with open a lap when I spoke to them that says that they only read from openldap they don't quite right to open held up and the only thing that really works correctly is 80 so that's why I had to bring an ad yeah oh

yeah we for the testing we actually had a second google account that we set up a brand new google apps domain that we test two things from before we hold it out to everybody but yeah so we had an ad apart or come to maine on google apps we set it set up at a part of them calm and we tested it on the second account nature everything work before rolling it account secondary google apps domain and the whole deal yeah forgot about that sir yeah other questions all right thank you so much thanks very much Karthik from Fitbit your Fitbit our sponsor thank you very much will be up again at 410 with Jeff man on