Opinionless Enforcement of Opinions on Operational Secrets

Jonathan thank you thank you it is true I am Jonathan I'm from Autodesk and I'm here to talk about specifically opinion 'less and port forcement of opinions on operational secrets sorry yes talking about a pinion enforcement of opinions on operational secrets later later not Thursday yet I don't know how many people here know about Autodesk or what it is we do there's some words up there if you want to read those while I say slightly different words we enable people to make things the more interesting things I have come across have been 3d printed prosthesis prosthetic hands there's USB powered soccer balls or rather USB batteries that are powered by kicking around the soccer ball they're

contained within I met somebody last year at one of our internal conferences who is working with a group out in Boston to assemble genomes in an attempt to fight cancer so there's lots of really interesting things going on there also civil infrastructure cars video games a lot of stuff coming out of Hollywood there's it's very unlikely that you've come across things that that if you have not come across things that Autodesk has had some hand in so we're Auto disk is right now for a bit of background I am in the cloud Operations Group I am doing whatever DevOps is supposed to be at Autodesk and I'm helping us essentially get all of our stuff kind of

consistent because we are after all the decades-old software company that has only recently started coming out of the habit of essentially shipping gold master desktop software where you have that gold master image and then it's done it's the users problem at that point any bugs while it's probably windows have you tried reinstalling Windows tough habit to break to have it to break so now we're transitioning to the cloud via the data center so we had a bit of stuff there played around with private clouds and now going out to whatever the cloud is an interesting thing about us is we've got a lot of very specialized software most of it much of it some of it still trapped

within applications some examples of some of the software I've come across that I don't have a personal use for but I find fascinating ignoring the design software that's used to create 3d prosthetic hands and little USB battery soccer balled mashups we have things that are very specialized such as crowd and traffic simulation for freeway systems conduit routing for how you actually run an optimum amount of electrical conduit and data conduit in a skyscraper how to make that skyscraper get the what is it the platinum LEED certification to be environmentally friendly a lot of super focused stuff there that being said we've been around we just celebrated our 35th birthday there was a lot of interesting swag that people were giving

around that they've been keeping for the past 35 years and it's not uncommon at the shop to see people who have been there for 15 plus years so where we are right now a lot of applications a lot of ways of doing things not only have we been developing our own IP in-house we've also been acquiring a bunch of companies all of these projects all of these acquisitions they come with their own opinions generally manifest it as a technology choices but not always so we have now unsustainable orchestration tooling there's a lot of it huge variety of it trying to get things in a single path can be a little frustrating we are however working super hard on getting to

whatever that paved road looks like for us and we're evolving from a DevOps team to a unified dev ops teams anyone want to take a guess at what the distinction is that's that's okay a good idea for indicating the DevOps what any DevOps team is so we can see the photo there of silos each one of those silos represents a product team everything flows through the DevOps engineers who are delilah out of that little box in the front of it not necessarily unified it's still essentially throwing things over the wall and we are moving away from that and we're fairly certain that the light of the tunnel is not actually a train we think it's kind of a door

that's being hung open not sure yet so in terms of some of the challenges we face we're coming from super traditional IT Service Management we have very traditional infrastructure you name it in terms of not to give too many deep there's a lot out there there's a bunch of stuff out there and we're very aggressively again moving to the cloud and in this case the cloud for us is Amazon Web Services starting to talk a little bit about the feelings of it one of the more interesting challenges here has not had anything to do with the technology it has not had to do with the opinions that people are manifesting as technology choices but it's just that culture shift trying to

shift towards something where it's uh there's more of an emphasis on crossing collaboration on projects and tools rather than the team that's making that tool where it becomes something that's shared across the organization so there's probably some interesting or familiar products on this you know how are we driving this move towards more collaboration well Enterprise github and slack are the the cliche examples here so we have a group chat system you know throwback to IRC but it renders shifts in line and you can have all these conversations in quasi real-time there's Enterprise github which if nothing else what it brings us is something to facilitate pull request culture something to essentially put in the gates for peer review where other people

can look at this project and look at whatever your changes are and not even necessarily be on your team or in your time zone but you can tag them and they can look at it and it turns into more of a community endeavor I was actually super happy to see we have an open-source policy and process there's a JIRA workflow mapping this and it's actually been super smooth to get stuff open sourced I've also I hadn't heard of the term inner source prior to coming to hot or desk anyone here heard the term inner source before look at all those hands that aren't going at awesome we'll talk a bit more about that in a bit

the bottom line of what we're trying to do and this is a huge part of what I'm trying to push is transparency and collaboration there's pockets of collaboration within the organization but in terms of how we manage something is key is operational secrets that's something that everyone should be involved in if not necessarily in making the tools although that would be cool too in making use of the tools and developing the best practices and the documentation together at the end of the day we have to ensure that everyone feels empowered to participate and I feel like it's relevant to mention that during my first draft of this it said and sure every engineer feels empowered

to participate no it's actually everyone some of the more some of the challenges have been around like project managers and scrum masters and making sure that they're taking into account the fact that we need to pay more attention to our individual team's goals to actually get to where we need to go some other forms of how we're getting there how are we making use of some of those technologies and and ideology so to speak well we're trying to provide some tools which follow some basic UNIX tenets the only two I care about well the two I care about the most is portability / efficiency we have people writing Windows on their desktop and Mac on their desktop and I'm not sure if

they're supposed to but I've seen a few people running Linux on their desktop so we need the stuff to be able to run everywhere and we want these individual tools to do one thing super well like we're not necessarily sure the tool is what we want in the longer the long game but it's a stepping stone in terms of creating that paved road that we all want to be on other than pushing the transparency and collaboration and everyone holding hands writing tools together also putting a huge emphasis on data models specifically for operational constructs there's lots of aspects of this out there now usually in the form of opinionated Jason opinionated yam well you can probably see where I am

going with this so I've got mixed feelings on Yambol and to kind of touch upon those I'm gonna read this quote from the Man of La Mancha she is when life itself seems lunatic who knows where madness lies perhaps to be too practical as madness to surrender dreams this may be madness too much sanity may be madness and maddest of all to see life as it is and not as it should be so clearly I've got a lot of feelings about yam all the at the end of the day my the anak data I've collected and I've sourced from people indicates that developers like they they're grumpy about anything with semantic whitespace but they're a little okay with ya mole

compared to having to write out configuration files in Jason or configuration files in Y ammo and as much as we strive to have generators for these configuration files though sometimes you still have to type some Jason or some now and llamó seems to be a little friendlier I'm not sure why less punctuation different punctuation it's hard to tell in terms of the Y Amal itself seem to mostly just be following the trend that parts of the industry is set kind of following behind ansible ya know to reflect what that data structure is going to be who here knows about hashey core vault let's start talking about operational secrets I see more hands going up them for inter source so

that's not surprising so for the rest of you hashey core vault gives you a set of web services around managing operational secrets it has some really neat things in there in terms of dynamic secret provisions so you need access to an AWS account you ask vault for AWS credentials on a particular role and it gives you either an iam or STS role credentials that are tied to that particular account tied to those roles and oh by the way they're going to get deleted after the TTLs up unless you manage to remember to renew them there's also other things like you can throw all of your SSL certificates or SSH user passwords whatever really into a generic

secret store which is basically just nice encryption stuff around what's in essence a JSON blob it has the idea of key rotation baked in at the forefront where everything runs around the idea of tokens every token has a TTL and when the tokens up the tokens up it's gone everything that you've requested with that token is now gone my favorite gimmick that vault does is everything's encrypted with a master key that's cool nobody ever sees the master key I haven't read the source to figure out how you can get in it I'm sure you can but in terms of operational nobody looks at it how do you access the master key well it's generated when you first

initialize your vault and then you get back a certain amount of unsealed keys and then you have to provide a certain amount of those generally less than the total amount back using something called Shamir's key sharing algorithm and this essentially allows you to have you know more than one person have access to unsealing this entire vault system I mean it might mean you have to wake someone else up when you're on call but the benefit in terms of ops X probably going to be worth it so we started making use of alt internally doing kind of an early the adoption thing and it got kind of interesting talk a little bit of that in the bit but to make it a little less

interesting a little more I'm gonna say sustainable we came up with a little tool that was before it was open source was imaginatively called vault rapper and I was told we can't call it vol rapper so he came up with a new name aomi which is a Anglo sation of the Chinese phrase profound mystery which seemed to fit so it's an open source tool out there on github full-on CI CD and although I can always add more tests ships as Python packages docker packages it's designed from our perspective to fit into continuous delivery pipelines we don't want anyone to ever see actually see the secret so we're going to talk a bit more about that in a

moment the oh look more ya know there's some ml up there so we're essentially looking for this basic Yamma model to describe what these secrets are going to be and we'll talk about exactly why we're keeping it separate from vault we're not allowing everyone to do their own stuff directly to vault you can probably guess what happened and oh look I have it right there it turned into madness we let everyone just write to vault however they wanted to and nobody could find stuff like you go to write something yourself when you haven't written down where it is and it's it turned into kind of a mess let alone if you're trying to share credentials and those sorts of

things or at least the access to credentials between teams it got pretty messy pretty quickly so a huge part of this and essentially everything we're trying to do as a community project at Autodesk focuses on transparency everything lives in github now that the secrets of course don't like we're not gonna keep our Amazon root keys and they were not gonna keep whatever things people decide to put as generic secrets in the vault in github but everything else does all of the metadata for these secrets there's no reason for an ECL necessarily to not be visible to the people inside the company there's no reason for example that a developer kits like oh I can't launch my thing into

staging because I don't have these AWS permissions well because we're working on pushing a pull request model they know where that repository is they can submit the pull request and back oh I actually need access to whatever of the Amazon new web services they launched last week was and we have a conversation in the pull request nice little change record there and then the change gets made merged in and deployed out to the appropriate servers so huge emphasis on pull request driven culture not just for the code itself but for all of the associated metadata around that and in order to do that like this data model presents everything right in there either in a set of shared repositories

or a project specific repository so that the developers themselves can like look them at these stuff and manipulate it as they as they need the for better for worse than most commonly formal use of secrets in our particular deployment is generic secrets which is the catch-all for everything that we don't want to have to implement in a dynamic secret I we could also call it legacy we can talk a bit more about that the basic generic secret representation you get a vault path an endpoint and you can throw a JSON blob in there basic key value mapping it doesn't really care what's going in there the opinions that vault itself is expressing are very minimal

and in terms of how a omie is interacting with the generic secret storage our back-end we tried to make it very easy you can throw a plain text file in there a plain text file you can use more yeah mol you can just populate it with random data like we shouldn't actually care what these secrets are gonna be rotating them constantly anyway ideally so in terms of the opinions aao meat doesn't care you can put whatever you want in for your vault paths and we're not really enforcing this at this point the emphasis is on the data model so we can find this stuff vault has its own constraints sometimes you can't I think the one that we've cost doubled on

across a few times as you can't mount another mount point on top of another one because they both disappear there's some weird voodoo there and Naomi doesn't actually enforce any of this stuff yet whatever you put in there it's going to do as long as you know you have the actual permission to the vault sir

now let's start talking a bit more about the data model itself the it's driven by a file that we've imaginatively called secret file it doesn't actually have to be called that but it fits in the Dockers file Jiang can file make file rake file I had another one I don't remember what it is but it's essentially just a llamo file and you have different yanil blob representations of different types of Secrets you have generic secrets you have AWS credentials local users do oh I could go on pull requests welcome the secret file itself is also a template it's yeah Mel Jinja to anyone who's used an Sable's familiar with that I learned a bunch of fascinating stuff

about Jinja 2 yesterday that's gonna cause me to re-evaluate that though and again to reiterate doesn't actually be called secret file you can call it whatever you want the intention here was for it to be almost danger well actually dangerously flexible it's built to just express those opinions on how you store operational secrets you can do things like device development deployment environment based templates so paths can be informed based on what your environment is whatever variables people are going past you can have the paths themselves or the secrets themselves constructed by data you're pulling out of who cares what you're feeding it yeah moe it doesn't actually matter the only limit is yourself an aspect of this that

we're trying to push so it's great to have the individual application teams in the products have a little secret file in there so anyone coming in you feel like oh this is how I get this credential this is how we this get this credential they can never actually see the credentials themselves because the environments themselves are locked down but they know how to access it but one thing we found to be kind of interesting is where you have managed accounts by one group or another and then oh we have a point of reference to we know every one of these accounts of a particular service because oh there is their in github oh one needs to be added let's

add the pull request for it oh there's something's not documented right or that needs different role but you can make the pull request so I talked I mentioned the idea of deployment based templating deployment environment based templating and pulling data from external sources we're trying to move into a point where nobody ever actually sees a secret except for security because I don't really have a choice sometimes and in order to do that we don't want to rather as part of doing that we don't necessarily want to slow down development efforts so everything around the secret phone this idea for provisioning on the operational secrets is happening as part of our pipeline we have tooling out there so that product

engineers manage the secrets that are contextual to their app and we have tooling that promotes that to different environments they'd have no access to that but the data model is in essence what gets promoted so we have some element of guarantee that those sets of Secrets potentially different paths but accessible nonetheless will be accessible throughout any environment they need to care about so I'm gonna talk just a little bit about this it's kind of just more boring examples but in terms of a generic secret you can represent it from a text file and then when you tell a oh me to go do its thing it'll write that out twelve paths involve so the little example down here

you can see the contents of the secret that are referenced by the yamo file and there's some hand-wavy there where we type a ohmy seed and her vault servers hopefully up and whatever I decided not to do a live demo because reasons and in the end the things there and you can read it out there's test coverage on this functionality though you can check that they're green last time I checked you can also write secrets from the amyl so it's basically another key values blob it just goes straight into Vall pretty boring stuff actually I'm gonna step back here a moment so the things we have seen people put in here are well anything that you would otherwise base64

and throw into an ansible vault or throw into an encrypted chef data bag very similar with the amyl files except that tends to be things like API keys user key pairs that kind of stuff this one is kind of fun and was put in just to enable the promotion between environments we don't want to ever look at secrets and we don't want our developers to be aware of them outside of a tangential sense of metadata and this just ensures that no when you run this it's gonna generate a random secret who cares it's nobody has to actually look at this thing as long as all the tooling that needs to exchange the secret can access it who cares

the the biggest win for us has been the Amazon Web service credentials this is dynamic AWS creds and this is a bad example because you know we shouldn't give anyone administrator access directly but whatever essentially what this will do is it'll provision a endpoint and vault that when I touched upon this a bit earlier that when you ask for credentials you'll get credentials associated with a given role and those are credentials that are good up until the default TTL of I have no idea what vault sets it to ours are a number that's based in reality I like to think and then after that's up the credentials disappear you can renew the token if it's and thus the AWS

credentials is that something that vault has been set up for but other than that it's on YouTube renew those credentials and there's lots of tooling baked in there to renew those credentials and the good news is is that when you use vault to mediate renewing these credentials everything downstream from that in Amazon or other dynamic secret sources dynamic secret endpoints I guess is handled so the different few thing other ways you can use again we force we just talked about getting data into that into vault there's aspects of it that I was super gung-ho about initially and now I'm kind of like yeah and it's essentially on things to support legacy applications and I'm sure people have

differing interpretations of what legacy applications are it's either a things that were there before you got there be things that don't have tests or see something that habitually wakes you up when you're on call I'm not going to define it past those vague things but the different things that we have added to it do support that seeding we talked a little bit about it though it's you're taking that secret file you're throwing it into vault it's a template can feed it with external data some things are gonna need root and it fits into a continuous delivery pipeline that part is super key for us we've given them probably too much flexibility my bad and with an emphasis based on placed on the

low friction for adoption we want people to be using this we want people to be using the data model in terms of seating it like you can have tags you can choose at runtime what you're gonna see it in maybe you're only going to just make sure all of your backends are there because you're the one with root and you don't want to deal with all the secrets so just make sure the backends are there make sure the developers can access it and you're good cold storage is a relic new feature in this and it's based around the idea that every security department has that at least that one person with a person size safe somewhere

and in there is where they actually write down like open SSL Certificates on post-it notes and whatever or hopefully keep things on GPG encrypted USB drives and that's what we're aiming for here it's the idea of the complete cold disaster recovery oh your backups are gone that sucks well you have your tooling somewhere so you can spin up vault again and we have all of our source code because hopefully we backed it up in a billion places and then oh we have the cold storage secrets we can just reprovision everything there and we're good it goes two ways I called it an ice file the little artifact because it sounded cool I don't I don't really

know it's just a GPG encrypted zip file or tar ball I don't even remember anymore does have test coverage though so this is where we get into the Linux sea bits or the legacy bits it's basically different ways you can pull secrets out okay so you can upload a file into vault using it you can extract it out to and it'll either spit it out to a file or standard out because legacy systems love plain text files and standard out you can do the same thing with environment variables I have a few rec rats about this and it basically renders these things out as environment variables whatever the secret is and there's a lot of flexibility in terms of

how you massage the full name of the vault path into whatever the actual environment variable is people are going to abuse this guilty you could do the same thing with AWS credentials guilty of abuse this but in essence it's doing the same thing as previous just contextualize down to AWS this is a really easy way of taking a legacy application that is using almost any app anything that's based on an Amazon SDK they all use the same environment variables this is a one line thing you can drop into your script and boom you have dynamic secrets you can render templates out because again a lot of legacy applications read all of their configuration including secrets from

templates so it's a Jinja to template and you can control how the keys and paths are translated there's a bunch of built in templates as well that I added because I was tired of trying to figure out how to deal with different credentials and some of our C i stuff's like oh cool built-in templates don't have to do anything I can just render out my credentials right there and much like with just the basic seating of vault you can pass an arbitrary data to this as well so you can do things I mean I guess a good example would be like an artifact or thing you pull the user pass user name and password from vault itself

using Yomi butBut URL doesn't need to be a secret that you just specified or pulled out of an environment variable or wherever it is so how did this go when we first tried it it was kind of a mess there were secrets everywhere but but they weren't in post-it notes they weren't an email I'm sure there were still seven box but there shouldn't have been they were no longer and yet this is the important thing we got secrets at a source control the biggest issue was there is confusion around where to actually put things again we're not enforcing any opinions at this point we're just like please use our data model it'll make things easier in the

future so people just put it what anywhere who cares because it's written down we know where to find it so who likes greenfield programming there's gonna be more hands and with inner source none really okay okay thank you and especially greenfield programming super flexible you're gonna do whatever you want that's what happened in our case here so in terms of where we're going from this hoping to heavily leveraged emergent behavior we're learning from each other on this we're gaining up gaining early adopters both inside and hopefully soon outside the company and trying to build up the importance in here is building up the momentum outside of the teams who stated task is to work on this tooling we're

trying to build a community around this so we're continuing to build at all of the initial concepts here based on how developers are making use of the system and again I've said this a few times but it's like I don't necessarily care too much that it's messy we can change the messy we have a data model we can gradually introduce stronger opinions one of the upcoming features on this is essentially like style guides on your vault paths it'll check your secret file I'll check the vault server and let you know if you're doing something you shouldn't be and in order to enable that's like there's basic semantics for migrating from one path to another and it's pretty straightforward you know

delete one path the other one gets added it's it's pretty straightforward stuff it's just boring channel the effect here is it'll end up moving the secret from one place to another you have a change record and that everything's tracked and get it's all nice and lovely now in terms of what comes next for us who the I think what's the joke there's two hard problems in computer science naming things in cash and validation and we're stuck on that first one right now there's a lot of confusion on what to actually name things once we've decided then we're gonna start enforcing things we'll be like oh why you can't write that secret there because it doesn't fit

the convention until we have that there's not much of a point to it the biggest emphasis in terms of going forward however is to build that community we have a few other tools on this around the idea of managing operational secrets this is the one that's the most I don't want to say polish to the most in use at this point and we're essentially trying to cater curate those inner and open-source tools projects around that and huge emphasis on pull request driven operations there and the emphasis on what the tools are hopefully helping us with is managing operational secrets in terms of Iommi itself its feature set is complete but not fleshed out like we're going to be

adding more dynamic secret providers however we're not really gonna be placing too much emphasis on the ability to pull secrets out there's a bunch of stuff out there for doing that already console templates kind of cool because it just runs as a daemon and handles updating Keys magically it's kind of cool we will be adding more secret types as I mentioned I'm gonna support different authentication flavors the priorities are pretty much being driven by community adoption at this point AWS is a big thing for us and then we'll see what other dynamic secrets people want before I wrap up I just want to talk a little bit about the a this idea of vault native applications where

everything I've talked about for extracting secrets from vault okay you write it to a text file environment variable template these are all things that aren't really getting us any better than the status quo other than that we don't keep things and get vault is a RESTful API driven application you can have your application talk directly to it it never used needs to write secret's out only keep them encrypted in-memory decrypt them when you actually talked to whatever API you're talking about key rotation you know nobody cares anymore because the system is just going to do it and the hope around this going back to what I was saying before like less emphasis on legacy secret extraction more emphasis

on seeding and more emphasis on these community endeavors so that we're not all duplicating our effort there and then in terms of the biggest thing and you know the past two days have been a great example of why we need to do this it needs better data validation that's pretty much it like we can't trust anything that people are putting into yamo files can't trust whatever is coming back from the vault server even so all of that has to be more rigorously validated and tested and this is up on github so if there's any questions I'm fairly easy to find on the internet and word around here I guess and yeah that's it thank you very much pull request

welcome [Applause] thank you very much Jonathan on behalf of Fitbit and besides SF we'd like to present you a Fitbit Alta reminding you that motivation is your best accessory so I've noticed we've been giving outfit bits and fit bits to sleep tracking there is a fascinating talk at develop stays a few years ago where they start attached pedro duty to their fitbit's I suggest you try and check that out I know right [Music] quick announcement if anyone is wanting an awesome bear board from bear sprite to populate your own electronic badge later when we open source the design I'll have a couple down here and in about a minute so just find me [Music]