Turnkey Code – Enhancing Secrets Management in Large Scale Organizations

hey everyone hope you all having a fun day and well it's the last talk let's go for be after so here it goes so the talk of TurnKey code is basically when we go for the concept of TurnKey and we bring it a little bit to the code and the concept of turn key is just something that is ready for immediately use it's mostly used on hous so when you have a turn key house it's something that you just buy and can go just right and go into living there at the moment that you buy the house and you well that concept also appears a little bit on the code itself and first of all let

me present myself my name is di Lush I'm a security engineer I've been working with security for over five years in different companies I started on check marks and joined blip Das flutter and now I am at OLX uh so introduction to TurnKey code TurnKey code is basically the same thing as turn key but when you bring it to the code so it's code that is ready to go it's immediately accessible and well the passwords are already in the code you don't need to do anything you just start code the passwords directly into the code well it's faster for development because if the passwords are already in the code it's faster for um doing everything you don't need to go to

configure the password you don't need to configure anything everything is just already the code you don't need to go look to password to connect to a certain app or password to connect to another app or usernames or anything you just put everything into the code and well it will work as fine I expect it's easy to share around when you want to share a password for application that you have you don't need to use some like private bin or slack Secrets or anything like that you just put the password directly into the code and then you do a git commit and the password would be on the repository and that seems okay isn't it it's also simplifi debugging because

everything will be in the code so it's pretty much a little bit more easy to do the debug itself and in the end it has one reduced dependency setup because you don't need to configure like Vault or any Secret store or Secrets manager because eventually you'll put everything into the code and the things will work fine and well it's faster isn't it it's better and that's a good approach but well that's not exactly so when you put a lot of turn key code and believe me your companies I'm almost sure that they have some turn ke code in there it brings a lot of issues itself and uh it brings security Bridges because Dart code passwords will be into the code and

if one person has access to the code then the passwords will be there it also gives a little bit of exposure because if you have a public repo with some some code in there the password will also be in there so if someone gets access to your GitHub or gitl they will have eventually access to everything it's pretty much more difficult if you have to ke to rotate Secrets because the secrets are there into the code you don't need to to change them in certain platform and that's quite more tricky I would say than if you had like a password manager that you just rotate password in one place and it will rotate for everything if you have in the code

maybe you need to replace it into two three four and more and it's also brings a lot of compliance violations and we can all agree on that it increase the attx surface because everything is in the code if someone wants to have a look at your company they just go to the code and well most probably they will have access with passwords or secrets and anything like that and you know it brings a lot of lack of access control because well everyone can be an admin if they just go there and look a little bit and well we have a little bit of turn turn key code in our organizations we need to do something about it so okay

what can we do to fix the turn key code we actually go for secret scanning and secret scanning is basically finding secrets and passwords and tokens in into the code and it actually needs to be specific for Secrets because secrets are passwords because it will find only that thing and that's one of the misconception because nowadays we are seeing the traditional s Solutions actually bring a lot of more effort into find passwords and all of that but the secret scanning itself is meant to be as a complement to those Solutions because when you go when we think about secret scanning we think about finding some kind of Secrets and things like that and it needs to have some kind of entropy to

check if a secret is valid or not because otherwise you will just have hundreds and hundreds of results and most of them will be false positives and you also need to go mostly important to have a comprehensive analysis because you need to go to the G story because the isue in secret scanning most of the cases are not the new commits where you already have your Vault or something like um uh a Secrets manager configured is that old old instance of GitHub that migrated from time to time that has a lot rapples that should been archi maybe should been removed for a long time and the passwords are still there and most of the cases they are not even on the

recent commit they are just there on Old commits and well you need to do something to find them and it also bring mitigates a lot of risks and streng strengthens the compliance so what we need to make secret scanning work and in then having no more turn keep code we actually need a tool that will be responsible for doing the secret scanning we actually need also rles the place where we will run all the scanners and we also have to have some kind of tool to import the results otherwise we'll just have 100 of results and we can't do anything on that we just need to put them in a platform where developers can go there and check if

everything is good do some tries on results and well things are improving that sense so what we need to choose a tool for doing the secret scanning we need the tool to be prop built so it will find things like password IP Keys Secrets tokens into the code it needs to have some kind of detection method and that's a big plus because as much as you can reduce the false positives that's a win because developers will go to the tool they will try the results and if you have less results they will be more happy I would say try those and it also brings the table pattern matching with in the end will bring a lot of L false F

and also validator so immediately check if a password a g up Tok or something like that it's valid or not because if you have the information that it's valid you just don't need to do the tries you just need to go there and rotate the SEC and that's it and it also brings a comprehensive search that I was talking about that will go into the G hisory and find the secrets that are there in the end because you can in the current code you may not have secrets but in Old commits there might be a lot of secrets in there it needs to have some kind of customizability because the secrets on my organization May different from your

organization every organization is different and you might need to set up a new set of rules find another secrets and it also needs to support multiple code bases because in a large scale organization you kind need to have support for GitHub gitlab because there are things always moving around and you have code on both places so if you have a tool that can run the scan in there well that's a big plus so one tool that actually does the trick is Stog it checks all the boxes and it's pretty much easy to install you just do a BRI install or you do anything and it actually works very good you just do like a TR git and you pass the git rep

or you can just also run it locally and something like that and well you have there immediately the validator to check if that token is valid or not and that's a big plus and the tool itself is pretty much easy to work with and then you actually need to have repository coverage and one misconception is also that when you do Secret scanning you put a secret scanning the first time on the cicd pipelines and that might work but I I guess the end goal of it is to have at least the first scan into scanning the whole infrastructure because otherwise those old rapples and those old commits I was talking about might not even be in

the new cic the process they just might be there lost in archived or something like that and if you just go for the currently cicd process you just lose all those rles that areti if your company be GitHub or gitlab is exposed those will come to the mind and most most of them might have passwords valid in there so you need to check everything you need to scan everything one tool that we actually use that works pretty much good with it which isn't in end built for having other rles in there it's built it's Source graph it's actually built for you can search things into the code into GitHub gitlab at the same time or

even more but it also keeps a shallow clone in there and then if you run the scans into Source craft you can actually have scans for your your all thing so well that's a big plus and the tool itself it's open source we actually use it for free and well it's a it's a nice one and well you need to import the results somewhere to um somehow you need to do the tri because even if you have hundreds of results you need to do something with the results and there's tool that I actually work with and it's built by current and past um employees of blip and floter and it's also open source it actually doesn't have support

for um secret right now however I already created a pool request it's up for review and you also can check out my private um GitHub and there's a fork of that with the secret scanning and basically the tool is just a tool thatto is written in Python Jango and correlates and syns data from several data source and things like that you can just import everything you can create a model and it will have an application itself that I will show in a bit and that actually is pretty much good because it has an built in that and it's quite a funny module in there so what we need to the process we have turn key

code so let's apply what we learned from before we'll do the Clone clone all reples that we have we'll do the scan and then we'll import the results into security surface and well that should be it and we'll have no more turn key code once the developers go there and reveal all the secrets but well it was not quite that like that because that thing that I talked about of the entropy that reduce false POS positives also was creating a huge blind spot on results that I guess most of the results shouldn't be in the code and most of them the first time that we looked and okay this is just test files or comments

or things like that but they were actually most of them valid passwords and valid secrets so we actually need to do custom rules to find the secrets because it's like I said a secret on our organization May differ from your organization and the last thing on the the red one is like truog also doesn't scan for sensitive files and we actually noticed that developers put sensitive files P2 JPS on the GitHub or gitlab and all of that and even though they are not exactly Secrets they are sensitive files like the name said and well if you can do some work and some rules to catch them well great so we refin the process we now do a clone and we can use Source

SC like I said to have all the rles there we'll do the scan with the truog rules and then we do new scans one to catch all the those results and truog has the customizability that I talked about to catch the new rules and find the other passwords and secrets that were missing but also with a scan to find like do a git loog to find those sensitive files what I was talking about and then you kind of need to have another thing which is the improvements because it's like you import and you some for some reason drog has that thing called entropy and pattern analysis so you don't have much false positives but when you scan all these results like API

keys and passwords equals to Portugal in that case you will find a lot of results believe me so in the end you need to go in there remove false POS REM what we actually did was if it's from a test file or from node models or something like that cases that we know that are not valid passwords if it's comment or something like that remove those results otherwise there will be hundred and hundreds of results for the developers to try they won't do it believe me so you need somehow to remove the secrets and also you need to remove some kind of duplicates because a secret might be a secret into several locations but in the

end it's the same secret and the same thing applies for the sensitive files that was talking about so you can have the same sensitive file the same p12 uh file into several locations on the same code and what we actually did on that is okay we'll create an ash of f file and then compare it if it's or not the same because if it's the same we not create new secret we'll just create a secret with Ash of that file and then compare if there's any more places with that file and well in the last part we just import the results into some solution where we can do the tri itself and well this is the current

solution it's actually running on my machine but you can set a surface on yours and do the things in there I actually have most of information inform in the G rep so take a look in there but well the tool looks like this you have the secret it shows the what type of kind of secret you have it shows the The Source where it came from and it shows a reple the sources are direct link to the for example that secret is on trock if you click on one of those five sources it will link you directly to the git reple and also to the line where the secret is directly into the code and in

the commit that you want you can have the styles to and the creality to do the tri yourself or the developers in then you have there to check if the secret is valid or not and if it's verified and the location itself is one thing that will show you right now because like the like I said before you can have the same secret and that's the secret and you have it in free files so you don't create free more entries you just create free more locations so that secret is on that free files and well that they actually are on the same commit but well that's there and you can also say like I said the sensitive files in the case

this is one of do mobile provision but it's the same thing because it's on that repo and you have two locations that that secret is so the first part of secret is an ash of the file itself well the work is not finished because here you expect the developers to go here and try everything and in a mature State you probably won't have many results and many passwords here to to trash but the work is not finished you just don't do the this process and everything is okay you need to keep on doing new things and things to mature your secrets management site and there are three things that you can do and I think they are the most important ones

when after you do the first part of it which have set up a promit set up a pre receive and set up cic scans doing the the secrets at the same time and you actually the best of course the best thing is to have all the free implemented but each of them has its own limitations and if you can just put two of them running well that's good because in the end the promit it's good because it catches the the the secrets even before they go to the G reple but at the same time they need to have some developer by in the pr re is actually one of the most most Smart Ones I would

say that requires less effort but at the same time they are ideal ideal for a selfed environment so might not work with most of the up that for all organizations and the last one cic scans okay but a secret in this case will already be into the G rep so when you get to the cicd you'll have just to rotate the secret on doing something like that the to be before prevent the secret itself for being in the G reple but if you have on the cic scans that's Co but well developers will also complain that you are putting more time into the cic pipelines that you are doing more results maybe you are blocking things so that's thing you need

to improve on the threee the best is to have the free if you cannot have the best of free choose what's best for you I have actually practical examples of this this is a rle that you can also check on my kitb and it's like an example for Tru log on the pr commit but uh and this is how it looks you do a commit of a file with will if it's all shck a box in this case just look for the TR log one but if it's p it will be committed to the grle but if it's not it will be failed and the commit won't even go through so you there will be a secret

and you don't you need to remove the secret in order to do the Commit This also can be set up on the CSD using the the workflows but yeah it's all the same so key takeaways and this is like a small presentation but I think it's important to to point out this Fe five things which is you need to scan everything you don't just not do Secret scanning into the current c i pipelines you need to scan everything that it's your our code because will be those reppel that are Arch that you don't look for several time several years that has Secrets or passwords in them and believe me there will be secrets in there you

need to do the tri I would say of course it's hard for push the developers to do and to try do results but results will come every day every day developers are doing code the results will up go up in there and you need to do some kind of Trish and well you can enforce at all Sage that is the longterm goal and that's like the Holy Grail but well if you cannot do atal SES maybe do at some best than nothing you need to be proactive because one time or another your GitHub or gitlab will be exposed or kind of some rapples or things like that if you are not prepared before you will

have a big issue in that like a little time frame to rate everything so you need to do this before your get of is EXP post and you need to stay consistent so repeat the same process try the results import new results implement the the wooks or the cic if needed and well everything is like that so this is a quick one if anybody has a question feel free to ask but well that's is all folks [Applause]