BSidesSF 2019 - DevSecOps State of the Union (Clint Gibler)

next up we have Clint Gibbler with dev suck-ups State of the Union hey everyone thank you very much for coming and thank you to the organizers for having me I'm very excited to be here so there's been a lot of great talks over the past year or two in the security automation or dev ops space and the point of this talk is to try to gather and organize some of the great work that's been done and provide references to specific topics that if you want to learn more about you can follow up there so I'm going to attempt to reference about 40 talks in this 30-minute talk so whoo let's begin first a little bit about me so I'm a

research director at NCC group we do a penetration testing of pretty much all the things most recently I've been involved in a number of sort of staff augmentation type work where we're helping companies roll out secure automation or tuning sass to things like that so I've seen a lot of this firsthand I'm also a co-founder of a small company called practical program analysis we focus on some automated bug finding techniques so before my current job I was a PhD student at the University of California Davis where I dressed like this and before that I lived in the Midwest where I dress like that true story I looked up some like generic stock photos for farmers and then I

realized I actually owned that shirt already yeah I found that out this morning and I thought it was too good to resist so what are we going to talk about today so a couple of things first a couple of principles and themes I've seen across many different talks and then some specific examples in that space I'm going to talk about some specific talks that I think are especially neat because they involve eliminating bug classes I'm going to talk about a sort of case study of a project I was on with NCC group where we helped a company try to do that as well I'm going to look at a blue team truism that I've recently been thinking might

be wrong and then lastly we'll do a blitz of a bunch of other useful talks okay so first let's talk just very quickly about minimizing minimizing developer friction so this is something pretty much every deaf Cyclops talks mentioned in a little bit so some examples of this are rather than having a separate security tool system and making developers go to that instead integrate as much as you can into the developer workflow so perhaps creating security issues as JIRA tickets or just basically streamlining everything as much as possible if you're using tools and you're trying to push results directly to developers ideally these are almost always true positives because once it gets even above like 5%

or so you start losing developer trust and then this idea of like a paved road concept so netflix talks about this a lot but basically the idea is how can we make the default very simple very easy and actually just better to do it the secure way than to just go do with sort of a standard there own way so one example of this is a John Eastman at DocuSign they wrote a secure wrapper library for XML parsing and basically not only does it prevent against xxe it also gives the developer some nice telemetry so developers want to use it because they're oh hey I want to know who we're reaching out to over the

network I want some performance metrics so you have this nice developer win bundled in with a security win and I see this a bunch of places so let's talk about some of the other themes in a little bit more detail okay so one idea is self-service threat modeling right so you don't necessarily have time to me with every single development team one on one to like threat model each new say epic or feature release so slack a couple of years ago released this tool called go STL which basically is sort of a questionnaire developers can fill out it's like you know what are you touching authentication is this externally facing what languages are you using basically

here are some things to consider also Netflix has a questionnaire as well and Google has a vendor security questionnaire so not not quite the same but I know some companies who have sort of customized the prompts in their tool to be more of like a threat modeling for developers so here's a quick screenshot of some of the things Netflix asked their developers to think about okay so a couple of more talks in this space so this one from Kohinoor Bride games at apps at Kali so what was interesting about this is rather than having a separate process they basically just created a culture of okay ever anytime you're building a new service or a new feature what's like one simple

question that we can get developers to discuss amongst themselves to make them more security conscious and sort of plan security and from the beginning so again it may not cover everything but here's at least one thing that can I at least get some coverage so you know for this new thing we're building how can a malicious use or intentionally abuse this functionality and then how can we prevent that so quick simple because it's so easy it's more likely to be adopted and then also user talk from Autodesk mention Pi TM and some other things as well okay so next idea rather than having your sock look at every single alert that comes along which might be very time intensive how do we

focus on the things that are more interesting so the idea is when a user triggers a potentially fishy event let's send them a message from a slack bot saying hey it looks like you just tried to do sensitive action like pseudo Ingham prod or something like that hey was this really you and you sort of - if they push them as well so the idea here is can we push some of this validation to developers to save us time so this was originally talked about by slack and a blog post Dropbox thought that was a great idea so they did it as well and released a tool and Pinterest talked about this most recently at this past

app sec USA okay so continuous visibility so when I think comment challenge is knowing what are developers working on you can't be everywhere at once so Doug de peri of data dog talked about hey we're our developers communicating and can we try to infer where we might want to reach out to them so by potentially like scraping slack or Trello or JIRA for security tags or hey they're mentioning crypto or security or something like that maybe we want to reach out to them and then having this idea of a rich risk dashboard so what is sort of the current state of our different services so dominoes at app sack USA gave an example of like for

each service what are the components and do we have known abilities in those like do we have good coverage and then Netflix again had something as well so I don't know if you can make this out very well but it has some interesting attributes in terms of you know are they using the TLS library that we built are they is the service edge facing how many AWS sinister instances do they have and so forth so it gives them sort of like a nice quick high-level view okay so one of the biggest most continuous like through threads I see everywhere is this idea of a knapsack pipeline so people are writing code and pushing code all the

time how do we continuously scan it with static and dynamic tools so I saw Twitter originally talked about this and then Salesforce did and released a tool and the Netflix did and really scum blur and then PayPal talked about it and then a coin base did and released a tool as well so lots of people think this is a good idea so that those were more on the tool side this is more unlike the process and lessons learned side so I want to call it the second one both are good but I think the second one is interesting because they have a section of like here's some unexpected bumps that we didn't foresee that ended up

being important in practice so I thought that was very nice okay we're doing well so far let's keep going all right so some neat talks on eliminating bug classes so as exit app SEC professionals it's nice to find bugs and to fix them but I think what's nice is solving bug classes right this is where like your scalable wins come in because you're not just like whack-a-mole an individual bugs so here's a couple of talks that I think we're especially good in this space and there's probably more that I don't know of but here's some that I personally like okay so the first is from Travis of Netflix so the idea here is developers and Netflix can create new

services without talking to the security team and all of these services get like a base set of AWS permissions and what repo kid does is it basically gathers data about how the app is behaving and then periodically takes permissions away and then watches the app to see if it fails so if it does then it gives the permit back and if it doesn't then it assumes oh it didn't need this permission so what's really cool about this is without ever talking to the security team without developers having in-depth knowledge of how AWS permissions work which is right quite complex you're just gradually converging on this steady state of least privilege so apps that aren't use converge to zero permissions

so I think this is pretty awesome so the next one from will if Netflix is how do we detect use of compromised AWS credentials outside of our environment so there's been a number of high-profile cases where people where companies have had credentials leaked or published at github and it's resulted in you know thousands of dollars of AWS bill or even a company closing down because all their backups were deleted and source code was deleted and sort of like a game over a scenario okay so let's just quickly talk about how the the flow overall works before we get into the details so you have an ec2 instance which calls assumed role so this is getting some credentials

back from iam these credentials can then be used to interact with the AWS metadata service which then allows you to talk with other services and meanwhile AWS cloud trail which is another service is logging all of these API calls so high level this is sort of what we're working with so this technique works by making a strong assumption at the beginning which is the first use of a AWS credential we assume is legitimate so when we see a new one we assume that it was done by a service week control and then the first use was not an attacker so if you make the strong assumption then you can say if we see the use of an AWS cred from an IP

that we haven't seen used this assumed role API call that is something that we've sort of added to this table that were iteratively keeping track of then a credential has been exfiltrated from our environment and thus we should alert so we'll gave this a very non intimidating diagram about how it all works so let's quickly talk about it so basically we're continuously watching cloud trail logs of which API calls are happening and then if you see a call to assume role we basically have this table we're keeping up to date with a mapping between the source IP and the AWS credential is being used so like okay we saw this IP using this one this one

using this one and so forth so if we see a API called it's not assumed role we check if the current user that is the assumed role has an IP in our table and if it does then it's like okay cool you know this is expected behavior but if we see a source IP that's not in our table then we alert because this is likely a compromised credential so we'll actually released a number of proof-of-concept codes for this implementation is the first two links and the third link is a example metadata proxy which basically can block a service from hitting the AWS metadata service at all to sort of prevent the creds from being leaked in

the first place so what we just talked about is like how do we detect it and then this is ideally preventing it from other ever happening and he said that this was able to significantly restrict the impact of bug bounty submissions in practice like they found an SS or F and he was like cool try to hit this metadata service and then they were really frustrated so they did a follow-up talk will and Travis about some other stuff that's also worth checking out ok cool so this is from max of Airbnb at cccc so a challenge for many companies is you have this crunchy exterior soft interior right so how do you switch to a secure internal network

while not halting development and starting over right so if you have you know a thousand engineers and tons and tons of services and say hundreds of deploys a day you can't really say like hey everyone please like stop working for a couple days or a week while I change things right because that's going to cost hundreds of thousands or millions of dollars in lost productivity so the goal here is how do we stay out of the way of engineers ideally this is transparent can we make it secure by default and not just that but also make it difficult for it to be insecure and ideally this is also agnostic to you know what language our framework the

service is using what protocol is being spoken all of those things because you don't necessarily know how things are going to work in the future so he described it quite nicely in two sentences and we're gonna break this down in a couple of slides so basically you use mutual TLS in service discovery for authentication and confidentiality and you discover access lists that is what service is allowed to talk to what automatically so it's basically zero config security okay so let's look at how this works so each node here at the top represents a service in like a micro service architecture so each service has a x.509 certificate for identity the communication between each service uses

mutual TLS so unlike when you're browsing the web and the TLS you're using allows you to authenticate the server you're talking to for mutual TLS the server also knows who's talking to it so it's sort of bi-directional identity is known and then the third part is you have this authorization map which is basically who can talk to who because ideally you want to restrict nodes to only be able to talk to who they need to be able to talk to so if an attacker compromises one they can't talk to say more privileged things that that service shouldn't be able to talk to okay so originally this is how their service-oriented architecture works so they have in service a and node 1 and

then service B on another node so they have a local reverse proxy that service a talks to which then talks over HTTP to service B so what they change this to is they actually added a reverse proxy on the receiving service that handles the mutual TLS so which really need and clever about this approach is that this mutual TLS is transparent to service a and service B they don't need to know or understand anything about it and what Mac said in the talk is that they were actually able to deploy this without changing any code from any of the services that the engineers wrote so basically it was just handled transparently which i think is very very

cool so the last part of this is okay we have identity we have secure communications but how do we know who should be able to talk to who so the interesting insight here is is you actually already know this information if you use a config management tool like ansible chef or puppet so for Airbnb specifically they use chef to configure services in in those scripts they already have these attributes that say ok this service needs to be network connected to this other service so they basically wrote a service that continuously watches chef scripts and then parses out these connections and then these connections are continuously enforced at a network level as well so you can imagine if you

didn't have a clever trick like this you would have to have like five engineers spend a day building the mapping about what can talk to what and then it's immediately out of date as soon as you do it so you're gonna spend a ton of time for not much benefit but I thought this was very clever and I liked it a lot okay so next talk so this is from Hong ji who of Dropbox so the idea here is how do you scale ibly protect a massive number of heterogeneous internal apps so for most companies you have one or a handful of public facing applications say like Dropbox com that has a very well-defined tech stack the

security team has put a lot of time into hardening it making nice secure wrapper libraries making nice tooling so this is like lockdown but for internal applications that is things that are not externally facing oftentimes developers can use whatever they want so there's you know hundreds of frameworks and libraries and it's difficult to sort of scale your efforts there so initially you might say oh let's just firewall it off let's say we have like some services where you can debug prod or like Business Analytics and so let go let's just firewall them up so people can access them unfortunately that doesn't work for say you have users on your internal network like employees who could be see surfed or you know attacked

by DNS rebinding and things like that so that would allow external attackers to still reach these internal services even if there's a firewall and to put out a shameless plug some colleagues of mine built singularity which is a pretty neat DNS rebinding tool so also if you already have an attacker on the inside again firewall doesn't help or if you're interested in the internal abuse case again a firewall it's not going to help okay so the cut to the chase the idea is having a proxy so all employee browsers talk over TLS to the proxy and then also the proxy have the tos connection to the internal services so what's nice about this is it has a centralized scalable

platform that you can build defenses into so each application written in many different languages on many stacks don't have to build these protections themselves and this is also useful for other security teams like the infrastructure security team detection and response and so forth so hungee gives a couple of pretty awesome examples so one is doing sea surf without necessarily caring about what language and framework the internal apps are using he also talks about content security policy spoiler alert it's still really hard but he also talked about detection and response and some other things so I would encourage you to check this out so yeah I noticed after writing these proxy seem to be pretty hot ok so

last one in this some one thing that I think that's interesting is so coming from a pen testing background and like manually finding bugs you're using like an IDE or perhaps a proxy like Berbers app and then on the other end of the spectrum you have some commercial static and dynamic analysis tools that are trying to like black box solve all your problems but what I think is interesting is in sort of the middle and maybe the human augmented space historically there was kind of this desert of sadness but there's a couple of new players in this field that I think are interesting so one is Cemil which is a number of UK academics built a static analysis

company and their primary thing is this query language or QL that finds bug variance so given a bug that's interesting let's find other instances of that and they found some cool bugs another company is shift left which is building a tool called ocular so if you're familiar with you earn this is basically a commercial version by Fabien Yamaguchi and others so basically this gives you a ripple that you can query for different code patterns and do like data flow analysis sort of interactively in a console and the original paper I think is this bottom link but he's done a number of papers that are really awesome so I encourage you to check those out ok

doing good 10 minutes left time to do it okay so I want to describe a case study of a client project that I that I was on there's very interesting both because I think the methodology might be useful for you as well as some of the results surprised me okay so a company came to me saying hey we're paying a lot of money for this asset when we're not getting a lot of value can you help us figure out what to do with this so I'm going to talk a little bit about how we scoped the problem our methodology like what we did what we found and then how you can do this too okay so initially this project is a bit

vague you know more value like what does that even mean so at first thought you might think okay well let's try to maximize the bugs found and then divide that by the cost of the tool and boom you have your security ROI but there's a lot of hidden costs there right in reality integrating the tool and your CITV pipeline tuning it the app psych engineer time triaging that's all a non-trivial amount of cost I don't know any company that's like oh we've got plenty of app sack engineers and they had so much free time they're just twiddling their thumbs right that's never the case and ultimately I think what's important to think about is you

know if we just have someone manually pentest all the things is that going to give us better bugs than using this tool because if the answer is yes then like don't use the tool right like that does it make sense so ultimately the success criteria here is you know given a limited realistic app psych engineer time and budget what's the best use of our time to reduce overall risk and the company was already doing all the things that you should be doing like threat modeling secure libraries fast custom linting internal and external pen testing bug bounty and so forth okay so the intuition behind what we did is that prior bugs are likely indicative of

future bugs right so there's always unknown unknowns but I think in most cases this is a reasonable assumption so the core idea for what we did is let's review all prior known bugs over the past couple of years and determine is this suitable for static analysis so first we worked with the security team to group a bunch of prior bugs by category so like this is XSS this is open redirect and so forth here's the PR fixing it and here's some meta information and of course we needed access to the code and the SAS tool and JIRA and so forth so in terms of the project work itself basically what we did was review as many bugs as possible and then

tried to see you know could we have detected this with grep did we need some abstract syntax tree type matching did we need data flow analysis as well as what mitigations and checks are already in place so the company already had some custom linting that was already finding some things which was awesome and also we did some research in terms of like in the languages and frameworks you're using is there some primitives that we could use so let's say the SAS tool isn't useful how easy is it gonna be to roll your own because ideally there's some stuff to build from and you're not starting from scratch okay so what was the output it was a report basically the

first part with you know here's some overview of static analysis theory just for your reference here's what we've seen using tuning SAS and a bunch of other companies but the main body of it was okay for every vulnerability class that we were able to look at you know does it make sense to use sassed should you instead have built some custom tooling should you write a secure wrapper library or something else and of course when you make recommendations like this you have to keep in mind you know what technology are you using what is your SDLC look like and where's the product going in the future and so forth okay so what we found some things

confirmed our intuition some things were a bit surprising so the first is that secure wrapper libraries for example they built one so all access of user data had to go through this wrapper library it killed sequel injection so awesome that's why you hope to see as does modern frameworks so new code was written in react and then there was legacy JavaScript as well that hadn't been ported so anytime there was XSS it was in the old code so I think both of those are things as apps like engineers like you want to believe but it was cool to see the data back this up so for surprises for this I never thought as a security consultant I would say you know

for this type of security issue like don't care about it you know it's fine but ultimately for things that are pretty infrequent or it's always low severity or it's just going to be mitigated completely by upcoming stuff like honestly that's kind of the real answer so I really like static analysis I think it's awesome and so I really wanted the answer to be all let's build like this crazy complex rad static analysis tool but what I found is that right a secure repre library and custom linting in at least for this company in their tech stack was much better than fast in general because most of the vulnerabilities were not something SAS tools are generally designed to find

like access control bugs or they required extensive custom rule writing which for this tool was not easy and very time intensive so I found this interesting because dataflow analysis which is sort of a selling point of most SAS tools at least in this case would not have helped too much okay so how to do this at your company so two things so first track your vulnerabilities so if you can group these by code base as well as type like access control access etc is very useful ideally you also have access to the poor request that fixed it so you know what does this bug look like and then how does the fix look like and

then also how do you find them right pentest bug bounty and so forth one trend that you might find interesting is is one of these finding most of the high severity bugs and how much is this costing you and keep in mind obviously time as well and then once you have a bunch of these review the vulnerabilities and then think like what can find these or prevent them entirely at scale and obviously you want to minimize cost in time another interesting thing is you know has your org or team had other nice wins in the past that you can leverage different cultures mean different solutions are more or less effective so it's always important to keep that in mind okay so

one thing I've been thinking recently that I wanted to mention to you all and I would love to have your feedback is historically as a community we've been very focused on finding all of the bugs and there's been a huge number of companies that we collectively have spent billions and billions of dollars with to find all the bugs but I think most people would agree that there are still bugs right and one thing my friend Zane said recently is that historically security has been based on the flawed assumption that we can find all the bugs right so but bugs I think are always going to exist as long as you have a non-trivial is non-trivial sized

codebase and tools generally can't find every bug there's always gonna be false to negative so in access control or business logic flaws or just fundamentally the flow is too complex or you don't have enough context and you can try to turn the tool knob up to being as over approximate as possible but then you're gonna get drowned in false positives so one thing I've been thinking about recently that I've been starting to chat with people about specifically Scott and Zane have influenced my thoughts in this space is I wonder if the next trend in security is secure defaults and then lightweight adoption checks to make sure you're using them right so would you find interprocedural data flow analysis of

backend code tracing that through multiple micro services into like a very complex front-end code would you rather do that or would you rather grep for dangerously set inner HTML which in react turns off the output encoding so I think this idea of like detecting use of secure defaults or that lack of use is just really an order of magnitude easier problem than finding bugs so this is something I've been thinking about I'm still thinking about it so if you have thoughts I would love to hear them and then in the real world obviously you went logging I'm monitoring as well for visibility okay cool let's do it finish strong okay so if you want some like

interesting like mindset and principles about desktops I would encourage Danes talk if you want sort of a collection of like practical tips and tricks from a bunch of companies my EPSA key you talk was just like I convinced many of those so if you are curious about the pros and cons like a nice overview of SAST and DAST overstock is quite nice just in the author of brakeman wrote this talk on like building custom aft matters in a couple different languages which i thought was quite nice if you're curious about what cases give desks tools trouble Roberts talk is quite good there's a number of talks about starting apps like programs one I want to call

out in particular is leaps because he also referenced a lot of research that he found useful so when I saw him referencing all those other things I was super excited because then you can follow up with those as well so I've been moderating a couple of security automation panels so this one from apps Akali was recorded unfortunately the other two or not but I think this is a nice opportunity to hear what smart people at a bunch of places have done and how they think about things a couple more talks so a lot of people are thinking about scanning like code using static and dynamic analysis but this first talk by Nick mentioned it was like hey we have these config

management tools can we just scan those for like potentially insecure configs so I think that's pretty cool Oxton and Patrick's talk on how do we empower developers as well and invest in long term wins I think it's great and then finally Arkady's data-driven bug bounty from V sides last year basically using bug bounty data to inform where do you invest app set time like what types of issues do you have and where we did it everybody so so thank you very much for your time so I've taken notes on most of these talks and I'm planning to write up some detailed summaries I'm probably going to regret this but I give you permission to flame me on the Internet's

if I don't publish at least one in the next two weeks actually a bunch of people taking pictures of this slightly scary but yeah so yeah feel free to reach out to me on either of those I'm happy to talk about this and I think it's almost exactly 30 minutes so I'm getting nods from the side so thank you very much for your time I'm gonna hang out after this and I would love to talk with you yeah thanks and have a great day Oh on behalf of the esight San Francisco I'd like to thank you for your presentation today