Security Is A Feature by Keith Hoodlet

the other speakers today so thank you for those of you that stuck around after that great keynote uh so of course i'm keith hoodlett i'm going to be talking about the concept of security is a feature um before we get started i'd like to give a little bit about sort of the perspective that i bring to the table and sort of the things that i've done that give me this perspective so my current role as a code security architect or sometimes a code scanning architect at github really that means i get to play with codeql all day and sort of find interesting bugs for companies that aren't sort of default within the product space sort of a great company to be doing it

for especially in open source but previously i worked as a director of application experience at thermo fisher scientific most people don't know who they are in fact i didn't know who they are until i got recruited there more than half of the world's covet 19 testing touches at least one of their products the human genome was initially sequenced in the 1990s on their device and the original cyrus cova 2 virus in wuhan was mapped on their device at the laboratories in muhan they also make freezers that sort of are used for transporting the mrna vaccines about but beyond that they work in the laboratory science space in forensics so interpol the department of justice and

the fbi in the united states and others use their products and devices laboratories especially universities use their products for chemical weapons testing can imagine who might not want that data to get out and of course olympic drug testing committee laboratories for checking if athletes are cheating also use their products so uh you can imagine the sort of fun things that we got to see and uh work with at thermo fisher i also previously hosted the application security weekly podcast with paula sidorian we start with zero whenever we count as a development team so we started with zero there i've done some bug bounty hunting especially against uh ford motor company netgear and others i love to write code and visual studio

code even before i worked for github and now especially python based and javascript are sort of the things you'll see on my open source projects and i've worked on a few as well other than these last two years i love to travel learn new languages i have yet to pick up irish but someday i promise and otherwise i'm reading and writing sometimes traveling and programming so let's start by thinking about what do your development teams your project management teams your product teams think about as a feature they're first going to think about what the customer is asking for i want an edit button on twitter i still haven't got one yet but again customers will ask you for

things right your company will spend time and money on the things that your customers are asking you for and often allocate people teams resources to that they will take those features and they'll make sure that they're tested that they don't break every time you have a new release and that over time they sort of get better right again i still want that edit button at twitter so again traditional features so why isn't security considered a feature by these development teams why aren't they allocating time just by a show of hands how many of you have some sort of banking or finance or investment application on your phone nearly all of you so that's the first question i ask the

development team the second question i ask them after we've done some sort of scan of some kind or some sort of manual security test is would you use it if it had the vulnerabilities that your application has and the look on their face is sort of one of three things either you've just called their baby ugly they can't believe you said that or total disbelief and absolutely not and it's it's sort of this expectation that the things you're using on your phone on your device or on your computer are in fact secure they won't ask you for it and i say in any of the countries i've visited so far and if you know of one that's different please tell

me after this talk when you go up to a water faucet that has two two handles and you twist the one that's on the left it's hot you didn't have to ask for it to be that way you didn't have to ask for that to be installed that way it is just that way and security is sort of that expectation that when you open that application when you give it credentials or your personal information that it is in fact secure hint it's usually not but again expectations of course same thing as well if you're not building security into your application it is both costly to fix and uh in some cases you have to sort of

do a total rebuild challenges that we saw at thermo fisher included software that had hard-coded credentials in laboratories that were doing chemical weapons analysis and the sort of sudden need to fix that was a breaking change for some of these laboratories and that wasn't going to work very well so of course many hours were spent a lot of money was spent to now have to go and remediate that because of regulate regulatory requirement and then finally your security will often be tested whether you know that or not or whether someone is willing to tell you is an entirely different thing and i say in some ways it's like cellular you sort of expect it to work everywhere and then

suddenly you get into a brick building and uh not so much um but again all these things are sort of pointing to that security should be thought of as a feature by your development teams and your product teams but it's just generally not so again those two questions do you have a finance or investment or banking application on your phone and would you use it if it had the vulnerabilities that your application now has and we know that they have so what happens when you don't treat security as a feature we have you know multiple examples of this um one sort of great example is capital one uh back in 2019 they had a breach of

over 110 million users credentials and other sort of information they were only fined 80 million us dollars which i thought was rather low but one of the sort of bigger problems that they had as a result of this is right after this breach occurred many of their security organization left because they knew about the vulnerabilities that were used to exploit these uh holes in their their code they shared them with development teams they brought them to the executive level the cso was screaming up and down and nobody did anything so the human cost as we all know it's hard to find retain train security talent and gone very quickly sort of overnight because they just didn't want to put up with it

anymore of course it also leads to bad publicity uber is another one of those companies where they had a bounty ass data breach where the cso or cso chief security officer or sometimes referred to as chiefs chief scapegoat said we're going to pay a hundred thousand dollars to this bounty hunter in florida to not release the information from this breach and ultimately that that individual was fired they were brought up on wire fraud charges and it cost in just fines alone 148 million us dollars to the company that's by the way if you take half of that for a fortune 100 company that's a pretty good budget for a security year just one year right

you probably don't even need say 10 of that to have a pretty good security budget in most organizations today and then there's sort of the interesting example of zoom people still continue to use it today and it's sort of wildly popular because it's free most of the time for those that use it but right at the beginning of the pandemic january of 2020 it turned out that they did not in fact have end-to-end encryption like they were claiming according to the fcc in the united states and a series of other vulnerabilities in their installed software was sort of piled on top of that now can you imagine the world is just about to go fully

remote companies are about to make multi-million dollar decisions for over multiple years for the solutions that they're going to use to now allow for people to be fully remote and you didn't treat security as a future and now the world knows it people are going to change their mind about whether or not they're going to use you or they're going to get a pretty steep discount in the offing for it so again things happen when you don't treat security as a feature it will eventually catch up to you and in some cases it has and it really doesn't have to be this way there's whole there's sort of several layers at which you can approach this

and my original background is in psychology so i like to go back to the 1990s in new york city in the united states where there was sort of this concept of broken windows theory which is you'd have rampant crime throughout the city but you saw that if a neighborhood had broken windows that were boarded up with ply board or plywood you'd start to see graffiti and as you start to see graffiti you start to see drug sales and you start to see drug sales receive violence and it all started with a broken window and in a lot of ways as a software engineering organization if you fix the broken windows you sort of stop all of the after

effects that take place and they they proved this out with the subway cars that have graffiti all over them in new york city they would take them out of uh sort of commission repaint them send them back out crime went down sort of quick fix right same thing can happen in your applications if you sort of catch things early sort of go about fixing those things early you can prevent those longer term security problems that ultimately crop up and then leads to some sort of event in your organization i also like to say security is unit testing my wife taught me how to cook and the first and most important thing let's check your ingredients right no

matter what recipe you're making if your ingredients are bad it will taste bad and possibly make you sick so when you're going about sort of starting at security think of it as your unit testing right everybody always thinks about unit testing and software development to make sure things are working well guess what security can in fact be a unit test as well look at static analysis and i have recommendations of some free tools available to folks as well on that front here in a couple of slides but just look at the recipe look at your code and determine what's bad about it also look at your supply chain or dependencies right what sort of ingredients are you pulling in from open

source like the was a node ipc open source project that recently ruined a lot of people's computers um i have sidebar commentary for that that i will share later over drinks but needless to say check your check your ingredients check your recipe start there because it's the earliest place that you can start to invest in security other than of course training and then begin to treat it like integration testing as well a lot of the times especially in static analysis if you're working across micro services you're going to have an input that makes it into another part of your application that isn't sort of enclosed in that micro service that you initially checked and it's not treated as bad in the next

microservice when it comes in so as i say throw a spaghetti at the wall to see if it sticks it's a great way to know that your spaghetti is done or in this case use fuzzing use dynamic analysis throw things at your application to check your mitigations your controls your actual sort of sanity checks inside of your code to make sure that in fact yeah this thing is secure for example when i went to a hospital recently and i had to sign in and put my birth date in there i hit the backspace key too many times then accidentally hit enter had a buffer underflow suddenly i was in a windows xp prompt on this in this hospital network and i

was just like nobody tested that nobody even thought that okay maybe if we give it less than what it's asking for it would be a problem again throw spaghetti at the wall fuzzing see what happens there are a lot of benefits to doing this first of all attack complexity increases now a lot of organizations continue to get breached because of phishing attempts and they're going after sort of traditional security vulnerabilities but if you look at payouts in the zero-day market for a lot of software especially in the mobile space both android and ios it's getting more expensive because those companies are investing in security and so therefore exploiting those applications or exploiting software on a phone is

getting harder it's the likes of the nso group that are able to continue to do that but again very small threat landscape by comparison which means that the people that you're worried about attacking you are a very specific set of actors and they're very they're usually very persistent about trying to get you one way or another it also differentiates in products uh the key example i always go back to here is apple while apple has its ups and downs and patrick wordle of objective c could point to a number of the different security vulnerabilities associated with it it is considered one of the the more secure mobile platforms out there by default which is a good thing especially

related to privacy we also see this in frameworks as well when you look at react js versus the original angularjs angularjs had this sort of interesting concept of the sandbox well the folks over at portswigger did a really good job of finding sandbox escapes i had the great pleasure of building a sandbox escape to full reverse shell using an angularjs framework with template injection during a training that i've given and again it was because they treated the sandbox as the sort of security thing released it outside i think version 1.8 or 1.9 and eventually they said we're going to scrap the entire sandbox well that brought it all the way back to just version 1.1 and you

still had the same vulnerability which became a breaking change when they released angular 2 whereas with react they sort of called those things right on the face of it you want to dangerously set inner html okay you can but it's dangerous and it tells you right there that it's dangerous and as a security professional kind of very easy to grip your code and say yeah this is dangerous on it are you sure you want to be doing that and why are you doing that and here's how you exploit that but it really sort of differentiated at the framework level and the applications that then were made off of those frameworks the sort of security that

could be implemented in them by default and ultimately that saves you time and money because usually when a an event occurs you're suddenly dropping everything all those features that you were working on the customers were asking you for now get delayed by weeks or months and uh you're suddenly now needing to reach out to a mandiant or somebody else and spending hundreds of thousands or millions on remediation efforts let alone actually now figuring out how this uh could be fixed in the long term so it will eventually cost you something but small investments made up front tend to have a bigger payoff over time so where to get started right as if you're in an organization i always say

the things that you actually need to care about are not necessarily security related but will have profound and compounding sort of impacts on the way that you implement security first your speed of deployment if it's two weeks it needs to go through a change review board it needs to be manually tested and then deployed you have a problem because as soon as a security incident occurs getting that thing out to production is going to be slow meaning that the pain is going to continue until you do that so making sure that your applications can get out into the world quickly and hopefully safely super important next is your breadth of testing the more things you test the better the more

tools you're implementing to make sure that you're sort of getting as much coverage as you possibly can the better but of course there is such a thing as too much so sort of fine balance there then once you've sort of implemented your speed of deployment making sure you're including security is part of that usually it's sort of low overhead if you get it early on and you get that breath of testing to sort of test it at the static analysis initial layers of putting code into production or putting code into your repository then you've got your breadth of testing for dynamic or fuzzing and on and on your mean time to resolution is sort of

your next best measurement to say how well are we actually doing it any of this stuff and i don't mean the won't fix you know accepting risk sort of things but the how quickly does your team actually recognize that something needs to be fixed starts working on that fix and deploys that fix out to production most of the time it's really slow and part of that is because your speed of deployment is slow but part of that is also because it goes to the backlog or as i say dev null and it never gets seen again and so now when you eventually have that breach or incident that occurs well guess what it was in the backlog capital

one and others and so that's sort of a good way to determine how healthy is your application or software security program and one of the most important things and this is controversial for most companies but i'll say it which is allocate time to technical debt and security is just another form of technical debt at the end of the day and your development teams usually know what is most broken and should be fixed inside of your application you need to sort of lean in on the security a little bit but i say at a minimum 10 of their time four hours a week should be spent on security really where you want to be is 20 of their time or want at least one

day a week sometimes it's a two-week sprint and sort of you know five sprint cycle or what have you but again the time investment up front made consciously makes a profound difference in the actual implementation of security in your applications without it it's constantly running from one emergency to the next so free tools to start with i will say especially for open source projects i work at github full disclosure but github does have freely available tools as part of the advanced security tool set for open source projects today so you can use them there are other great free tools such as semgrep a really neat tool and a sonar cube again all for sort of static analysis

looking at your code at the very beginning dynamic analysis the open web application security project or owasp zed attack proxy or zap really great sort of starter tool if you're looking to get into some of the web hacking especially a little bit more manual but they have some great automated stuff as well great tool to get started with the let's see here the web attack and audit framework w3af uh by andreas riancho that owl looking symbol also really great free open source tool to sort of play around with dynamic analysis especially good against java applications then looking at your ingredients that you're pulling in your dependency checking so dependabot or dependency check by owasp again great tools to sort

of quickly grab test them and then finally secrets management is one of those sort of banes of most security people's existence because as soon as you start to find secrets in your code the next thing you need to do is tell your developers how to solve those problems usually they'll just say well we'll put it in an operating system environment variable that is still hard coded somewhere and still gets pulled in and it's like no just use the secrets vaults use hashicorp vaults use cyber art conjure pull in something and teach them how to sort of create those api calls to pull those keys in dynamically so that you can sort of reset them as you need

to great tool to make that possible so if you're looking for further reading you'll notice none of these are security books which is actually sort of on purpose the very first book and the one that i kind of constantly go back to is the devops handbook read the whole thing cover to cover because part six is about security with really great security people implementing and how to sort of put that in your organization the funny thing is is i'm on the record of saying that devsecops doesn't exist it's just devops it was in the devops handbook but of course devsecops sells and it's sort of the um the newer mantra so eventually because it became my title at thermo fisher i

sort of had to embrace it but great book if you're sort of more curious about knowing not only if you are interested in getting on the manager's path yourself or sort of how the managers think about things at different levels of the organization from sort of a lead engineer all the way up to a cto the manager's path does a really good job sort of walking you through that so you understand the mentality sort of things that they care about you know as was mentioned in the keynote as well the sort of risks involved in risk-based decision-making that you have to come up with and then finally lean enterprise the first half of that book is very much

like the devops handbook in a lot of ways the last half of that book is really where the true gold lies and it's sort of gives you the idea that businesses don't exist to be secure i know shocking right they don't exist to be secure they exist to sell you a product and security is sort of a byproduct of that and it gives you a lot of insights into sort of the hows and whys behind business decisions how to sort of rehash or think about how you want to sell security back to the business and it's a really good way to think about those things of course special thanks i like to thank just a few people in my life that have

really helped me in my career namely most importantly my wife who couldn't join me on this trip but um she shares my time with the rest of the world and so i always like to send a nice thank you to her hopefully next time i'll be able to bring her and have a glass of champagne or guinness or some some other such thing then my sensei and good friend jason haddocks who when i was at bug crowd taught me a whole lot about bug bounty hunting and one of the very best human beings that i know and then my good friend who recruited me to thermo fisher and was sort of uh pushing me to do more which is brian

inagaki um who has again really pushed me in sort of my career and encouraging me to do sort of new and crazy things so with that said it looks like we have just over um five minutes or so happy to you know have any q a now or even after but um would love to get your questions happy to provide any sort of perspective that i can offer uh with the asterisk that these are my opinions uh not the opinions of anyone else or any other organization that i'm affiliated with so any questions

hi thank you for the excellent talk uh if there's one you presented a bunch of uh security tools over there that uh i'm some of them are familiar with i will others i will definitely be looking into but what do you feel is really missing from the free security tool uh landscape right now if you would like you would uh you would get one wish uh to be completed what what what uh tool would you require someone to build i know exactly what i'd ask for um so i'm a very uh good friend with some of the folks over at signal sciences who built just a really wonderful sort of in-application security firewall and of course there are free tools out

there today um but the implementation that they put together to just sort of catch things at the application layer that i'll tell you so many network-based firewalls just absolutely missed and it's sort of hilarious putting it in and bringing it behind those and saying you still have two million attacks you're missing guys like come on um that's if i had a free open source tool that i was looking for that would be it for sure great question yeah

thank you uh really good presentation first um you mentioned a little bit about dependencies right like in the case of node.js yes but what is your opinion about other languages i think some of them are very overlooked such as dotnet uh we don't see too much you know when a bit is happening there but doesn't really mean like they are they don't have the same issues um i guess from the node.js perspective is because units is pretty widely open and i don't know if that has to do with the way it's developed but what is your opinion about other dependencies and the issues that come along with that oh yeah i mean in terms of frameworks

and languages you know especially like python for example right very large sort of dependency landscape there net java as well right if you think of apache struts 2 was made famous by the experian breach super important right especially across the sort of broad ecosystem that being said you also sort of had to think about from a business perspective when you're building coverage against that and from a sort of the way the world is going perspective php is still something like 70 of the internet uh in terms of all the wordpress sites that are out there but very few people actually develop php on a daily basis anymore right um so so you sort of have to measure that

back and forth and say okay does it make sense to cover these things in terms of an exposure i think over time you know for better or worse javascript will become sort of the only language that we see on the web um i really hate their sort of laissez-faire use of semicolons but um again it's it's sort of i think that that's where a lot of companies are going is is thinking about what's being used by the rest of the world what's sort of um going to be up and coming what's still sort of around i mean even cobalt and fortran is still used wildly in you know banks and other places but nobody thinks

about dependency checking for cobalt or fortran because well it's on a mainframe and you know it's hard to get how to get those in the first place so that's sort of a hand-wavy way of saying yeah javascript python maybe some java um but after that it sort of becomes niche you know net especially it's sort of making a resurgence but not as much as i say some of the other languages great question have time for maybe one more yeah one or two more depends on how fast i can talk i suppose i was interested you mentioned lots of tools and um things you'd like to you you should include in an application security program and would you say these would be

you would would these be best done with a dedicated application security team or a is it do you have kind of a good approach to how you get the development team itself to take on these tasks and build it into the culture yeah it's an excellent question that was one of the sort of first challenges i had at thermo fisher um so starting with a dedicated team is is almost um essential because the the sort of idea is what we did is we had members of that team who would hold sort of open forum training classes every wednesday afternoon at one one eastern ish to get as much coverage of time zones and we'd invite

development teams who we had found vulnerabilities in their code to those and you'd start to get sort of one or two interested parties that would come in you'd teach them how to do some really interesting exploitation you sort of give them access to the tools and then they'd sort of some of them caught the bug as we say like they they'd go back and then they'd start doing all sort of these implementations and then they became sort of champions within those teams honestly how i expanded my team was than hiring those people especially the ones that were really good and really knew what they were doing i just bring them in and then suddenly they became

part of the program and we expanded from there and now they were sort of a trusted resource right security is sort of still considered the outsider but if you have someone who was a developer in the team doing those things day to day knows the applications and suddenly they're a security person the developers listen to them and that's sort of the approach that i had to take with it is think of yourself as a developer build the tools for them show them the tools and train them and sort of evangelize for lack of a better term that's that's really but you do need a dedicated team i mean for 3 500 or so developers today

uh at thermo fisher there was only maybe a team of eight so talking about scale but it was really because we had all those other champions so anyway i'm being told that i'm out of time thank you everyone i'll be around for a little bit so please feel free to walk up to me and say hello