Preaching to More than Just the Choir: Working with Developers to Improve Security

you you everybody can hear me hi welcome to preachings more than just the fire I of course nothing works Wow next slide there we go I'm mr. glass area sober mints I'm a developer a on cyber solutions they have no idea what I'm talking about here today they were very kind about sending me here but no this represents them it's all me I'm involved in all sorts of community things many of you know me from there not hi mr. glass been a developer for about a decade now worth about 10 organizations because startups are terrible so hope you like to talk to me about battle scene and the craziness is involved and why you end up in 10

jobs in 10 years I can but the good thing is I've had the opportunity to work in a a lot of organizations and every one of them works differently being worked with a lot of different technologies and wear a lot of hats as you change roles so I really got a whole breadth of what's going on focused mostly on back-end web application development I'm security so while I do front end jobs occasionally most of what I do is api's authentication things of that nature laws cloud engineering and applications like that I often get suckered into CIC DS ops work so for OC ICD our integration continuous delivery terminology and DevOps for automated processes for building and publishing

code I'll get into that a lot more later I get often get into that for two reasons first of all starts or small you only have a few people on your team somebody needs to do work getting back-end I tend to be more into the Linux server stuff than the front-end developers so it often falls on me and the other thing is DevOps just the general move in the world to integrating these specialties means that I've had to learn more of any do more of it so I often end up being the guy on the dev team who spends a lot of time not and I've always been obsessed with security of going depth on college

things like that so I've always ended up being the guy who did the security work at the company and more recently I've actually ended up to more of a security resource level sometimes so I've also done compliance work in small organizations and things like that that's me what we're going to talk about today so this talk comes from a bunch of rants on Twitter basically since I've been in this community for years I've seen all sorts of people who get angry when a developer won't listen to them or why can't the developers just do this right or why can't we do this and half the time you're totally right that developers being annoying he should be

fixing his damn code but half the time it's like you don't understand there's you know hours of processes involved and you know it takes two days to publish something and there's all sorts of stuff going on and you know if you spent time learning how developers work and how you can work with them you might actually get them to listen to you so this is what inspired me to make this talk is just all these rants people who don't know how developers work haven't I haven't worked with them at all and don't spend the time learning and so that's why I'm here to talk about we're gonna tell you the process we go through for developing a product the different

I'm gonna touch on agile little but not a whole lot and how you can integrate into agile as well as how you can use what we have the tools that we use for other work for security purposes and use your leverage as a security person to get security features built in by the dead developers our focus is shipping features that's what makes the company money when you work as a developer most of us tend to want to go and dig deep into a subject or spend hours figuring out this one little bug right but you spend 10 hours fixing it one-line bug that you could have done by duplicating that line of code somewhere else or

something you've just wasted a ton of money and time right so we have to balance good with actually getting something out the door and this is kind of pervasive in what I do now it's a lot of what agile is about and so we have a mentality about pushing code at the door and so we don't necessarily go through like the process you would do if you're writing code at home or like we don't dig in and learn a subject fully we just can't okay it was a bit of a rant if you saw my bio on the website you might have read that I hate DevOps AB seconds anybody want to raise their hand and

tell me what DevOps is you

okay that's a different one and I have heard it's good that you bring up agile though because DevOps reminds me of agile and that it means something else to everybody some everybody has if you matter what company you go to they will do agile differently than the last one you are at agile has a bunch of principles most the places that do it end up sticky notes pencils but DevOps has been a little bit different DevOps started right when I was getting out of college and the idea was hey you know the operations team is siloed the development team is siloed the security team is siloed right let's go ahead and make these people work together let's

collaborate let's automate and use coders do some of the ops work let's work together and do this right right and I originally develops had a huge push for automation before reacting to high load and for optimization and things like that and also for optimum ating processes that they couldn't go wrong so build processes and things like that or heavily automated now and that's the devops world that's all from DevOps and that's great but what happened was they stopped talking about this very clearly the businessí people know stood horrible cliff you know horrible stock our business guy they stop saying that and it started being more like this oh hey all about you all of our developers

are responsible for all the security and all the ops work I literally have interviewed of companies where like everybody's expected to be a sponsor all the office everybody spent all the code I'm vice-consul for this and this sounds great in theory and I'm not trying to be here and say I have no responsibility when the server goes down I am also on doing things to fix it but we don't know opps the Gophers don't understand how Linux works right I have my favorite example is what's an inode I've heard of it once when a server crashed and the other developer who had heard of it before fixed it right and we don't know how these things work an ops

guy is trained in this stuff and we have no idea and there simply is no way we're gonna get that knowledge so when you make DevOps in your organization like we're gonna be DevOps now you all are in charge of DevOps they won't know what to do and it's all you can send them to a DevOps training camp they'll learn the automation stuff but they're still not gonna know how Linux works I literally the only developers I have met who could fix a like the Linux server is broken and we need to bring it back alive we're ones who are obsessed with open-source freaks who did that as a hobby you need a trained professional and the real

reason for that is this is your typical learning curve it's the horrible graph and I just made look like darby untrained worker right and then you slowly learn and maybe it gets harder or easier as you go up and then you hit a plateau when you're an expert you don't learn much out there because you've learned most of it that's fine when you're developer you're always on this learning curve every day is another puzzle I have to solve usually lots of googling and so you end up doing learning but you have to stop again you have to should feature so typically I get to about there which is literally about two minutes after I've gotten a

level of knowledge or I'm comfortable pushing my codes that's it so if I need to figure out this Linux problem I learn enough to say hey I have solved the Linux problem and I don't learn how the file system works because I don't need to that's the efficient thing for the business and that's my job what we need to do instead break down walls work together you know have teamwork they we cannot have people who are good at all these specialties at once it's simply not possible right how hard is it to get skilled InfoSec people right you're gonna get a skilled InfoSec person who also knows hops and who also knows development it's not happening

so most of the teams I have worked on that have been good and what I look for now in a company is these cross-discipline product teams you have anywhere from four to ten people they have everything on their son you'll have your developers in the room you may have your ops people in the room you may have QA engineers in the room have design you may have you definitely have a project manager these are everybody involved in the process and the thing is especially on the technical side you want to have everybody's knowledge to like this is our effective our ops guy everybody has the wool but you all share these responsibilities everybody does like 10% ops and 10% this and 10% back

and when they have a problem you don't do what we call my company now is throwing the ball over the wall right I have this thing I'm trying to fix it I need ops help I cannot figure out the sauce problem I call the ops guy and we don't just say he Rob's guy here's a broken server fix it no you sit with the ops guy you pair up you figure it out together and then the next time that sort of breaks you both can do it this is the only way to get them to learn I like this is the only way to share real speciality work and it prevents mistakes because you have the good ops guy paying

attention to what they're doing this is the best day and you end up people who have some of that skill but again we'll never have that real ops person knowledge all right I'm sorry for the long tangent about DevOps but it kind of is a background for what we're gonna go into but let's talk about now agile and sprint planning I'm planning backlog grooming these are parts of agile if you want to learn more about gel their entire talks about agile specifically for the security audience circuits Wan did what a couple years ago besides look it up you learn all the different parts of the process and the principles and whatever the prom with agile is every

company does it differently oh my company doesn't have backlog grooming so as much as and some company is all this planning I'm gonna talk about what happened back on grooming a bit my company happen sprint planning cuz that's what they do but you should talk to project manager and get involved in their agile process these planning meetings are all team members who have any any stakeholder in the project right so QA sales people sometimes will show up the project managers the product managers and the core of agile is this collaboration between these two teams the the before agile came about you had what's called waterfall process somebody would sit in the room decide what features would be like write an email

send it to the dev team and the dev team would stare at the email go okay I'll make our webpage that does that but you run into issues right away you need to collaborate need to talk to them so agile it's about everybody gets in a room hour to a week it's and talks about what are the features we're building this week and you go through all the problems and the developer might go oh you know it's hard for me to build a three dimensional table like that but you know if we get rid of this part of the feature do you care about that okay it'll take me a tenth amount of time right and that you can only do that if

you have this cross meaning which is why you should get involved with it to ask for the security stuff right you're the stakeholder for security now so get involved has to be added to the meeting invite check the agendas you do not need to show up every time you do not need to say anything half the time so my project my product manager is in the meeting every week and I've heard her say five words in past two months but if we have a question we turn to her um you do not need to go to every meeting and especially if you start going to these meetings you start establishing yourself you can get the point where they'll come

to you with problems so your goal is to ask for security features in the requirements so hey you're making a new log in page can you make sure the login page is mfa can you make sure that the login page actually doesn't let you interview enter in the wrong password right these are requirements it's very important to get them in early and you should also raise security concerns if they're starting a new project that's gonna gather a lot of data if you're the person in the room who's responsible raise your hand going hey maybe we shouldn't gather that data right you know hey how are you going to secure this data in the cloud how are you going

to encrypt it right you need to raise those concerns early also want to answer questions this is a my role a lot in the room there's gonna be a new feature somebody's asked to make a login page they're like how do you make a secure login page and then your job is to go and tell them you know use this use that you helped and establish self has this resource they could come to with these questions because developers are spending all their time trying to learn about the problems that they can code it so if you're the resource they'll come to you and ask you for help I have been googling it hopefully it's important to get your requirements

early I can't stress that enough not just because you don't want to be the guy who comes in as we're about to put code out in the door I mean like oh actually you can't release that you need to add encryption I that will make you enemies fast it always happens and everything there's always a change requirements at the last minute but if every time there's a release you're coming in at the end and being like that you're gonna make enemies which is why agile was invented it was to prevent this like long process of that the helpers are working on the thing for three months and then somebody goes and sees what they ask for and it's not what

they asked for it so get involved in time get your requirements in early it also helps because the team needs to estimate time and if the requirement is in there they're not gonna put timing for it right we and often the companies will have a QA person whose entire job is going down the list of requirements and making sure they're actually fulfilled well if you want during the process of this feature going out the door somebody sits and goes through and make sure that security feature is in there you put in the requirements and often don't actually be out as unit tests so it's not even a manual person so every time the code gets tested with

the unit test program it'll run through a test of whatever you want security feature you once you get into the requirements and you get all these things that we use for general software development right but you can do them for security too and there's no reason not to but you have to get them into the requirements and also ask for security features right mfa is my favorite example of this MFE is a feature it is a stronger authentication mechanism companies sell it companies saw it as an add-on there you have to pay more money for how is this not a feature right so going at it through the future backlog and that gives you a different status this is no

longer a bug this is something we're working on this is a product we're shipping whoa but it's this is it's not a functional requirement rights isn't requirement so requirements go on features requirements are when you're making long a page it has to do this this is more we're gonna add a new application thing to our application it's gonna do something new so ask for security features right people are all sorts of companies are selling encryption and an MFA as add-ons to their existing web applications so you know getting this feature to get you a whole new status and developers now their job is to work on your feature and you're not even a bug slowing them down

my engineers time to refactor code so we don't get to decide what we're working on most of the time we'll sit there in the meeting and we'll go man I really want to fix you know the server's the server's keep breaking on us but you know where you have this quarterly project due and I need to get that out the door literally a conversation I had with my boss yesterday right so I have to stop working on the servers and I have to go get the project out the door if you come into security and you're like no this is vital this needs more time this needs to go it done you're changing these priorities and now that

quarterly project may get backward while I go and fix the security thing so going in and being like hey that thing that's causing you pain points that piece of you know that piece of code that's causing you issues that's also causing security issues right the authentication keeps breaking this thing gets killing servers this is a problem that keeps hitting us know them and say hey this piece of code keeps you in security issues so maybe go and refactor it bolster their arguments for when they want to refactor it because if helpers always have a list of things they want to refactor similarly we have a huge list of stuff we want to spend time in

terms of infrastructure and tooling so scripts for automation and deployment scripts for checking stuff etc a lot of companies again you need balance developing these in turtle doing with actually getting a product out the door these are not even part of the product right so the season even like oh I need to fix this bug or I need to deal with this issue in the product this is completely the customers will never see this all it does is make our jobs more efficient right and does love that because we're lazy so if I were job gets more efficient it's easier for us but we aren't given time for that long abrogation automated testing infrastructure is code so the idea that

you hit a button and it makes your service for you based on the script right monitoring all these are tools that developers often are like yeah we should really turn that on but we don't have time and again if you come in as a security person it's like we really need Monterrey for our security apparatus too now you have two people in the room who want this feature and it gains priority leveraging DevOps tools so most companies every company I've worked at most companies out there have an automated process for doing their films and getting code actually up on to wherever it's going right this is actually I work with working websites a lot of terminology I'm using websites

but the same stuff happens with phone apps the same stuff happens with Windows computer applications everything these days go through your CI and CV pipelines they they check out the code they test it they build it to see if it builds right they then go through a functional test etc they do all sorts of things and the thing is these are all pretty much bash scripting or something similar you can hit a button to do it you can trigger an event so a very common one is hey not all these tests every time done somebody does a pull request or somebody merges two master right order you doing my schedule I of stuff that runs nightly

right or every single time we hit the build button to push code out just stick this in there right and also they're all built for alerting as well so something goes wrong or you want alerts based on what you're checking you can get that right so it's typical to do unit tests for just general features colanders to check style and things that the code stays nice it's easier to maintain I don't make sure it works simple check scripts like that and they'll build the application actually run it of course even add security checks writing functional tests I don't understand why I don't see these more it takes you about two minutes to write a test there probably is already a

test actually that goes and logs in is somebody right they'll take you two minutes to doubt to take the code for that test and modify it to put in the wrong password and check that it says password incorrect right how many websites every year so you hear about a website that just stopped checking passwords and let everybody in they added something like this it won't happen right you can test the access controls in the same way you can test your input validation I have caught scripts that like taking input have impaired validation code that was running and always returning true right oh it's valid it's just always valid so I've caught that by having these

tests that actually put something back up and see that says nope invalid so you can write those tests you can ask them to be written you can also just run premade tools so so literally add in a bash line that runs that checks all your packages and your package manager every package manager has it there's static vulnerability Alice analysis stuff as well so again you run the script and it checks for known things that often go wrong or known problems like sequel injection right it's very easy to look at code and say hey this might have sequel injection so you can run the stack analysis and it alert you and you can program it to

email you pop up and slack every day now in my company I have we run all of our unit tests and Konya security ones our full functional test so it actually goes through a full usage of our application it checks all the libraries for security vulnerabilities and I'm looking into different statical in the billing analysis tools but they're very they're noisy at first so I haven't turned any on yet but every morning at 9 a.m. I get a little alert in slack weather this all passed or something isn't right right and I've caught in the past two weeks I've caught three vulnerable NPM dependencies because every day I get an alert whether it is if you wanted more of this and this is

my favorite thing to do right now there was a huge conference a few days ago called all-day DevOps it was all online and all the videos are online now and they had several talks on just this they had a dev SEC ops track pacifically on security and they had several talks on just the tools and how to run okay at a certain point you're going to need to walk over to the engineering team be like hey you guys have this around this book right you don't need to go in and go hey guys we noticed that you're not actually checking people's passwords when they log in hopefully that goes great before that thing goes oh my gosh that's

terrible I mean if it's not checking password so probably gonna be oh my gosh that's terrible but if you're like in olders this bit of code and you can't actually run this unless you're to an admin but if you do that as an admin you can then do this thing and that triggers a vulnerability developers might go uh-huh and that's a problem we have many stakeholders this goes back to the buying US time argument so if you come to us with that problem we might say listen I'm not saying that's not a problem but I have bigger fish to fry right I have a long list of features in my backlog like literally you want this

MFA but there are people waiting a year for multi tenants right having two counts for the same actual account to sign into the same account bomb tendency engineers they may not understand the problem when you come to them you're like hey you have this weird problem in your Olaf scheme and the engineer will be does not know Allah I'm the only engineer I know that that actually understands their life works there are others out there but I'm the only Olaf guy ever company I'd bet right so they're gonna need to look into it and they should be willing to learn and you should be able to tell them at least getting started right so if you come to

them and they like have no idea about OAuth recognize that their job is learning every day all right hatching something is far easier than in theory than in practice so you go to them and you're like oh man can you fix this they're like oh it's gonna take me to a show like what it's just gonna take to date why do these developers can't get this fix out in 24 hours and it's like yeah but you don't know that this framework needs this thing and this and I've had things that literally should have been one-line change take three weeks it's just nature of the beast and these are these are all really the things that I can rants about on Twitter

and like you need to understand that you're working with people who have jobs that isn't just security we often don't get the chance to fix I have been chastised at several companies or working on a fix or dealing with an incident rather than writing code that several times in the state law hi it has happened because they wanted me to be working on code rather than fixing some security bug that they weren't concerned about most important and I see us a lot sometimes pen testers walk in like jocks like oh man your scoot is terrible I found this problem that's how you make enemies this isn't really a security or developer thing you know be nice when

you come somebody and go hey your codes broken you made something bad understand that you're walking over somebody and saying your work is terrible and so when you do that approach it knowing that you're saying that you have to like you have to go and tell me did something bad but approach it that way and know that you're doing you're telling somebody something that they're not going to be happy about that's pretty much it establish yourself as a knowledge resource try to work your goals into the requirements and some features here's a capital buy time for engineers to do what they want please please please look into these tools if you have them your company if you have developers please

tell them to look into doing it and be tactful for godsakes you

right so that that goes for to learn anything right if you go to me like hey and I looked into it and you need to flip the switch in your library or you can do this they'll usually appreciate that you careful about talking down to people right you know want to go a developer be like oh you have a sickle infection do you want me to do you know what a sequel injection is every developer has heard of a sequel injection but they may be like oh yeah I've heard of a sequel injection but how do I prevent that right and you you need to be prepared to be like oh I'll go look into stuff for you or I'll sit with

you and I'll work on it or here's a jumping-off point that I found you know this other guy I don't know about sequel injection but here's this talk I go be prepared to help them that's huge because they don't know security I've had this thing recently when I interview people I asked them three basic security questions and I do when I asked them how do you store a password how do you prevent against I have somewhat XSS is and what xsrf is and I have never gotten the correct answer hopefully never now these are web developers right these are professionals in this field and they don't know how to store a password or about half will have a store password

almost none can I decked Lee differentiate between xss and xsrf now it's partly because those are terrible terms and partly because you don't have to deal with it because it's all baked into the framework now so usually what happens me no do you know what xsrf is now in the cross-site request forgery I have no idea you know it's something with talking between sites again terrible name cross-site request forgery worst name I've ever seen some talk about different sites whatever and I go yeah it's not that you know that token that's in all your forms automatically for you from me like yeah yeah that's the CSR thorny you're right yeah that's prevent this thing and they've seen it

but they don't know the security portion of they don't know why it's there it's just part of their lives any other questions

honestly I really don't think it would be very valuable I think it's important for developers to be exposed by winch and the training sessions and send them to things like this right give them a day so that they were exposed to it so they can see the concept can see how hackers learn I don't think sitting them in a five-week course is gonna help much

oh there definitely is and I'm not talking about security awareness training security practices are interesting they're not bad but the kind of standards a lot of the stuff you see in devops world and DevOps now DevOps has transformed a little bit again and now it's just modern ops right this is what every option CD working together things like that and so instead you know secure coding practices are part of that and they're a lot better than where twenty years ago so most developers are not durable about it and again you can you can get mostly to secure development practices in a day all right a two hour training course just like security awareness you get to 80% by having that

stupid two hours thing that we all hate so I don't think you want to give them like a five-week intent five-day intensive course in the middle of nowhere I don't think there's a huge utility to it unless they want to do security unless they're actually building security features or doing security engineering or you don't have the security to engineer on your team if you don't have a security engineer on your team maybe send one guy over to become a security engineer part-time can I

you you yeah you so startups in particular and most companies these days are obsessive being flat structures oh you know how all of them don't even be have a manager but we pretend he's not a manager right the the main stakeholders in that meeting who actually called the shots will be your product manager product manager is in charge of what features are there and what the product is and a project manager is making sure times are met things get out the door etc so your product manager will know about the thing they're selling the project manager will know nothing about it but will know how to get the developers to actually put code out the door they are

the two usually calling the shots if it's an actual technical question usually there's some senior technical person an engineering manager who can call those shots but usually if it's a one feature a on one feature B that's the product manager saying this feature is more important because I manage the product or the project manager saying you don't have time to spend on that product that feature you need to work on this other one

yeah you so um I touched on this a little but I might not gone into it really a lot of the cross-functional teams some of the people will be on multiple P my project manager works for three teams if you have QA and design they're not spending as much time on an individual thing as a developers so they'll work with several teams that's one thing right one person can cover multiple teams if the load on them from that team isn't bad but they can't be in meetings all day every day to attend these meetings different companies the different things the bigger companies have moved toward making secure libraries and other tools like that so Google all my Google

engineer friends have no idea about secure I say something to them they're like oh yeah what's the password hashing algorithm these days because there's a list of algorithms are approved in they use that right you're encrypting something to use Google pink which is open-source highly recommended option library easy to use it's made by the security team the security engineering team I believe be almost idiot-proof so if you use Google tink there aren't all the weird settings you can get wrong with in cryptography they're not thinner you're they're probably under the hood somewhere you can probably change there you want to but the way you're supposed to use Google pink you don't get to change the settings that everybody gets

wrong so they're working on libraries that developers can use without security knowledge the biggest mistake I see for security is people are they're not using a framework so I do web dev and there are web dev frameworks in every language people either don't use the framework or don't use the tools of the framework they're in so your framework will have almost always a form processing library which will do validation for you but if you don't use the form processing library to process the form you don't get the validation

you

yeah you

another thing I have done developers tend to be extremely curious and loved learning about this stuff so if you see an interesting talk that's slightly dev centric or an interesting article or some goal post in their chat book ended phenomen right I've sent the POC or gtfo to numerous developer teams and they'll go in a meeting which gets them in that mindset right gets it in their head they're thinking about security it's not gonna be anything practical what they're coding right now but hey look at all these weird traits you can do maybe think about what that means for your code any other questions I think we're over time Thanks

if anybody would like a card contact me later come see me

you I mean that's true of libraries and NPM um but that's the known vulnerability analysis right so you put that tool in your CI and you just get alerts there are some companies I've hooked it up and automatically update it which is recipe for disaster presentation