Why Can't We Be Friends? Building Relationships with Developers to Improve Application Security

my name is melody moorefield wilson i'm here to talk to you about a topic that's very near and dear to my heart uh why can't we be friends building relationships between security and software developers

as i mentioned my name is melody moorfield wilson i'm a security engineering manager for pendo here in raleigh north carolina um and i'm focused on user behavioral analytics for pendo um if you're interested in learning more um chuck is a way better sales person than i am so please go chat with him just to give you a little more background about myself i've been working in software for about 10 years now and various roles including writing and designing software for clinical research i currently manage a very small product security program where which is growing we're so excited about that where we work closely with product and software engineers to design write and implement and test secure

software so let's start from the very beginning how did i end up here as a software developer i was required to reach out to security teams in more than one role and with several different projects to begin the security review process every company had their own process and engagement with security teams varied wildly by company by project and by individual my experiences with those review processes were that security would often say no without any concrete explanation for why it was impossible for me to prepare and gather the appropriate information that i needed to help the security team review my software design ahead of time because they didn't publish those requirements and in some cases there was

no queue or backlog visible so it was impossible for me to know when a design review was to be completed which made estimations of when i was going to complete my software project nearly impossible and finally many of the individuals on my of those security teams struggled to understand the questions that i was asking to help answer their questions and they would sometimes go even so far as just to refuse to get back to me at all conversely as a security engineer i now understand that security teams struggle to respond to software developers in language that they can understand and within the timelines required for all the reasons that we all know in this room

there are constant fires and constant requests review software features after they've already been built in some cases already put in production any request for changes causes huge rollout delays and in some cases rework of features that have already been deployed without going through security at all additionally feature teams rarely plan ahead for non-feature work and what i mean by non-future work is fixing bugs fixing security vulnerabilities fixing pretty much anything that isn't on the road map so when a security vulnerability is discovered software development teams have to choose between delaying a future release or with releasing it anyway and hoping that we have time to fix the bug later let's puts security teams and software development teams at

odds with each other because there's a constant conflict in priorities additionally many engineering organizations see security as just a checkbox a thing to get through without any real effort to involve secure to actually fix security issues but just to get a feature through a list of compliance checks so they can move forward so i'm going to tell you horror stories of the things i've experienced as a software engineer and also as a security engineer as i mentioned to you before i worked in a few regulated environments as a software engineer and i had varying levels of delays caused by security teams that seemed to have absolutely no understanding of deadlines once my manager required me to build a

web app to track our customer deliverables this is an internal application that only my team would have access to and i went through a security team review to get a vm set up because remember this is a regulated environment and i got just a flat out no within a week no reason for why just no so i decided to submit again same same request and my response for that was we'll review your architecture and we'll get you a vm but it'll take you nine months i had six weeks to deliver that application so what do you think i did built it anyway healthcare had different security challenges meeting hip requirements following appropriate guidelines for data protection and clinical research

and on and on and on one of the challenges with working in healthcare is that new technologies took a really long time to be approved by security teams some of that is fda regulations some of it was not i could submit architecture review documents to the security team and it would take months to review we had the choice to start development while the review was in progress to meet grant funding deadlines or wait for approval and get nothing accomplished until it was proved if the design change at all during that development process we'd have to start all over again with security review so i promised you i'd tell you horror stories as a security engineer as well

or someone who has a security mindset so i don't feel like everyone's getting picked on from the security side but we're going to pick on on the software engineers as well there was a system that i worked on that used a database with a shared username and password i hope you know where this is going this shared username and password combo was hard-coded into our data pipelines across our platform it was read write it was shared with more than 70 engineers it had never been changed when i asked about this and how much of a risk that was to use that singular username and password for all of our data pipelines i got a little bit of a laugh oh it's

fine after all you have to ssh into the environment and there's no possible way that's a security issue so just move along another time i mentioned that our systems once you authenticated via ssh could be accessed as a pseudo-user by everyone who could ssh into the system and might that be a little bit problematic and bear in mind this is anyone in the company anyone who can ssh into the system had access to this and i got another laugh because there's no way an internal threat actor could possibly uh not pass a background check and be able to just poke around in our system so i was just let's just move along very silly so as you can see as i've illustrated

both sides of the camp just really don't seem to understand each other at all so how do we fix this problem well first we have to consider what the needs are for developers as well as security engineers as i mentioned before developers are under really tight deadlines so they need a smooth unblocking security review process developers also need security concepts explained to them they don't always understand the why and you must give that to them to help make sure that they understand why they're doing the things they're doing and developers need clear tactical solutions if you just give them high level views of like you just need to do this thing this this is the best practice but you don't give

them discreet steps on how to make that happen they're probably just going to ignore your recommendations conversely security teams need a few things from software engineers security needs that time to evaluate architectural changes they also need to have tech that addressed as we've discussed earlier today patching is one of the biggest things that you can do to help solve your security issues often that's not something that developers prioritize and security needs to be included in all places in the software development life cycle not just at the end hopefully not just at the beginning but across the board so where do we go from here we know now what we each other needs but how do we

get there so security teams can do very well by treating their software engineers like consulting clients you're here to provide them a service but not to dictate to them collaborate with them early the sooner in the software development life cycle you begin engaging with software engineers the sooner that they can pivot when they need to so also alleviate to some degree the time crunch of trying to fit security at the very end of a developed feature and will save your engineers time and frustration and i want to reiterate something no question is too simple to explain in order to build a culture of collaboration and encourage your engineers to continue to reach out to your team for any and all questions you

must not be snarky treat them with respect software engineers know a lot that you don't know and you know a lot that they don't know if you meet in the middle then you can solve this shared problem understand deadline commitments your engineers are really really busy they often have absolutely no control whatsoever on feature delivery timelines and keep that in mind and finally partner with your developers to help guide more secure software development processes a lot of times when engineers feel that you're there to help help you will be really amazed at how open and willing they are to advocate for security on their teams that ripple effect can be really informal like you have a team member on

a feature team that is the go-to person for other engineers to answer security-related questions or likes he's going to talk about a little bit later it might be a formal security champions program the important thing is to expand your influence by being approachable and helpful so what can developers do to help bridge this gap treat your security team like development partners they're they're here to help they're here to consult and advise and let them know what your timeline restrictions are if they're not aware of that they certainly can't adjust their own workflows and backlogs to accommodate you security engineers cannot help to be proactive if they don't know when you need their help plan to speak to security at the

beginning of a new project and as part of your software development process always do it consistently do it often and you'll save yourself time in the long run work to understand the why of the requests this is task required to maintain essential compliance requirements often us security engineers are attempting to follow strict requirements that we have no control over either a great example is password requirements for pci we talked about that a little bit today we're required to enforce these types of regulations and there are serious implications for ourselves and for the company if we don't follow them and finally include security and design sessions security teams are uniquely trained to help potential flaws and software architecture be alleviated

and pointed out we spend a lot of time training and researching to help software engineers find these potential flaws it's beneficial to everyone to include your security team and your software design whether it be a real-time discussion a easy zoom call or in the form of written documentation so as security engineers how do we approach this problem and and continue to move forward to solve it like i said before asked to be included in design sessions you'd be incredibly surprised by how open and honest software engineers are and they really do want your advice it's much easier to change design at the beginning of a software project than it is at the end another thing you can do is collaborate

with engineers on threat models threat models are great tool to introduce software engineers because it provides them with an opportunity to start to think through potential security threats on their own we've seen this appendo and it works it's a great way to introduce them to adversarial thinking which is not something they're accustomed to and it only helps them to be better software engineers additionally if software engineers include threat models as part of their design process they will have they will save yet more time reducing rework another thing you can do is help to review code for common security flaws sometimes it's done manually and sometimes it's done with automatic scanning in my experience both are very valuable

it's yet another opportunity for you to build relationships and have conversations with software engineers as to the why why are they changing this code why do they need to follow a different design pattern than they're accustomed to in some cases being part of a particularly security conscious software updates and features are really a good place for you to also inject your advisory services so for example if you're implementing oauth or some other very security conscious thing you can ensure that you're helping those software engineers make the best design choices and i can't reiterate this enough don't be judgmental be willing to educate and explain software engineers are working very very hard to do the right things

they're under super tight deadlines they mostly want to help so just keep that in mind another thing you can do is write stuff down it's really helpful security engineers should document best practices so that your software engineers don't have to go on a zoom call or 45 minute training session they can just read it when they have the time make it prominent make it share it widely and be open to questions or feedback from your engineers on how to tactically implement those best practices anything you introduce to your engineers as a requirement must not cause too much undue friction because they will straight up ignore it if you do nobody wins in that situation so i use a little bit of an example of

some of our documentation here at pindo um pretty high level stuff but it at least gives some baseline information on best practices additionally one of the things that we are really careful about is not assuming that certain terminology is widely understood so we make sure that we actually define those types of things so we have a baseline to start from another thing that can be done to prevent an influx of queries that could take time away from more pressing work from your software engineers is to provide things like decision trees or flow diagrams to help them make decisions on when they should engage with you and how this gives them a really clear distinct process to follow on when and how to

start a given process so this is one that we use for threat models um appendo keep in mind that every organization is a little bit different so our approach may not work for you but it just gives you a baseline on what you can start from more importantly of all this we want to weigh the value of including processes across every design session and but including them only when they're deemed necessary so instead of saying you must do this always at all times we want to make sure we provide them the resources to know when it's the most important to do something like a threat model we also provide bite size training this is really

important um we try to provide short videos or one pagers um so that our engineers will actually read and watch it um we're a software company we're moving super fast um and i'm sure every other person in this room is also moving much faster than they probably ever expected to move um and we're we never have enough time um so anytime that i can make it easier for them to ingest this information provide value from it and then move on about their days is is a win for us and i will point out that engineers care far less about production value and more about content so i have some very goofy videos that i made they're not production quality by

any stretch but that's not really the point the point is to provide that information in a way that they can understand

and i know i've said this over and over again but i'll continue to reiterate that the best thing we can do as security engineers is to have empathy for our colleagues most developers care about security they really do i work with engineers every day i promise they do care conversely we should be extremely cautious about how we implement security controls so we don't want to put in extremely uh restrictive security controls that prevent software engineers from being able to their jobs but we still want to make sure that we're doing everything we can to help secure our environments you must continue to build bridges with your engineers it's very important or you will not be effective at protecting

your business from security threats we're all in it together we're trying to build software we're trying to secure our environments all of that wraps up in with high quality products you have the option to collaborate with your engineers or suffer the consequences if you create an adversarial environment you will not be the best at what you do remember we have a vested interest in building secure robust high quality software for our customers for our vendors for ourselves since we have the same common goal there's no reason we can't work together to help make our shared mission happen to that end build bridges treat your engineers like teammates because they are and be excellent to each other

i don't know if this video is going to play but i think you know what it does let's try it you can't hear it but he's saying be excellent to each other and party on

thanks everybody