Are We Too Early for the Party? The Perils of Baking Cyber in from the Beginning

all right everybody the next talk is very interesting are we too early for the party are we the parents of baking cyber in from the beginning please welcome our speakers lilan Ash Baker and Steve bickler over to you guys all right' thank you so we're going to do a quick little introduction of ourselves my name is Lily and as or lily uh a little bit about myself I work for a major airframe manufacturer in Aerospace um prior to that I worked at Collins Aerospace for 15 years in uh primarily in navigation avionics doing uh civil certification across support um development test manufacturing everything in between uh I also spent a lot of time in a small

airplane doing circles a lot of the time so Vick yeah Steve bickler go by Bick like the pens and the razors cheap disposable marginally effective uh 23 years in it and cyber most of that was military time about 21 years now I also work for a major Aeros Aerospace airframe manufacturer there's two of them and my opsec was really really bad in the uh in the Bios online so just don't tell my company um that that got through um so yeah so here we are let's talk we wanted to talk a little bit about where we are in our current development with an uncrewed aerial vehicle so the company that we're working with has done a lot of

prototyping and preliminary products and they're about to do their first civil certification how many have done Serv certification on air frames here before not miss not miss not your DOD stuff if any of you guys are goes like so doing civil SE is a completely different Beast than doing any sort of other civil uh certifications so we're kind of in the very beginning section of civil certified product development which is to say they're building a product and we're the Cyber SE folks trying to make sure the cybers are right at this point where nobody really knows what they're building or how it's going to turn out so a little bit of a crowd acity here please raise your hand if you have

said if we would have built cyber in early the system architecture would have been easier to secure hands hands yeah okay all right all right our operational system would have been better integrated with our security solution hands okay great great security and compliance measures wouldn't be so expensive you guys are probably still right on that one actually the Security Solutions would integrate better okay all right good good all right the red team wouldn't be able to get in oh thank God thank God I threw that one in there as a as a test for the audience thank God like yes the red team would still get in but we wouldn't have been breached no no we're not going to do

that we wouldn't have had issues meeting compliance okay that's okay that's okay it's okay this is a safe space safe space if you don't tell our bosses we won't tell that you guys said it we wouldn't have had to pay for Recovery Services and that's all just real real easy to say isn't it okay and one thing that I want to highlight here is that we're talking about compliance uh quite a bit because in civil certification compliance through every stage of development is incredibly important is as important as the final certification itself so here's our reality upper management of course is saying what upper management always says everything's important especially cyber so we take cyber real seriously all

right and then of course cyber security we're giving them all the requirements here here here's what we think meets this situation we also made ourselves Jennifer Aniston because you know hot and smart right um then of course there's the compliance folks and it's in civil certification there is no list it's all standard so it's like you can do whatever you want but we have a giant manual so you might want to check that out and then our product ipts are so so who are you guys which the first version of this was you guys have requirements because that's seriously where our product IPS were as well but we've learned over time interacting with them and they're like

so what's a cyber again uh that happens and yes I have fielded that question from some of our product teams yes oh IP integrated product team integrated product team integrated process team for some folks but for us it's product team so this is what we're dealing with we have a chief engineer who says build us a comprehensive cyber security solution for our program but under no circumstances can you interfere with the current development as it's ongoing it's a paradox Marty I'm telling you it's a paradox and we've actually had that exact conversation we were trying to put out a memo that just says these are the basic cyber things we need you to do for

your first development of this airframe and he like okay I don't like that I don't like that why do I need encryption why would anybody want to encrypt aom link things like that hey did that ever get released no no it never did well actually no wait we just ignored him and we released the letter anyway because nobody's reading it which gets into the cons all right the cons of building cyber in early especially when you're building it in as early as we are is everything is variable so my example on this one is I'm supposed to be the test director but I'm really just the threat modeler right now while Lily does all the work which is great for me and not

so great for Lily but everything's variable so we built this beautiful threat model after requirements were done is the entire architecture changed because everything right now is just a block on a diagram we're not talking about actual having gear or networks done it's just a block so it's like ooh you gave us a security recommendation wait it's gone it changed it twice yeah it changed it twice um this the tldr too long didn't read yeah I think we just explained that about our chief engineer that's pretty much what happened and then we released the letter anyway and said we're going to go do this for all the it folks in the room the certification Focus over cyber Focus

it's a con but it's also like necessary with civil aviation like we we have to do the certification right we have to make the certifiers happy especially because everything's kind of fungible with civil Aerospace as opposed to in like the dod world where I come from where everything's n and it's checklist based so there's a part of that that's there but at the same time we all know I think we all know from the hands before is like just because you built a certified solution doesn't mean it's a secure solution and so that's kind of where we're at um and then I think you probably most of you if you've done any sort of early stage development whether

it's it or some sort of OT system we're kind of in the way when we're early or at least that's what product managers think like Hey we're you guys are slowing us down by continuing to show up at these meetings like Lily does and say hey have you guys thought about this yeah well that's going to slow down progress and we don't want that because remember they plan to have 50 requirements reviewed this week but now there's 60 requirements because you just dumped your tent on top of them and said you guys got to go deal with this yeah um how agile is your agile this one is like everybody's does agile differently so like on the scale of ion

Rand to Carl Marx when it comes to Agile we're somewhere around Joseph Stalin so it's kind of like if you don't get your tickets done by the end of the Sprint we might Purge you forever we'll delete you from the pictures you'll never be in the program ever again so like we're dealing with a pretty agile or a pretty unile agile right now and that makes stuff hard as far as like trying to show like look cyber doesn't fit well and agile anyway okay so here we are trying to figure that stuff out and then kind of goes with all of these if you've got all that other stuff going on how do you Justify Your Existence how do you say

you need we started this what almost a year ago now now the idea was we'd have six people you're looking at the people still still still the people right we've got beautiful charts that say what we need as far as people and manpower to take this project through till actually flying this thing and one point that I'll make here because I have to talk about Regulatory and certification compliance is that this part about justifying our existence it's Justified for us yeah except the cfrs have not been updated to justify our existence so we don't exist yeah Lily's more positive so she's going to tell you Pros so one great thing is that of course in product development

especially in system development the sooner you get your requirements in the less rework you have across the requirement life cycle right this very simple V model uh waterfall model that gets used quite a bit earlier in development we means less development costs everyone's been saying this for a very long time also when we're talking about Hardware cyber requirements Hardware is working on their certified Hardware solution very early on you have to have your certifi your cyber Hardware requirements in early enough to get into initial prototypes because if you don't it's going to cause respins it's causing rework new hardware has to be ordered and developed that's expensive you get a chance to actually mature your cyber solution as you go

through this process so as we said where we started from and where we are today our thinking has really shifted along with where the architecture is going so all of our assumptions up front have either been retired or changed into realities and so now we can go in inte that into our cyber solution as we keep moving forward compliance in civil certification is tightly coupled into the Cyber solution we have to produce documentation at some point that says what testing we did at of the Cyber solution in our system so the better documented that is for Regulators the more proof we have of our tests to as the system is developed then the better off we are

overall with our certification basis uh there's also a our focus is with the security solution and compliance in mind we're thinking about how we're going to structure our documentation and all of our testing to take what we've done put it into a package so when we go to our regulator FAA or FAA delegates that we can actually explain to them what we've done and where the work came from yep so how best to secure the beginning so first of all if any of you find yourself in this situation it's not going to be easy I think we've already start to outline that and the big thing is don't expect to be on keyboard um I had I moved from

another program with Promises of being on keyboard and doing pen testing and you know and early process pen testing stuff yeah I'm not doing any of that I'm not doing any of that and the thing with civil aviation is well you say that you're not on keyboard but you're producing a lot of documentation a lot of documents I am really really good at making documents now I am the the highest paid Tech writer at our company I think um which is okay but like you know it's all right um so like that's a big thing I think for folks is just just understand if you're in this situation it's your expertise that's going to be

in involved it's not necessarily going to be your technical skills it's going to be using those technical skills to produce a road map for them as opposed to being the person who's doing it yourself yeah and we say that you know is being a problem but we're engaging into these teams even though that they may not be coming to us we're going to them we're telling them we're here we're available we need to work with you what's been really promising is that there are teams that have been thinking about cyber security in their design but had no one to talk to because they didn't know we existed so when we roll in and start talking to them they say

we're scared we are so scared about the cyber security part of this that we overbuilt this can you please tell us where we went wrong and why is it overbuilt can we reduce this um and with that some other teams are better in taking our requirements and our inputs from cyber security we've actually had very positive feedback from these groups of like we've been able to sit with them review their requirements review their models and get feedback immediately about what they're not doing right and what they're not taking care of now they are the same people that also changed the diet changed the entire threat model on us but at the same time they did it

for the better so like that becomes like the thing is like they are engaged they were thinking about it they are the most networka aware internet facing publicly facing thing it was great to see that they had done that engagement like they were willing to talk to us it was awesome governance is good uh you know I jump in there as just saying that about our our team that helped us like we build the processes we can stick to them right like I'm not a big process guy I don't like building processes I'm pretty disorganized you can ask my girlfriend in the back or my kid like I'm pretty disorganized But ultimately when it comes to this

business you got to you got to lay some groundwork especially when folks don't understand cyber security the way we in this room or in this profession would on an i in an IT cyber security realm if it was this was just networks so you look at the requirements and you build governance processes around that requirements plans strategies God I can't I got two more to write this Sprint I can't anymore but ultimately it is good because those things are there and they're there in the beginning and we've given these folks the road map which makes our requirements building hopefully a little bit easier which gets to compliance as the basis for security I know compliance

and security aren't the same at the same time if your compliance isn't right your network probably sucks let's just be honest like you got to get the compliance at least kind of right and then you start doing the security from there yeah oh crap we're going too slow sorry Lily well and with that since you're building out these models and you've had to change the models and you've had to change the models and you changed the models the third or fourth or fifth time you can sit there and refine and iterate through those models it gives you a fresh start every single time to get through the model so that when you do go do your table talk

exercises or your threat models you have something that should be fairly mature overall in the process it's a good way to work on your assumptions and move through them early so the bottom line is it's better to be early than late right so all of our all of our assumptions that we said at the beginning when we were all raising our hands it's all true earlier is better it's just you got to be flexible you got to you got to be willing to see that everything may change on you in in a whim and and if it and if it does you got to be ready to go back to the drawing board and start working your

requirements again chances are it won't be all the way to the beginning but it it might be it very well could be yeah but with that in governance if it's a big company that has a lot of policy and procedures in place use those policies and procedures to guide the process and keep it going forward if it's a small startup you know that your gu your governance is going to have to be there to back stop you as things change you have to use it as some kind of basis point to move forward again uh with that building relationships with the product teams and also the product or the teams that are built outside of your product

so if there's any sort of support functions or ultimate customers that are going to be taking over your platform at some point you can bring them in and start building those relationships to understand how they're actually going to start using the product so that your cyber design takes that into consideration and model what matters right so what we found in our first model was super detailed and super expansive it also wasn't super realistic for what we were dealing with at the time so if you're going to do your threat model threat model based on what your actual environment what you know is tangible what you can actually sit down and go for us it's an aircraft and a

ground system okay cool that's way easier to start going down into the down the road of subsystems and interactions from there as opposed to trying to model out all the external factors that might be in case in place I should say say yeah and then don't push off your work do what you can when you can right I write a bunch of policies and procedures right now because that's what I can do right now and that's you know it may not be what we wanted to do but it's what we're going to do because that's what's needed right now and management needs to understand that you need flexibility and agile environments so one of the big problems

that we started to experience is siloing all these different product teams were starting to become siloed within their sprints and their uh agile teams we were also being siloed within them you know their answer was well go talk to the teams and get them get them to put you on to your Sprints but that's planning work out almost three four weeks a month maybe two sprints ahead of time cuz they got their own stuff to do they're not super interested and vice versa we have to add them on to our Sprints early in order to get them to come participate in our stuff agile kind of needs to be a little bit more flexible in that where

we can pull in individuals and start working between them so what you got for questions for us I know it was not super we we knew it wasn't going to be super technical hacking stuff but like what do you got for us ma'am hello okay hello um can you hear me absolutely um going back to your point about how to navigate conversations with product managers who are pushing you know the speed of things right it kind of remed me of this quote speed is the natural enemy of good security design yep how do you on the compliance side you know trying to have these conversation and push you know what really are regulations and protocols how do you what sort of like

social skills do you use in conversations with those product managers given that they're obviously going to push their agenda and their priorities you have social skills other than just like you're going to be fined if you don't comply what sort of you know what I mean yeah so I think one is happy hour y that's been a really good thing um the couple times that we actually all meet in person happy hour has helped yeah so so like the one of the managers that was giving us a lot of push back his organization is cyber plus systems teams governance teams um IV ivnv process so like we're a small slice of his world we have to

prove to him why we're important and why we matter and the best way to do that is get talking to him about what makes us excited about cyber talking to him about Defcon or bides or any other security RSA since they're pretty close to that side of the world like getting them in in interested in this world that they don't understand and why it matters um that's that's the advocacy piece of it right cuz we're doing a lot of advocacy for cyber security the other piece of it that comes is and it's it's boring you know cissp sism stuff but it's hey here's the risks you're taking right we we wrote a memo they're taking some

risks to get to a first flight of this thing we're like okay you don't care about cyber because you want to get this thing flown here's a memo that says what we're going to give you for cyber for this first time but oh by the way here's the the list below that is all the things you're not getting which means your timeline's going to get either really compressed later on or it's going to be real you know it's going to take a lot longer than you think and that goes to that same person and we built that relationship with him and he's like oo I don't like that I'm like well you don't have to like it but it's kind of reality

at this point and since it is a um a civil certified product and we have to take our documentation through a game process in reviews with our regulatory reviewers it means that if our documentation doesn't show up at that review we don't pass the gate so it also puts up roadblocks of like if you want to pass this gate we have to start our work six months beforehand and make it to that point with everybody else they all have to sign off on a document for anybody who's done compliance before they have to sign off on a document that says how they're going to do compliance essentially three to five years before they can make changes to that document

along the way but they at least have to here is the way to get to the answer or here's the way we're going to take to get to the answer and that's an advantage we do have in this regard it's like that's stuff's got to get turned in y any other questions from want more question one one you had mentioned and I think aptly that uh security doesn't really fit into to the agile framework I was curious if you have any tips or practical advice for explaining and working with teams to help them better understand that it doesn't fit that way so I like that question because it it's not just a security issue that we're all

dealing with here it is more of an organizational issue because we're not the only ones feeling that pain we don't think that we're the only ones feeling that pain and we can see other organizations or other teams also feeling the same pains and like we all kind of have to commiserate together in order to figure out how to fix the I mean let's be honest in the cyber security engineering side of this where you're at Building Solutions like we're a support function if it's not a network that you're building we're a support function for this airplane that's being built so like we're not the most important thing and that that kind of becomes like the issue and agile of all

right like you want me to get my tickets done but this is going to take six months well I need you to fit it into a neat little four-week block all right cool there's a phase one phase two phase three phase four phase five phas 6 phas 7 8 9 10 11 12 ticket for this and I'll show you what the progress is but that's as far as I can give you this is a six-month thing and you want to see something in four weeks that's what we've done is we've phased tickets that's one of our that's one of our work rounds I should say for agile thank you guys great JG thank you everyone for joining thanks everyone for

coming thank you everyone for coming