Threat Modeling: Removing the Mystery

and let's get started I'm gonna introduce our keynote speaker dr. Frederick Scholl here Frederick shoal is a program director and associate professor of Quinnipiac snoo master's of cybersecurity program and prior to this position he started the cybersecurity program at Lipscomb University in Nashville he taught information security at Vanderbilt University and managed enterprise security for Nissan Americas he has also consulted on information security since 1991 and he is a senior member of I used to say and ie so put your hands together and welcome dr. Frederick show thank you can you guys hear me okay if anybody wants to move up I won't be I think it might be a good idea I made the slides big but not so

big for the back of the room but aren't these things cool these are the best badges I've ever seen at any Security Conference right you all agree how many people are new here just out of curiosity - besides CT how many new attendees Wow quite a few right holy cow plus I saw it was sold out I'm so happy you guys invited me to be a keynote speaker it's almost impossible to get into any of these besides so so my my talk is gonna be about threat modeling but then I remembered this is supposed to be a keynote speech this is only my second keynote speech so I had to try it you know figure out what should I do for

a keynote speech so I I texted Jack Daniel right you all know him he kind of started besides and he said well don't make it too technical so I said alright that's a good idea then I texted wind swart ow I'm actually based in Nashville now and he's in Nashville and he gives like keynote speeches every week and he said well try to be a little controversial then how many people went to der beek on one way in the back anyway there was the keynote speaker there gave the presentation in his underwear so I'm not gonna do that I can't I can't go that far I'm in academia I have to be a straight forward

so but I am gonna try to think it slides to work I am going to talk about threat modeling which is some work that I've done for two agencies when I was doing consulting for Center for Medicare and Medicaid and also Vanderbilt University now I'm in academia but I have to have a word from my sponsor and actually I'm pretty excited about that so in September I started a new position I'm actually moving from Nashville to Connecticut so if you know anything bad about Connecticut don't tell me okay I'm already here it's too late I know about i-95 and I know about the Merritt Parkway I don't want to hear about anything else so so we are starting a

new online cybersecurity master's degree program it started in September I'm excited about it for some reasons I'm not gonna spend a lot of time on this but we are we are focusing on NSA nice cyber defender compliance but that's not all we're doing I mean that's kind of the minimum I'm really focusing on cloud security I'm I just see so many things going into the cloud so we've got to get people practicing security sort of as well in the cloud as they do on Prem and then the same thing with software security I have an electrical engineering background unfortunately so I don't have a really good software background but I have been really trying to learn this stuff and I see so much

pretty much everything in security is going into software so I'm emphasizing that it's fully online and we have enrollment in fall spring and summer so if you're interested in a master's degree obviously reach out to me you know we have our next classes starting in January on the other hand if you're interested in teaching one of these courses reach out to me to our courses our one credit hour courses so we purposely divided it up into really small bite-size chunks so we can get outside industry experts to teach a specific area that they know well and also if you if you're from the vendor side or even user side and you want to do a guest lecture I can easily bring

you in as a guest lecturer using zoom and then the last thing is if you're a vendor and you have products you want to sell I'm happy to consider those two you know we can incorporate them into our class as long as they're free I'm happy to use them so so the agenda for today I wanted to throw in a little about where do I what are the big picture topics that I see insecurity and then relate that to threat modeling tool so I'm going to spend a few minutes on that then talk about what is threat modeling it's something that I see sort of coming up in the world or people are becoming more

aware of it then I heard this good talk by wind swart ow my mentioned he said he's had this awareness training company in Nashville for many years and then so he gave a good talk on how to fail at awareness training so I'm kind of copying his idea and then I will go back and talk a little bit about how to succeed at threat modeling I want this about eight different tools that you can use you don't really need any tool you can use a spreadsheet but I want to talk about some of the tools that are out there and then one of the big challenges where do you get good baseline threat libraries and then I want to talk about

some of the use cases where I've used threat modeling as I mentioned that CMS in Baltimore and then Vanderbilt medical in Nashville and then some of the opportunities in the future so this is the one continent that I see in our industry right now everything seems to be changing I what what do you guys think what what is what I have to turn the mic I have to stand in front of the mic okay so repeat the question which is what what is this group seeing in terms of changes in the field or changes in your job and security I specifically in the last like 12 to 24 months anything more mobile device use or okay

what's that crypto jackin ya pardon a PT's okay I haven't seen that one okay that must be pretty new okay ransomware is a cloud service all right what else in the back exactly mass yeah like lemmings yeah mass mass rush to the cloud that's hard to pronounce yeah anything else has got to be more exactly privacy standards yep that's another one yes sir okay how about IOT exploits right they're coming how about in terms of your jobs or positions or roles our company's more open to security initiatives or spending more money or less open or about the same same okay

yes sir they want to spend more on products and professionals yeah what could we do about that that's a problem we're perfect we're not products here right so

so I think these are all good points this is somehow I see the field is like in the last 24 months it's partly because of the cloud or other newer kinds of things that we have to do right one gentleman mentioned privacy fake news is another one IOT so we're being given new things that we have to do and so I've been tracking this I should have tracked more of these things then I could be in the recruiting business but I tracked one job trend since 2012 so I started teaching security in 2012 and I always like to look at what do people need to know when they get out of the program right so I

started looking at this and I saw I can't move here cloud cloud jobs were going up a little bit right didn't nothing too exciting but I did include cloud in some of the courses I was teaching 2012 then at 2013 a small increase didn't go up too much then 2014 wow it really started to go up wave is kind of stopping right it's sort of leveling off there and then look what happened in 17 18 19 we're not up to 19 yet right was up to 18 anyway I just did that last data point yesterday 114,000 cloud jobs now this isn't a scientific survey I am an electrical engineer but this is I didn't use data analytics because this includes

jobs in st. Cloud Minnesota and Kate you know if they're I don't think they're growing that fast though right so I think it's you know I'll stand by these numbers so this is kind of how many people are working in the cloud now in their in their current job well not that many so there's still a lot there's still a lot of room for growth right so this curves gonna keep going up that's what this quick poll shows me Wow exciting okay but I think this is changing everything because so many of the tools are in the cloud you need you need to set up networks at the same time you hear them as well as securing your on-prem data's

your traditional network so this is a huge driver and then here's another one I don't know has anybody read this book by Mark Schwartz it came out last year seat at the table nobody so it's this I I was in this area lived in this area for about 25 years moved to Tennessee for 12 years sent two kids to college they're still there thank goodness and so now we're moving back but I was a member of the Fairfield Westchester sim chapter Society for information management and the reason I stayed in that chapter it was a an organization of CIOs and I always wanted to know what our CIO is thinking and this guy mark Schwartz was a CIO of the US Immigration

Society and in in DC and you can imagine what a tough job that is right so I guess he left now he's working for Amazon but he wrote this book talking about the the trends in IT and the trend he highlights was simply that the business people are taking over more and more of the technology function right it's not just standing in the CIO world and he said well you if you want a seat at the table this is what you have to do and so for example when I did this threat modeling work at Center for Medicare and Medicaid in Baltimore it's a very interesting project we were developing the next generation Medicare and Medicaid payment systems and guess

where the developers and technology people were they were not in the office of the CIO they were out in an agent in a sub division of CMS that's they were all there so we in the I was consulting in the central security group we had to influence them they weren't even in the same department that we were and so this is what he's saying CIOs need to make this transition from running everything to kind of setting standards and being influencers and I see the same thing could happen to security I know some organizations that have very strong security folks out in the business units and so security could follow I think it will follow get

we're engaged with the business so just out of curiosity how many people here in their organization see the security team are parts of the security team directly in the business units themselves in any of your organization's they are okay and anybody else so in the back also Oh quite a few people then right so that's not a mega trend yet but it's kind of an initial trend that you know more of us need to learn more about the business and so that's what this book was about about a year old he switched to Amazon but I still think it's worth reading so I'm gonna propose threat modeling as a way it's a tool it's a technology but

it's a way to engage that it's a way to manage risk and to engage the business because it looks at risks values threats vulnerabilities all in one screen and that's kind of the holy grail of security right if you can connect your threats to your risk to your assets and you can make a gazillion dollars if you know how to do that so this is one possible way of doing that so so I as I said I did I don't claim to be an expert in threat modeling I'm I'm a change agent now I'm really trying to help working professionals make as one student said 90 degree changes in their career path to get into security

hopefully not a hundred and eighty degree changes right I'm glad he just said 90 degree changes but anyway so the work I've done over the last few years was for CMS and Vanderbilt health and I'd played with threat modeling and it was kind of frustrating because what is threat modeling this is such a dumb term what does this mean and then finally I figured out that it's really just risk analysis for applications and systems and this guy Richard bate leaked he's pretty famous right I forgot exactly what he did but he's like a household world in security and he wrote this very good blog post back in 2007 talking about hey this really isn't threat

modeling per se it's just risk analysis that we're doing it's a risk analysis method the term threat modeling came about from Microsoft back when they came up with this right early 2000s so all you're really doing is looking at risk and you're starting with your threats usually we do vulnerability management we start with vulnerabilities or we look at asset values in the case of modeling you're looking at what are the threats that are out there so it forces you to go through catalog your threats and understand what the impact or the tool helps you figure out what the impact could be on the business as I said it's a process and a tool so why bother to do this so I'm

going to throw in some compliance stuff of course we have to start with compliance so if you are working with 853 you're not required to do threat modeling but there is something called a control enhancement sa 15 control enhancement how many people do work with 853 anybody quite a few right it's sort of pervasive if are you guys government contractors or part of the federal government or both both okay so 853 is kind of complicated right but if you if you have a sensitive system you can put in place what's called a control enhancement and threat modeling is one of those control enhancements if you work for DoD they have something called the DISA stick anybody working for DoD

here one in the back okay so you guys you raise your hand for a lot of stuff right good they have the DISA stick for application security and development does require threat modeling for level 2 applications and what else do we have so good practice under a wasp Sam security application security maturity model I think that stands for it is required for maturity level 1 safe code safe code was kind of started by Microsoft by the way safe code how many people go to safe code or scan that site anybody one or two that's a really good site to understand application security they have a lot of good training videos and things there they have stuff on how to use threat

modeling and it sort they have a desert best practice of application security and they do require threat modeling also and now that I'm in the education business we're using the nice NSA knowledge units and they do they have it in their lifecycle security knowledge unit so I thought that was interesting what else so it simply a means to facilitate conversation with business and security if you're in security you want to get out of the security silo and get out and collaborate with your developers how many people are developers here all right I hope I can talk to you guys because that's that's a challenge for the they speak different language than the business guys do so

I'm trying to learn the application development this is a tool to get your developers and your security people in the same room so also Gartner I just saw anybody from Gartner here nope okay well I guess I can say whatever I want but anyhow they they we're pretty close to Gartner they might be right outside they do have this hype cycle right you've seen the hype cycle and this is for applications security I thought this was pretty interesting because they do when something becomes almost mainstream they give it an acronym and so they've given threat modeling and applications ASTM now that's their acronym so I don't know if that means that the whole field is gonna collapse in two years or it's

gonna grow one or the other right but it's it's interesting it's on the rising this is the expectations up at the top as they call it so modeling is trying to get up to that area we'll see if it gets there okay so what are the goals of threat modeling you want to add everything in security I had the good opportunity to meet Ron Ross the head of the cyber program at NIST when I was working for CMS so he's really big on and everybody's big on let's identify risks upfront identify risks up front that's his mantra and I make so much sense because we in security it's amazing that we accept the idea that

well we're gonna have to do static testing dynamic testing we have to do pen testing we're gonna have to fix things after they're built imagine I used to work for Nissan imagine you know cars coming off the production line people testing them allows this car working or not working you know nobody would do that right people would get fired when those cars come off the production line they work and they work for a hundred thousand miles but in this field security systems development we accept the fact that they're going to be errors in the system and then we'll fix them later and hopefully before the hackers do so one of the goals of threat modeling is especially identify your

architectural risk for new systems it doesn't find code flaws it finds design flaws and I think right now we have enough code flaws and design flaws I think some of the code flaws are gonna get fixed as people use more standard libraries you know I'm an electrical engineer you wouldn't go build you know create your own chip if you're building an integrated circuit board right you just buy standard parts that had standard test and I think that's I see that kind of happening in the software world I don't know if the developers here would agree but more and more available libraries are going to be used instead of creating your own code and then neutral platforms so not all the

tools I'm talking about or neutral something like that Microsoft has a threat modeling tool I'm calling that sort of neutral maybe not right if you're working at Amazon but you want some kind of a neutral tool that enables collaboration of security and development and the business people and you want to get a different point of view as I said looking at things from the point of view of threats not vulnerabilities so a different point of view I pulled that couple slides here so the trick about risk is always what point of view do you have right are you looking down in the weeds are you looking up at you know ten thousand feet up in the air neither one

of them gets all the risks right so you want to look at both I saw this mouse here I guess he missed the risks from up above and this mouse missed the risks down below right so there's risks at all levels and as part of for security professionals we really need to be looking at all different levels and how to fail at threat modeling and it's pretty easy to fail at threat modeling so one thing is so you have these tools whether it's Microsoft or some other tool and they have the ability to suck in a lot of data and a lot of information about systems so if you pick the scope too big guess what you're

going to get you're gonna get hundreds of threats and your developers aren't gonna be nobody's gonna be able to handle it it's like vona it's kind of like vulnerability scanning right so if you scan your whole system and just look at all the threats without any kind of way of prioritizing them you're not gonna be able to do anything so the same thing with threat modeling you have to come up with a scope the scope could be a system it could be a switch it could be a new cloud application you have to look at it at the right level too and have some agreement with your business folks this has anybody seen this movie

the birds maybe I'm too old a few people have seen it I kind of like this movie so this is where the birds are chasing the kids out of the school so this this will be your developers you know if you don't have a well-chosen scope and you run one of the threat modeling tools they'll just head for the hills so you have to really pick the scope correctly it needs to be you know the Goldilocks here not too big not too small just right so what about what other ways are there so one way is to think you're modeling the kill chain do people use the kill chain here the mitre kill chain a couple people use it

three four I don't know if everybody else is just not paying attention or tired or whatever but you're not really modeling the kill chain I wish I could see my I can't I'm back so the kill the kill chain so just as an example the beginning of the kill chain has reconnaissance right you're looking to see what holes does the system have then your weaponizing you're delivering so fit modeling doesn't model this whole thing you're not modeling reconnaissance for example you're not modeling maintain persistence on the right you're only modeling one piece which is when the person breaks into the system you're not so it's not perfect it has a lot of things it doesn't do it doesn't model

lateral movement so if somebody breaks into a system we don't have good enough threat modeling tools so you can model how you hop around from one system to the next that's what we'd like to have but the tools don't do that yet they just say they do which one a tivo are you a vendor are you here outstanding okay I'm gonna talk to you afterwards very good so most vendors do not allow you to do that put it that way I don't mind being corrected so that's very interesting so in general yeah you have to be sure what what what are you modeling and there's also the the mitre attack framework you're not modeling all the

steps except for maybe the gentleman in the back you're not modeling all the steps in the in the mitre attack framework I tried to me I tried to do a tool comparison can anybody in the back see this you can like so exciting art I tried to do this last night so people like 20 feet away could see this but and I guess the slides are gonna be available right so if you want any follow-up information I assume you can get these slides so I just did a super quick evaluation of the different tools that are out there the only ones I've really used in depth are the first two microsoft threat model err I mean

Microsoft threat modeling tool and then threat model or the second one the others I've looked at a little bit so Microsoft TMT whatsits I put strengths and cautions that's what Gartner does right I wasn't going to put strengths and weaknesses I thought he might get in trouble so I put strengths and cautions so one strengths is it's free you can download it and start running it I think it's very cool and it has really added in the last I'm not working for Microsoft but they've added a tremendous number of new icons based on Azure so and they've also probably tripled the number of threats that are in there in the last 18 months or so so it's it's a

great way to get started it's free one caution is they don't have a lot of information on how to fix things so you find okay here's a threat they don't really give a lot of information on what to do now this next tool threat and it's also as I've seen it it's more of a one-person tool one person use it maybe you have a group sitting in a room but it's not an enterprise shared application this next to a threat modeler is a commercial tool it's web-based software as a service multiple people can use it at the same time it's so commercial it's enterprise strapped modeling tool it's not free it has a built in it does have a very good

built-in knowledge base of threats it has I don't know four or five hundred different kinds of threats associated with different components so it's very good caution it's a smaller company with seven kind of deliver partners and another company that I learned about his continuum security they have a product called arias there in Spain the strength of this product is very closely aligned with the development process the guy who's running it I'm blanking on his name now but he's been in application security for many years he's like a real guru in that area so that's good if you want to use this specifically for doing threat modeling with the developers so it ties into things like JIRA then fair is

another one has anybody used fair in their company you have how is it working out + - no comment

yeah yeah that sounds fair so yeah I described it as an I have not personally used it I've seen demos it's an enterprise tool it may not be suitable for some fur it's great for enterprise I think but it might not be so good for fast-moving agile development to development teams it might you know it might be really hard to implement and then another one security compass up in Canada they use a simple questionnaire approach so the team fills out a questionnaire and then from that they're able to do analyze threats that might be out there there's some other open source ones oh wasps threat dragon so I think I've seen a number of these open-source

projects get going they don't seem to have gotten traction this one only has four contributors sea sponge is another one last commit a couple years ago and then the newest one is two to Mantic has anyone tried that by any chance er it's it's a startup company and they're their premises they want to take a Visio diagram of your network and applications you feed it into them and then they'll generate a threat model I haven't acted done a lot with it but it seems like a good idea cuz with these other tools you sort of have to create your own diagram just for the tool which is pretty horrible you should be able to get some

from your CMDB or some other tool should be able to import information into threat modeling but I haven't seen that yet so those are quick comparison the tools so this is going to be hard to see I'm sorry so I'm illustrating some things that I did with threat modeling this was for Center for Medicare Medicaid and so I picked the scope as a single transaction going from an authentication service back to a key-value database request response I put one machine trust boundary in there this was an AWS so this prints out a list of threats associated just with this this would be connected with or in this case I'd be collaborating with a specific development team for

implementing a use case around dedication so I picked a really simple scope here's another one in this case this was also a CMS system but I took out a lot of the stuff for the details here I was using the tool Thor third party risk modeling this particular application had many partners and you can see in each one of these end-users so by running a model like this you get a list of threats for each one of these interactions with the different third parties and everybody knows how important third-party risk is I'm not showing the results of this but when you run this you get a list you kind of get a checklist here are all the threats you

sit down with the systems engineers you say did you address this did you address this if you say if the answer is yes then you check that off so it raises issues in an organized hopefully not not confrontational or not too confrontational way and again this is not going to be possible to see in the back but this is the output of the threat modeling tool so here it comes out with the Microsoft threat modeling tool here it says are you doing enough auditing are you collecting logs for this particular process and the other fun thing about it is additional fields in there I did some customization how does this threat relate to 853 or how

does it relate to cwe can you put in a user story I was working with developers so I wanted to put in a security user story so you can add in a lot of custom features into this tool and that's true of the other ones I've looked at too and here's another one that's going to be not recognizable this is threat modeler the commercial tool I talked about the enterprise tool that enables multiple users the good thing is it has a ton of components that you can just click and drag into your model so if you want to use as your traffic into your model it has built-in threats and remediations so these guys go a lot

further than Microsoft in terms of built-in threats and remediations and they had this probably four or five hundred different things in their toolkit so you're basically paying them for their knowledge base and this is another one you can't see so the output of threat modeler so you create the diagram and then it will output a threat and it says this is where this threat is coming from so then it enables you to go back you can't just deal with the whole system at once so it'll tell you where the threats are where they originate from which is important this is Arius risk I won't spend too much time on this they go through another interview

process with developers they have a rules engine and then they output different kinds of threats so one of the big kind of missing links is where do you get a baseline of threats because your business is going to be different now you just if you use any of these tools you just can't use them out of the box right that's not due diligence you just you know I'm gonna buy this tool start running it you have to make sure that it's incorporating the threats that you have for your business so where do you get a library of threats that's one of the challenges so you can start with kind of a baseline that maybe comes with

the tool and the big plus of this is once you create that library you can use it for other projects in your company you're not reinventing the wheel all the time you have to keep it up to date so libraries that I've seen or Capek it's almost got too many attack patterns like 517 attack patterns can you manage that Microsoft the threat modeling tool I think I just counted last night a hundred and seventy seven threats built into that and I'm excited because they were eighteen months ago there are only 41 threats so they're really keeping it up to date with as your threats threat modeler the commercial tool is for five six hundred threats built into that tool

the latest one was high trust has anybody seen the high trust threat catalog yes a couple of people have seen it so I just got an email about that I not being being an academic I downloaded I looked at it it seems pretty interesting it seems like it has about a hundred and fifty threats so it's not too many not too few pretty well-documented the only bad news is you can't use it unless you subscribe to high trust I guess right so I don't know how much that is so but it's nice to look at so if you could work out a way to use that that might be good so I like that the

other ones I put in need of work your Google cloud platform just a curiosity one two three a few verses about AWS or how about a sure I think Azure and AWS are kind of tied and GCP is kind of behind so I didn't see a lot of threat modeling work on GCP I used it a little bit they were using it at Vanderbilt Medical I sure I put in need of work on the other hand it's in need of work in the sense that they don't have a lot of remediations in the Microsoft threat modeling tool they do they have been doing a really good job on putting more threats into the into the modeling tool

which I'm impressed with so some application development use cases this is some other work I did for CMS it's kind of related but while I was at CMS Trump said everybody has to follow the NIST cybersecurity framework in the federal government right they already have to follow 800-53 now they all have to follow the cybersecurity framework so I was kind of excited about that I was working with these application developers so I said how can we use the cybersecurity framework in the application development lifecycle so this is kind of what I came up with I said you know got identified protect detect respond recover we're going to identify vulnerabilities we're gonna protect the codebase from those we're gonna detect

vulnerabilities that get through respond to issues and then recover so that was my idea and then threat modeling I put in the identify it's not going to identify coding errors but it can identify architectural errors you know the big the big picture of things that can cause huge failure so I put it over there then down in the bottom I put this relative mitigation cost this is if you look at software development you know cost studies they'll say that if you fix a bug in the you know at the developer desktop it may cost $1 if you fix it when in production it's a hundred and we I feel we don't put enough emphasis on

that in the security field because our bugs are the same kind of bugs as functional bugs right so this should really Drive this curve should really Drive more emphasis on getting stuff right before it even goes further down the pipeline and this is also too hard to read but this was just you could you can look at it in the slides afterwards this is how to incorporate threat modeling in an agile devops process so what are some of the opportunities so move the security conversation to the front end and it's I haven't had a hundred percent success getting developers engaged but I think it's the the goal not to make them security professionals but use these kind of

tools to have more conversations at the beginning enable a smarter risk analysis so if you have the tools built out then you can apply them to the next project without reinventing the wheel what else find new threats to code architecture and reusability I already mentioned that reusability of analysis and then this is another thing that Ron Ross emphasizes that NIST take the system's point of view take a systems point of view so if you use the model that I built with all the third parties in there then you can in that context you can definitely take a systems point of view so I hope that these tools enable people to translate between DevOps business and security and come up

with a more resilient system in the beginning so I started out talking about change I think the field is changing it's just unbelievable how much it's changing I think the opportunities are for us to help create that change and if it you know it's gonna mean collaborating more with the business for sure taking new taking new courses on some online courses getting new degrees the whole the whole field is just incredibly exciting these days with all the new opportunities so let's all create change I have a couple of references Adams Shostak is the Guru in threat modeling he kind of really popularized it at Microsoft and he gave a very good presentation at the last

blackhat I wasn't there but it's online I definitely recommend that and then there's also an OU wasp threat modeling slack Channel I think it has like 500 people in it now so this field could be growing and it might be good to just connect with people in that slack Channel and you can reach me if you have more questions or don't get the slides or have any other thoughts or comments and that's about it anybody have any any comments they want to make on anything I said or

yes sir yeah that is what I have done and that's not the that's like at a low maturity well that's maturity level one and that's about as far as I've gotten it should be that you're kind of sitting around the you know you're part of the sprint and working together and one of the things I've found that's difficult is like if I was using when I was using the threat modeling tool the Microsoft threat modeling tool it deals with data flow it wants to know data is flowing from this node to this node and it was it was difficult to have that I found it hard to have that conversation with the developers because they weren't thinking

about that they were thinking about api's they're connecting you know only after a lot of the project development work had been done then they would document it so that was the challenge I faced yes this is the way I did it as I gleaned I had to wait until they'd kind of documented a lot of the project and then run the threat model and that's not the right way to do it so there's still more work so we have several programming courses in our curriculum I'm hoping we can you know get more conversation going with developers and figure out what information can they give us that we can put into that threat modeling tool or

the other one so it's it's an open area for work in my opinion three minutes left yes yes sir open source libraries it makes me really nervous I'm sort of a traditional engineer we you know you you know what's in your system they were doing this at CMS they had so many open source libraries coming into these systems and they you know they used black duck I think to look for vulnerabilities so it's an area that makes me nervous and I'm not sure who's solved the problem how do we do we have the assurance that these libraries are going to not work today but then they're gonna keep on working and some of these you know

that people do things like looking at how many developers are there like I wouldn't use sea sponge there's only four developers they haven't made any recent commits but so I think it's an area that more work is needed I don't have good answers for that that's a good question any other questions I got one minute so anyway thanks for having me and I look forward to hearing all the other talks and you know if you have comments the things that we shouldn't put in our masters degree program or want to be part of it in any way shape or form let me know so thanks very much [Applause]