The Secure Software Supply Chain Function S3C

foreign

er um I started my career started out as a support engineer then I became a consultant and now I'm this devsecops creature I'm saying that because there's a cops actually has a lot of different it's an inter interdisciplinarist field which has a lot of roles in itself such as engineering but also architecture a bit of operations development as the name implies and yeah I finished the implementing this cops Linux course which was great but the the funny thing is that it's so this this field of security is so Dynamic that even it contained some stuff that were already obsolete uh I know bash python I have experience so going through writing an application going through the change change

management process at IBM and Publishing it to production environment and um yeah that's how I know a little bit about the from the developer side of things um I have varied experience in maybe a unique blend of different organizations uh private uh government non-government and really there only I would say two types of organizations so there are executors and there are innovators the difference being in the culture um but we're going to mention that a little bit later in during the presentation something about me I have a wife and a daughter who are in this Hall with us uh for the record um okay so the topics you can read them faster than I can say them but basically

we are going to start off really slow and really basic but then we're going to advance uh very fast and deep dive into this whole area um so a bit of context um please raise your hand if you think that an application is a product here we have one two okay if you if you think yeah if you think if it's a service

someone thinks that it could be both yeah okay okay that's great yeah so an application can be many things it can even uh so it um and when we say a product it can even be a physical product uh but that's getting uh very complicated because then we need to consider some safety uh issues ETC so we're not going to go there we're just going to focus on the software for this presentation but I would like to start actually to go back one step and start from the very beginning and that's actually an application is an agreement because an application does not usually exist in a vacuum we are sure the Creator can use their own application

never share them but um this is not what we're speaking about we're speaking about even if I use an application which is [Music] um which is open source if it's free there is still this license which is provided as this without warranty of any kind so it's an agreement always uh but yeah we're going to go for uh we're going to examine the open source applications a bit later in this in this talk um on the other side there's the commercial uh applications which have different business models and we need to be aware of those uh because uh like if if we're tasked with protecting them um so yeah uh traditionally in uh transact one-time transaction uh now

it's the more modern approaches pay as you use it or as we also know it as as a service and then there is a subscription but really maybe what most people would expect from a presentation about this is the technical side of things an application is an interface it's a bridge between the two sides the producer and the consumer um so what does securing applications mean um basically if if I'm predict so if I work at the security operations center then I would protect the organization I would protect the assets uh the servers the workstations the bring your own devices the mobile phones is all that if I'm a consultant I would protect the the customers of my organization uh

obviously I I could protect the organization itself as well but I'm not expected to do it and I'm not getting paid that I'm uh like I'm protecting the customers but if I if I am protecting an application I'm actually protecting both sides because remember it's a bridge so there's no other way to do it so how are applications built um well we could imagine some technical genius that created their own operating system their own language programming language that is and they write some complex complex applications but the issue with people is that we do not scale so we need to work together to produce something uh really massive and at large scale and reliable so we share we share

a lot of the code that we that we a lot of the logic that we found by trial and error and there are a lot of teams like front and back and QA Etc that are all working together towards this um so yeah and obviously there is the economy side of things where um we are um yeah obviously the the organization which we are predicting cannot operate for a long time without any Revenue so there are usually some deadlines not everyone is Blizzard where they say well it's going to be done when it's ready most most of the organizations have some sort of uh deadline so yeah we we share uh the logic that we found uh because because

of the economy but also because um obviously for the reliability The Proven results and hopefully for a security so why why supply chain has an attack Vector well because um you could Target one organization and impact uh a lot more even you could impact like it's a bit chaotic because you never know who you end up impacting because it it do not you cannot possibly know all the clients uh victim supplier could have um so sonotype the the software and security vendor actually came up with two three phases of the supply chain attacks the first one is they call the zero day which is basically to find some unknown vulnerability which uh even the the supplier this is not aware of and

there's good reason for that actually in the black market that these types of vulnerabilities can be traded for a huge uh prices like thousands maybe thousands of dollars or tens of thousands of dollars or whatever um and we all know love for sure uh we all experience that and the the sad news is that even one year or more later uh some some people are still downloading the vulnerable uh versions of it um so we have according to sonotype we have the second phase which is the actual supply chain um so um malicious actors quickly realized that well there are some very popular packages so if we actually compromise one of them we can reach millions of

developers and thousands of organizations so um yeah that's pretty bad sounds pretty bad then there's this third phase which is the corporate developer and um these phases are actually they I don't think any of them has finished they're just like they can some type uh describe them in this way and then there's the corporate developer which who so in this case there are dependency confusion attacks which saw an explosion of uh hundreds of so each year they grow exponentially so how can we protect the applications then there are different approaches to this question one is often centered on the devops like you just inject different security measures into the infinity infinite infinite Loop there is the secure software development

lifecycle which is more of a management approach to this and it really focuses on the whole application life cycle we have also the salsa it's pronounced also even though the acronym is not really reflecting the the whole frame the name of the framework but it's separate on the build um so the levels of for artifacts is a great great framework and if you go to their website they have very good explanations and great examples um but the issue there is one issue though um and that is that the using compromise dependencies is mentioned only in the fifth step so in their point of view the source like source code is only the code that the we our developers write and

there that's too late and that's that's the wrong way to think about it and it's too late as the fifth step um as uh for for two reasons uh one of them is that um if you're a developer you would it would be pretty obvious to you because uh you're writing your code uh so you're building upon some uh shared library and uh then you put so you spend some hours doing the logic and then you push it and because that's where usually the scanning takes place it's at the source uh control versioning system um so it's only then that you realize that well there's some issue with this Library so now you have to go back and

rework uh it may be that you need to change the the dependency and well there's the risk that the newer version is not backwards compatible with the previous one um that you were using etc etc it's so but maybe the not so obvious uh problem with this is that um it you could be violating a license without even realizing it because you never checked and this would result in the revocation of this lessons so um you know a whole organization could spend a lot of resources on developing their product and in the end uh they cannot they need to re-architect it and rework it because they relied on some uh like in this case it was nutanix it

relied on Min IO probably in this case they they knew that they were breaching it but still it's a it's a powerful example of uh the need the risk of licenses um all this brings us to um to trying to figure out okay so I'm devsecops engineer or yeah practitioner um so what do I need to do like how how can I explain these things uh in uh in a better way in a more understandable way because actually yeah you have the devops Centric one the management Centric one and the more like technical focused one like from the approaches so we have so this is what I came up with it's um actually very simple and that's the whole point behind

it it's a function so every developer is aware of what the function is and this is actually the easiest way to explain uh what devsecops is um so you have the input you have the research and development and you have the output and this is not necessarily a technical uh way like model mental model to see things because you can because in the input we have also the customer requirements or the product management requirements and we also have the reference architecture so it's really it's I'm trying to to make it more realistic the other stuff we're probably aware of it of most of it um so this is the so this is my idea and in the next

slides I'm going to share a bit of experience with actually implementing this so we have the input okay uh licenses again so we need to understand what the licenses there are what types of them some some are more permissive which is great especially uh Linux foundation and Cloud native Computing Foundation licenses which are usually Apache too as the graphic shows but increasingly businesses are using a dual model of license sync which means that they have an open source offer which is sort of to hook you up and to but also to like just rank in um developing together and they have a paid version which is managed for those users who are maybe don't have the

time or resources to to deal with the free version and GitHub has a pretty good assistant with that with the permissions limitations and conditions um interestingly a lot of people are using Docker desktop but many uh but maybe not so many remember that they actually changed their licenses so even though Docker engine is free to use and it can be like used in the build server at the actual Docker desktop with the graphq user interface is not free and so yeah this is something that we should be aware of uh okay so we chose the software that we want to use and actually um this is very important to have great communication with your developers

because um if they somehow let you notify you about the software that they're planning to use then you can check it in advance and save them a lot of save them some of the risk so we chose our software and then we start um and then we are then then we downloaded it and we verified the hash of the like the checksum of the file for the Integrity but um that only tells us that the software is um the the one that the Creator um was um like we we have some assurance that this is a software that we want but it doesn't tell us much about much much else so a great example with the in

during log per share is that actually the the login library was borrowed in almost 800 other projects and uh and seen in over 90 000 individual components so in order to understand [Music] um like what we are what we have what we are protecting we really need something different like not just the checksum um and this is yeah uh so that's why we have the software bill of materials actually uh it the practice of software composition analysis was uh something that existed for years but only after only uh because of uh look for Shell um a lot of people started paying attention including regulators and so they came up with this name um so there are three types three main

times to to create a software materials one is to just scan the static Source uh using a tool such as Swift which is very famous um but that wouldn't really that wouldn't necessarily tell you which um which libraries are actually used during runtime so that so there's also this great tool called j-bomb by contrast security um that can do just that but it's only for Java but probably they're going to be more such tools in the future okay so we have uh so we have the software so we know what is inside the the dependency that we rely on uh but what do we do with this um well fortunately there's a great open source project called the OAS so open

web application security Foundation dependency track uh which is acts as a central repository of uh such abuse of materials uh it's awesome it it does it so it is it has a different logic to it you don't scan the uh the view of materials during the build time for vulnerabilities and fail the build if there are if critical vulnerabilities are found uh usually but actually what it does is that it checks it continues and it continuously connects to the usual vulnerability databases to find um to match the non-vulnerabilities to the used components so you get up-to-date information about it uh the bad thing about is that it does not export reports out of the box and

you cannot filter based on vulnerabilities in the UI which is something that is strange that's missing but hey it's uh open free um so we have created the inventory we have scanned it for like for non-vulnerabilities now it's time to Aggregate and prioritize the vulnerabilities and not always the it's the developers or the security Personnel who make that decision and finally we remediate them but if you consider the number of applications that we are using the number of depend multiply that by the number of dependencies there are and multiply that by the regular updates that each of those dependencies has it's just too much so we need to use some automated solution such as those which are visible

so the consumption framework um is the great maturity roadmap that was originally developed by Microsoft and it was donated to the opens software security Foundation uh some of the the first two levels we already mentioned um but what's uh also great about this is that we really begin to think about how um as I mentioned so application or shared logic shared code is some sort of a bridge or an interface between organizations so if we are using some external application then we really need to think about we're having a sort of a database of the relationships with the different businesses who the contact person is for that business uh in so that in case something happens we know

who to contact quickly um and then the the next two levels are like very Advanced and probably if you go there you have a very good reason and you understand what you're doing but yeah we're not going to go there um the openness of open source security Foundation also has a very good project called scorecard that has different criteria for weather a project whether application a public application is look how secure it is actually and it's funny because these are the same criteria that that people are going to judge our work as well so you can see the cycle um there uh so we go to so that was the input um and we have the research and development

we begin with the risk assessment this is sort of a behavior tree so you go from left to top to to right and back again so we can try to So based on the vulnerability that we found we may decide to fix it if that's not possible we try to mitigate if that's not possible can we accept it um or maybe put some uh if if we cannot accept it then the only thing that we could do is to transfer the responsibility to insurers um so we have also so to have a proper inventory of what we have what we are protecting we need to have reproducible bills it's very important because uh because uh a

checksum well not there was will not help as much if we do not [Music] um are not certain that of what we are building so um for example if you compile an application um just a moment after you compiled it then you would the checksum would be different because of the timestamps so this is the first thing that we do is that we just make it uh zero or in Linux Epoch time that would be 1970 because for historical reasons um and and the second thing is that we need to download specific versions of the of the software that we are using so it's not a good practice to just uh just not put anything or just put the latest

version of this software we need to specify the exact version and this is also good for reliability then we use open source EA we love open source uh there is actually last week um a new project was announced which is called chain loop which is great if you have if you are responsible for multiple different build Source control versioning systems and build servers because it works with all of them and it makes the work allegedly I cannot say because it's very early in development but that's the whole point is that it makes working between the developers and the security so the application security Personnel easier and it also integrates with dependency track which is great

um feedback loops so it's it's not enough to just use some kind of Technology it's very communication is very important and so we need to to somehow um integrate the technology and the processes and the communication uh so that there is a self-generating and self-sustaining Loops um so uh for for developers that would be self-service static analysis Dynamic analysis part of the build process um and then together maybe we review the statistics together with the product owners managers whatever we end up developer team leads we prioritize that's for first party proprietary code not for the open source and then we proceed with the mitigation and uh as the as time progresses and we um and people get into the habit of doing

this it's becoming self uh sustaining so during the research and development there is quality assurance but why not it's easy to actually work together with qas to uh to build the to integrate the security Assurance inside so here is an example of Postman uh which is a very famous for API [Music] um and this is the GitHub issue that is still open even though it's from 2021 uh about integrating it with open web applications uh security Foundation uh zet attack proxy and we love open source because not only because it's free but because there is a great collaboration between people and a lot of knowledge and experience being shared so this is a great example

Okay so uh so if we're to Del um So within the research and development so maybe someone's thinking okay if I've I've shifted left I've done all that they can I've uh scanning for vulnerabilities misconfigurations compliance violations um so I have prevented threats well not exactly so in this case organizational security um is wiser let's say it has more experience so we can borrow from there from the cyber security framework so um yeah we need to also detect and respond to threats it's not enough to to try to block everything because it's probably not possible as well really this is a very complicated uh because there are just so many pieces of this and they're all interacting with

each other um but just to give an example with uh so on the left column you can see the code container cluster and computer Cloud which is the four C's as they are known uh this is a and on the right side it's uh it's let's say my interpretation of what the different aspects of each of those is and so they're they're the same but they're just applies to each one of them so for storage for example uh obviously the the programming the program itself uh stores its data on the random access memory and maybe even in the in data in memory database the containers are storing in uh environmental variables as well as volume mounts the cluster is

storing data on the in config Maps secrets and um volume claims persistent volume claims and then the compute obviously has the databases and the storage drives um so this is like just on the one level and there are many levels and they're all interacting with each other so uh this is very broad topic but just wanted to give an example so we have so this is actually from Red Hat they donated their incident so product so their product security team donated their process to the first organization is called um and it's great I recommend everyone to check it and just include it here um so there's also a lot of innovation Happening Here with uh sandboxed

containers or container runtimes and actually if you if you have a really sensitive component of your application such as a database you can just for the note where the database is you can select the different runtime environment and just for that note and kubernetes is going to take care of that so we traditionally there is the intrusion detection prevention systems [Music] there is the web application firewalls of which again you can see I recommend the cloud native Computing Foundation project called query fence and then and then like um the more modern approach where this is all going it's a runtime application self-protection which is great unfortunately there is no open source project for that there used to be some

years ago I think five but it got abandoned so uh interestingly enough maybe some of you went to the workshop for contrast security assess which which has that included so the future is confidential so-called confidential Computing we have the homomorphic encryption which is uh which is basically magic because you could uh never have to decipher um data like it can always stay uh in ciphertext encrypted and you can still compute with it which is yeah uh crazy stuff and then there is the trusted execution environments which are basically part of the central processing units units which are safe they they cannot be breached so this would really make our work easier and protect our customers a lot

um so what are the takeaways um as I hope this illustrates it's um and many in the industry are now are now saying that shifting lift is not enough because we cannot block every thread sooner or later the practice matures such as uh just like security so Enterprise or information security matured so we need to shift smart or whatever like we we need to to cover the entire process [Music] and uh yeah that's it any questions

okay thank you very much foreign