BSidesNcl 2021 Piacere - DevSecOps automated Radoslaw Piliszek

okay so i've been told to start oh my god can you can you mute yourself oh my please make it yourself because i can hear myself um so okay i've seen both muted now okay okay i hope you can hear me now and i don't hear myself so welcome everyone for today's talk uh presented by me uh it was prepared by me and my colleague pavel but probably will not be joining us today i will be available on slack still even after this presentation so if you have any questions that we don't answer during this session then i will be still present and available for discussion on slack uh and later we can always get in touch via

mail and for now i'll start so today i want to talk to you about pleasure actually uh because pietera is the italian word for pleasure uh it's actually an acronym for this project's name i'll show it later a bit but it revolves around this concept of devsecops and automation in devsecops so things like cacd in the cops okay and the company that i represent so this is a european union project and the company that i represent is called sevenbulls.com actually so uh the company's name is actually the the dns name that you can find us on they're very useful i would say um we act here as a research and development center we are actually

registered as such in france and poland so our our main country is poland and we are also registered in france you can now see how research and development center is is named in france and in polish in french polish um apart from r d we mo we are mostly an i.t company so like uh the majority of our employer employees are iot specialists uh we are doing it4it by 80 basically um and in various branches of it so like the the consulting software engineering uh and various others uh you can find all the details also in english on our main page symbols.com if you are especially interested in in rnd stuff then it's slash rnd and you'll find it

there um and now a little bit about devsecops because obviously it's it's some other uh funky name that the community is using for the technologies that are being prepared nowadays for coordinating i.t infrastructure companies and this is the definition it's taken from the gardener's i.t glossary so the go-to point for for the information about the the different names that are being used nowadays and you've most likely recognized the dev and ops part of this so the devops as it's been the most much more popular name for the concept of making operations in it uh more automated using the development tools using the developmental approach that is used for for building software also for building the infrastructure infrastructure itself

so the concepts like infrastructure esco etc and now this is actually enhancing it with security in mind the security is is placed uh in center on purpose so that the this says that the security is the most important part of this deaf psychopath dev set ops approach so the real focus of this new trend um and the goal of of of death develops is to naturally not impede the progress that is made by moving from the ordinary to link to devops tooling so that the agility that is gained is not lost in the process of of doing um the security stuff additionally to to doing devops so that the devsecops is assisted as devops but includes security as a main

point as a main focus that's what the definition says and um i hope i don't have to persuade you why security is important so like almost every day if we go to look at some hacking happening around the world we always see there is something to improve in the security that we have in itp globally speaking um but if we consider that moving from the traditional roles were traditional models were there was a lot of nitpicking on the configuration and a lot of manual process and design um in this new devops world there is much more automation and this automation can introduce other surfaces of risk other uh waste for our it infrastructure to be insecure

as a result uh so we argue that it is still important to focus on security or or even more important to focus on security in this new devops world by introducing this deaf cyclops approach um and um the other parts that that are arose from devops mostly so the heterogeneity and the the fact that this it's happening frequently so so it's more often uh the case that changes are changing change is the only constant basically that's the saying um and introducing security into that constant sounds like a good promise and uh here we are introducing piaceres so so i have said previously it's an italian word for pleasure but the acronym of the project is actually

programming transfer of infrastructure as code in a secure framework uh i always hope that nobody walks wakes me up at 2 a.m and asks me what bachelor stands for because i will not be able to tell that basically this is too complicated but focusing on the fact that this is about the infrastructure's code and about doing it in a secure way by offering a framework to do that basically what this project is is all about uh as the slide says it's a horizon 2020 project uh so it's by [Music] it's financed by the european union the conservation that we have consists of 12 organizations they are from different [Music] they have different experiences so we

have academia business government like for example from from the business we have large enterprises like hpe and ericsson uh from academia we've got polytechnical milano from italy and the project itself is led by a group from technolia from spain uh they are in similar role to us so like a research and development center there and the schedule of the project it's actually in an early stage so it hasn't not even a year has already passed so i'm showing you really early designs today and every approaches and i'm very open to the discussion in general uh so that's what you are going to what you can expect for today we have seven bulls in this project we

are mostly responsible for the integration of the the final solution and the canary sandbox environment i will also explain what it means later uh on the slides um and what about the goals of this project so picture dev sec ops the project is called picture and the circus is what it is meant to achieve in that so basically what the devops uh allows you to do so developing building deploying application that's that's what you are provisioning it infrastructure for basically so uh it's still must to do that additionally to that um you're always speaking about this avoid being called vendor lock-in so going for hybrid and multi-cloud deployments with cloud agnosticism uh that you don't have to go and choose

only one cloud provider which also brings other benefits later like for example this bullet on the optimization of usage of resources since you can easily swap out the environments for the other clouds if other clouds are more optimal optimal for for your solution at the moment uh of course including the costs of transit so like if the transit is too costly then this solution wouldn't be proposed as an optimal one um additionally testing because testing is actually paramount for the security so basically how we uh how we achieve the security is that we propose testing and the testing is is customized for the user uh but there are always general principles that this framework is going

to apply uh to the process of their circums and another paramount thing to avoid the slow flakes so um like before the devops era we had this trend that every server was a pet uh called otherwise a snowflake that it was crafted to work the way it worked and this philosophy kind of unfiltered the devops approach so that the devops is mostly as as it can be seen in various companies it's it's not clear devops we still see uh snowflakes happening so this this project is also aiming to ensure that the users are more likely to focus on the devops as it was meant to be rather than devops revolving around the snowflakes but obviously uh

we cannot block that um okay so regarding key features uh as i've already mentioned i guess most of those points so integrating the security principles in the devops agility without losing the security or otherwise security without losing their agility uh the the sandboxings of the canary sunrocks environment that was already mentioned so an easy way for you to set up an environment that behaves similarly to the cloud provider so that you can test it you can test your infrastructures code in a similar environment to test some dynamic properties and also regarding those avoidance of snowflakes etc conflict drifts single source of truth so if you if you recall the concept of key tops i will show that

on the screen as well uh cloud diagnosticism said that automatic healing optimization so if we if we consider optimization some optimization is is like for the cause for the traders but uh a more general concept of healing so when the when the toolkit is actually able to discover that there was a drift down an unplanned drift that uh you could say in general should be optima optimized away but this is a quick action like uh restoring your your your service to work in the same in the very same environment actually uh that is that is what we call safe healing automatic healing um and here is a little uh diagram for you how this picture devsecops framework is

thought to work we've got a part for design time this is in the left upper corner of the diagram i don't know if your screen is able to display it in zoom enough uh to be readable but i will try to explain what's in there anyway so at the same time the tooling that we are providing uh in the framework uh is plugins for eclipse ide this is actually uh we are still on a stage of researching whether the eclipse the classical eclipse that you know in java that is desktop-based will be better or if the new studio code eclipse share will be better solution for this particular project also because there is there is quite an

abundance for for tooling for of tooling for modeling in the eclipse traditional that we are still considering trade-offs that we have uh in this particular choice but uh basically there will be some ide either web-based or desktop-based that will be able to manipulate the new language that we are going to propose in this project for modeling the iot infrastructure the generalization of infrastructure scope so to speak uh this language is called dommel there will be a screen explaining it in more detail soon now just focus that the dom is the language that that is going to be used for modeling uh in this ide and then this dome model that the user prepares for themselves

is going to go into the infrastructure code generator and parallel to the verification tool and the infrastructure code generator is actually to reformulate this demo model into infrastructures code that is generally acceptable like for example the terraform or the ansible playbooks etc um and and the verification tool also gets fed that so then the verification tool is able to read what the user had in mind in the demo model and we will see the the the complexity the extensivity let's say of the download soon and also the generated infrastructures go and do all the necessary static analyzers uh on that uh then with that it moves further uh if it's accepted so if either there were no

issues or the issues that they were discovered uh user is aware of and accepts then this moves on to the git repository so some tooling gets a pr this pr uh can be approved then by the the person that is allowed in this git repository doing something like github or gitlab or garrett perhaps um some some so owner of the application of the deployment is able to decide that it goes in and as it goes in there's the runtime controller that watches that uh gets notified and retriggers the deployment the deployment is actually handled by the thing that we call it executor manager as it manages the executors of iac like for example this ansible or this

airform and the iem then moves some information to the infrastructure elements catalog this catalog is also usable by the ide for the discovery of the various infrastructure elements that are available in their history and the iac executive manager executes against some resource providers so thinking like amazon thinking like asia thinking like something local some other public openstack installation um things like that also on premises if if just a bunch of virtual machines or or similar stuff like iot and again this is like a resource provider to this project uh then we also have on the right this infrastructure advisor which includes things like monitoring the optimization part of the project this actually analyzes the deployment

and also saves some historic information to the infrastructure elements catalog database uh on the left with all we can also see the canon sandbox tooling that is responsible for spawning the canary sandbox resource provider so like if the user is not able to provide any other resource provider uh to test it locally and has no experience in deploying projects like kubernetes or openstack this is a little helpful project that is going to help with that exactly okay so i will move forward and that and uh i think uh you've probably seen this image from xkcd it's quite popular at least here um so i think i must still answer this question why why are we doing this uh

basically because it's introducing another standard uh but we believe that uh also as the people depicted here and that that it somehow covers the um the needs of the use cases better than than the solutions that we have nowadays and our result is the following so uh this is a simplification of devsecops approach so the the devops is is quite well understood but def sec ops not really perhaps uh unification of this way so like uh the devops most of the time is all we still seen as something targeting a single environment rather than being natively uh designed to deploy to hybrid heterogeneous multi-cloud environments um especially also suggesting that the the use of multiple

iac languages is is probably more uh applicable in general solutions other than trying to stick a particular one single language that is going to target everything at least with the solution that we have at the moment and an optimization of course and enforcing this this git tops approach that uh while you you with devops have moved away from the the manual process of approving changes to the configuration as in some sense the sysadmin when when doing the configuration changes was approving them at least by by themselves on the machines uh now this this approach uh is going to be centralized in in in some git repositories so that it's observable and can be compared to this

the actual state can be always compared to the desired state of the environment um and now for those key elements that were shown somehow on this diagram that's what i was showing previously so this is the dom the the language for the modeling it's actually def sec ops modeling language but the diagram only uses the devops modeling language uh verification tool uh that's what was doing that static verification then we have this central dom and iac repositories so this was this git repository turing actually it it covers the dom and iac parts so that was what was going to live in there the runtime controller that triggers triggers the actions around direct executor manager the canary etc

etc it will be described on the following slides so don't worry if you haven't read the slide looking at the time dom so dom devsecops modeling language uh the goal of the modeling uh is to cover all the use cases at least those that we are aware of and the goal is to be cloud agnostic able in that so what we mean by being able to be cloud agnostic so that it isn't still enforced on the on on the users so if the users are for example very focused on very specific services of one cloud provider and want to stay uh on that service provider and use those very specific features then this language is allowing them to do

that basically uh use all those specific features that may not be available somewhere else so that the cloud agnosticism is is somehow optional but still deserved in this environment and this is achieved in a very flexible way let's say with having modeling in multiple layers that can be presented independently as well um so if we think about the the modeling of the application of the infrastructure mostly in here we may mean components connections different security aspects properties of that uh what are the non-functional requirements the technological requirements uh of the of the desired infrastructure and we can always apply this at an abstract level that is environment agnostic so if you are starting from this level you are allowing yourself to

go full agnostic to your environment so like for example you say you have your applications that they require this amount of ram this amount of cpu time etc the this amount of of some person's storage or perhaps the storage doesn't have to be persistent like it it can always be ephemeral solid because this is uh somehow not persistent for this application whatever uh then at this level uh you are allowing to just specify that and then uh you can say that this model can be satisfied for example by the vms that you have registered in your catalog of infrastructural elements so like you you have those statics perhaps not even virtual perhaps you just have physical machines there and uh

you have them registered and this can be fulfilled by by them then that's fine and the model is applicable uh then again if you if you have a cloud provider then if it's possible to find the offer in the infrastructure elements catalog that is sufficient to satisfy this those requirements that were given uh you are basically going to have this this smaller concrete for you and then again you've got this level of concrete uh modeling where you say that i know this is my abstract model but uh i want it to to be like this i want it to be satisfied this way like uh at the moment i really want this part to use my on-premise virtual machines

but this part i don't really care about this can go to the cloud uh or perhaps the other way around right and this is what the concrete level of of modeling allows uh there are also further levels like for the um for modeling the deployment of the actual application because the early levels actually concern themselves only with the infrastructure uh layer and the requirements of that but the further levels can can satisfy the application as well um and uh from this donald donald language uh we are able to uh generate some ayac that can be run by the the tools that are already available in the market uh and the modeling as i was saying will

be in some kind of eclipse ide i can promise if it will be in eclipse classic the java based desktop one or the eclipse ship based on visual studio code so we'll see about that and on to the verification tool the verification tool is for that static analysis and it can see about the domal so the different layers of the domo that were described by the user or by the satisfier and the generated iac uh from that and based on that run any possible analyzes analysis that can be run statically like for example um imagining if you have this model inansible and this ansible actually uses uh some collections so you you can check if the collections are

affected by some security vulnerability similarly if it's modeling an application [Music] uh to some degree it can run the the checks for the vulnerabilities in dependencies and things like that i'm among other things of course uh it also verifies the aspects of correctness so in the demo you can also model uh you you model it to to the level of detail that you can use this verifier tool to actually verify the connections that were expected to exist uh in the in the moderate and generated infrastructures code are going to be at least tried to be created in this way and for the dynamics we'll see another solution later on latest slide okay so so this

central repository actually this git tooling and the running material is what throws everything around there um and what's it for it's for achieving the single flow that whatever we are doing uh we are avoiding the configuration drift by going from the always going from the same central source of information the single source of truth basically uh that whenever you need whenever the tooling needs to check whether the the infrastructure has drifted away from the desired state it's in there already like the github suggests uh how it should be done uh this also simplifies the the management for for companies and symbolize this access control because yeah all the access control to what is actually going to happen what is

actually going to change in the environment uh is via this one single well one single obviously this sounds like a single point of failure but basically if we consider that the git repositories can be easily made highly available in such scenarios then this is no longer a single part of failure um but otherwise from the management perspective uh from the security perspective this is a single source of truth the single uh target for for where to look what this what the desired state should be and the branding controller is i would say something basic considering the uh or the other tooling this is actually based on ppmn so business process model and notation uh it also an extensive vernacular of

the operations that are going to be executed in this flow uh of the operations but other than that it's it's pretty straightforward and that it it orchestrates the the flow of the framework actually only that um and for the iac executor manager this is for the execution of iac and the infrastructure code um the goal of that is to understand actually what's happening like for example if the terraform is creating the resources the resources that the terraform creates they're normally uh tracked uh like in the same directory that is that where terraform was run uh with all the metadata that the terraform has saved there about the resources it created the goal of this and similarly

for ansible ansible most of the time people don't even bother uh saving the information about what was deployed where with ansible it just happened uh something happened and and we are fine with that but the goal of this tooling is so that for for this crafted um ansible call for this crafted terraform code from the icg from the code generator uh this tooling is going to be able to understand what was deployed and save this information to the catalog so that the catalog can be used for tracking the various events the various metrics that are happening advanced events that are happening the various metrics that describe the state of the uh of the deployment of the deployed

infrastructure elements thinking like vms that the storage that was provisioned etc um and it also has the support for configuration and reconfiguration sorry and and scaling uh so this is actually paramount to be able to make modifications to your code and not trash everything around again uh this some responsibilities obviously still on the code generator on the on the actual transformation of the demo of the input demo model uh but on the other hand the iq security manager thanks to saving this information what's already in there uh can do a comparison uh what's going to happen as well and uh this automatic deployment to chosen canary environment if it's configured uh then the process can include this

automatically it could also be a manual process like from the github's perspective uh that you could have the branch for what's on the staging sandbox or canva or whatever call it and that would go to that specific uh to that specific environment or else the flow could be [Music] programmed such that it is always deployed before production so that if one just goes on on a single branch then uh at least it is verified in some environment that the infrastructure scope is able to be run and the tests if there are dynamic tests of the environment if they are provided they are run in such an environment to verify whether the the solution that is

going to be deployed is actually not going to kill your production uh uh at this point uh and it also um has to be concerned with the secure access security use of credentials to those target environments and now onto the economy sandbox environment tooling so the tooling that is going to deploy this kind of environment can i resource provide actually there are two main tools in that the one is the provisioner so the provisioner that's what it says so it provisions uh the deployments of selected environments like the openstack the kubernetes in opinion united way so that the there aren't many knobs for the users to mishandle let's say and the simple instructions how to get yourself

a ready cloud for yourself at the level of the openstack so the infrastructure a service basically and or the kubernetes uh deployment when you are concerned mostly about the the containers already so yeah about the container orchestration that and the other solution is smallclaw this actually was born in this project this idea that there is a need for testing the apis of the cloud providers for basically not the apis but the infrastructure's code that is generated to be operating on those apis and uh whether the state that we at least expect from the uh from the cloud providers to how how they should behave whether this infrastructure scope is actually trying to achieve their desired state

so so this is smoking cap the club providers like for example mocking up the the amazon [Music] ic2 apis that spawn the vms on amazon web servers and this is this is as i was mentioning this is to be able to test dynamically uh the aspects of deployment so like not everything can be tested statically as we know so this is a way to those do those let's say late tests in a known environment so that you can verify whether the infrastructure code is going to deploy what you desired to be deployed and also including those tests they can also be run in such a deployment either automatically in that flow or just using another branch for that environment

um uh infrastructure ims catalog so this is like a database of of what's there what's deployed and what's the history of that what's available also uh so this is the central storage uh the very brain of of the tooling because it's used by so many components uh also the optimizer that we'll see soon uh yeah it doesn't stop there it doesn't stop at the level of of showing what the offers are it actually tracks the the created ones and and their state like for example we can understand that uh this specific zone uh of amazon was having issues with with the bandwidth for our application they will latencies uh beyond what we uh deem appropriate for

our application so that the optimizer will say no no we are not going to deploy there because we know it's bad for our application uh that's what it is covering in here and finally i hope i'm in time still and finally uh the infrastructure advisor so that's a cool name for this component it's actually made of many smaller components i'm sparing you the the very details of that focusing on two um most well understood let's say the random monitoring which does monitoring for for the performance and security characteristics of the deployment that are operating already and the iop the infrastructure optimization platform that is basically the optimizer uh the tooling that is able to look at the deployment and say

well you could do that thing better uh due to various reasons yeah um this is so the monetary punk is is the part that is collecting the metrics the events and sending it to the catalog actually where they are stored and then they can be retrieved by the various other tooling

and it's deployed alongside the infrastructure when the well at least parts of it are deployed uh along the infrastructure so the agents for the monitoring are deployed during the iem the inspector the iac execution manager uh run uh self-learning self-healing are also included in the stack of infrastructure advisors so so for for learning the the characteristics uh using machine learning algorithms and for self-healing in in cases where it's obvious what should be done to repair the environment to quickly improve it so that it it is sufficiently it is working sufficiently good that it can progress further that's what self-healing is for the iop the optimization yes based on the collected metrics so that that's your optimizer

um yeah it integrates with the catalog i've said that and the optimization if you consider the optimization finally uh this is actually balancing the trade off so like you you can't have the cake and they did uh you have to decide uh what's most important to you you're balancing the cost performance availability and other characteristics and based on the weights that you have uh the optimizer will be able to read the the historic information uh and new preferences and what's defined in the demo as well to uh give you suggestions on how it's going to be improved and if you remember the diagram there was a little arrow from the iop to the git repository again

so that the the the advisor the name comes from that the advisor is actually able to advise you uh what should be improved in the demo in the tag yeah and and concerning the self-learning this is machine learning also in the optimizer you could say that some of those algorithms are less classic ones one one could say so much training as well um okay i think this should be last slide yeah uh so thank you much so thank you very much for today for for hearing me out uh you can stay in touch with us through the project this is the the website the address of the website of the project samuelscom is the address of my company

if you want to reach out to me i'm still available on slack and i will be reachable by mail if you can be shared if you are interested so thank you