BSidesMunich 2022 Main stage

so

[Applause]

hello perfect good morning everyone hope everyone is doing well and thank you so much for helping us and supporting us with this community for security so here i sneha your humble uh host for today welcome you to besides munich 2022 and i wish you all have a great day today it's going to be fantastic and we are going to make it super awesome we have some ground rules today and the very first ground rule is you have to wear mask all the time this is not just for your safety but it's for everybody else's safety as well so masks should be on because we are a community and our idea our responsibility is to make sure that

everybody in our community is safe and sound when we are talking about this community we actually derive this entire meaning and this actually goes with our principles at b sides besides security this is a group of community of engineers security researchers hackers individuals just like you all almost around 13 years ago these individual ingenious people they somehow couldn't make it to the mainstream conferences and they thought that no their voices needs to be heard and that's when they started a new community called besides security and today they are present all over the world and we are humbled to be a part of it besides has been a great part for us and i would definitely like to say

for every one of you please give a huge shout out and applause for besides munich today thank you so much besides munich for helping us share the ideas and knowledge what we have done so far sorry is i would like to give you an overview of the entire achievement for the year 2022 we are a small set of organizers we like to call ourselves as the besides munich team we have incredible 20 sponsors who are the backbone for the conference this year and especially thanks to their support we are all present here and we do have some great stuff lined up throughout the day apart from that sorry we had this year 57 cfp proposals call for papers that

was like super amazing 57 papers well there's one thing that i definitely need to tell you there were actually 58 papers but there was this one paper which was so super genius none of us like i'm telling you very frankly whoever is present in this room no one of us was qualified to judge that paper the title of the paper was laura ipsum that was fantastic and it was the huge paragraph so unfortunately we did not select that paper and hence we have 57 papers this year out of which we have 31 awesome speakers who are present here who are going to rock your world with the share of their knowledge and some amazing demonstrations and some super amazing

ideas and some of them are also going to share how they perform the research but they failed but we'll let you know what are the lessons they learn which is fantastic it's an amazing power packed knowledge sharing session the whole day that we have today yesterday we had four ingenious hands-on workshop which was again amazing and those who were present at the workshop i'm sure you had fun and you learned something new and if not new you at least had to polish at least got to polish your skills so today's agenda is pretty simple we start with the remarks i'll be here talking right now and after that we have the keynote the fantastic keynote and the morning

sessions wherein we have the morning talks then we have some coffee breaks then again the mid-afternoon sessions lunch break evening sessions closing keynote and there is one special session this year that we have it's called heartfelt goodbye and what exactly this goodbye would be we will let talk about that in the evening because talking about something that is going to end is always difficult but there's always this excitement that there's something new that is going to begin so please be here with us next so about us a quick overview so just just give me a raise of hands of how many of you are here for the very first time very first time for besides

munich wow that's a huge number how many of you all came to know about besides like a few weeks ago or this is your first besides ever doesn't matter which country wow that's a huge number that is amazing so let me just introduce quickly to what we are how we grew so in back in 2017 we started a small community conference wherein we had only 75 people to 78 people attending the conference we had 10 talks ingenious presentations and only three workshops and we grew and today we have over 380 registrations we have amazing presentations more than uh 57 proposals this is how we are growing and now we are also expanding how are we expanding

that's in the evening you will come to know before that before i leave this stage i do want to give a huge round of applause to the entire organizers team that is present here dagmar morton jen chris anna and of course me right oliver engmar max silvio stefan and a very heartfelt thank you to nicole unfortunately she did support us during some reserved timings itself she's not here at this moment with us today but for next conference i'm pretty sure nicole will also be with us apart from that we have our lightning striking amazing volunteers you will find them in the blue t-shirts a huge shout out to them can i have a round of applause for all the volunteers

please thank you

thank you so much and before i wrap up there's one thing that i definitely want to say morning sessions are really amazing but you know what the actual fun is when the good things are kept for the last and today for our closing keynote we have one such amazing lady who would be presenting us sharing us the knowledge she is the author of best-selling books who which basically helps you guides you and supports you in performing secure security implementing security in devops operations and how do you shift left and calculate your success and the talk would be presented by tanya jenka so before i go orbit uh goodbye to you for now i would like to give a huge round of applause

again to all of our sponsors especially the platinum sponsors thank you so much we also have gold silver and the brown sponsors thank you all so much because of you plenty of things were possible today and thank you so much for your support so one more thing and i'm damn serious here now this is where i get talking about the real business code of conduct we do not tolerate any inappropriate behavior if you see something you should say something if you find that there is something wrong if you hear any racist any derogatory statements from someone please speak up please reach out to all of us who are wearing these orange t-shirts we are here to help you

okay no illogical behavior we are one we are one community we respect each other that is what we are and again i'm super serious about this one masks on i know i'm not wearing a mask right now because we have huge distance and i'm wearing the mic it becomes difficult to talk so speakers are given a liberty to not wear masks on the stage inside this room and the other track room you're not supposed to eat or drink if you want to eat something you can step outside and eat but please keep your masks on all the time and yeah just a heads up for the today evening session we welcome you to join us we welcome you

to help us grow so please be present and if you're interested just reach out to us because this is how we grow whenever i wanted to uh ask something how do i progress in my life in my career there was always this question whom should i ask i've always dreamt of having a community a group of individual people who think alike like me who support my ideas and there has always been this question with me that how do i get this information from but to be true uh to tell you the truth i was also intimidated like oh my god that person's like super awesome he or she they are at a great level how

do i go and ask them i always took a step back from asking that question and always in my mind i thought that would there be someone who would be willing to answer these questions who would not laugh at my silly questions and say that's okay i have an answer for you i always had this huge question and i'm pretty sure some of you also especially the ones who are freshly out of your graduation and college you have this question is there someone who can guide me is there anybody out there who can guide me and the answer is yes and to give this answer today i would like to call upon the stage our keynote mr thomas shrek

who is the phd holder and the professor at the munich university and today he has decided to take this mental and give us all an answer how do you grow in your career what are the risks what are the downfalls even if you fail that's okay always treat your failure as success or at least a learning and if not nothing you can definitely write inspiring quotes on instagram that also works so without any further ado i would like to have a round of applause for mr thomas shrek thank you so much thomas [Applause] i think my mom's get off good luck thank you so let's see if the technical work is working okay perfect yeah thank you very much for the nice

introduction and also for the besides munich how you called organizers to invite me to give this keynote so when they asked me about can you give the keynote i was asking myself what should i talk about it's always always the same thing what do you want to do and for me when i was young and listen to keynotes and still doing listening to keynotes i always think how should a keynote look like it should be inspiring it should be giving the new sword so the first obvious choice was let's talk about cyber but i hate this word cyber so here's the normal slide like about all the cyber buzzwords around there and i will not

talk anymore about cyber so you have set one so the next topic which came to mind let's talk about blockchain because i don't know how it's you how you are doing but i get a lot with this buzzword so every manager coming to me every company let's do something about blockchain blockchain blockchain blockchain i can't hear it anymore so here's the obvious slide deck slide about blockchain i will not talk anymore about that so then we had a really good talk with the business organizat organizers and we discussed about what can give back to the community so i i grew up in a large organization you know sometimes very bureaucratic organization and every time i went there i was like okay

this is really the way people are doing that so i learned a lot when i joined communities as the introduction was saying communities are so important you learn things and you get new ins and inspiration and so on and therefore we said okay let's talk about communities so why can't i talk about communities so i joined very young with a community and i was there as a yeah newbie first time attendee however you get called and then i was at this large organization called form of internet response and security teams which is the largest organization for security teams in the world i was there on the board of directors for six years and were the

chairman for that and believe me or not it's it sounds great but it's a lot of work so i know what the besides people are doing in the volunteer time i had professional staff around so i want to use this opportunity now that you give them all and not only the volunteers because near forgot about thanking the whole organization team are a lot of our plows because what they are doing is amazing and they're doing all that for you so please give a lot of applause to them now [Applause] so i want to share the experience i had with communities building up communities running communities learning from the communities and so on so this is also a bit of a personal talk

so when i was finishing university i was i thought i know everything and i learned the total difference when i joined the first organization first company and i was very overwhelmed i learned new buzzwords every day and i was like oh god university have not prepared me for the job and what did i do wrong with with what i learned and so on so i was really i was really sometimes a bit nervous if if i'm as an engineer doing the right thing and back then when i started we had not had this company cultures where it was still hierarchy driven and so on so i looked at the topics what what we're doing on a

daily business and i always thought is it really how this work is it not really boring is that not a lot of problems and on so i had a lot of questions every day and all these questions were not really answered by my teammates because those were the ones were doing that all the time the same thing so i said question not what they were doing so i felt really alone i felt alone with my questions i was like there am i in the right place am i doing something wrong and so on so the question came to me is there anybody out there who has the same problems i do who has the same

perspective on topic than i do and that was for a large i think for over one year a huge topic i thought about i i questioned myself and so on so is there anybody out there by the way which band no one pink floyd right so my favorite band so it was obvious choice so is there anybody out there and i can tell you yes there are there are a lot of people out there you see it here in the community there are a lot of people out there and there are a lot of people out there in in the whole in the whole industry we have who are doing work who do volunteer work who share experience

on their own but you need to find that you need to engage with them and what i want to do today is to discuss with you about my experience i had when i joined those communities learned grew failed and in the end were able to run off one of those communities and spend a lot of time in there and it's a it was a great experience it's still a great experience and i want to give you i hope that you when you go out of this of this talk today that you yeah have ideas about what what may you what you can do uh also to give back the knowledge you have so how's this talk structured today i

have three important topics which i want to cover the first is what is this about so what is about communities what you really should learn from communities and so on the next one is what you need to do so how can you give back what you experience what you learned and so on and last but not least what you gain so what you can get out if you do that if you do it and on so let's start about what this is about so who if you are part of a community so who so the people that still have her hands raised who are part of the community so who thinks that's a lot of work being in

a community okay that's interesting i have a totally different perspective on that so when i talk about communities what do i mean so i'm there are a lot of communities out there there are this super exclusive globs where companies pay five five numbered membership fees annually and then there are the more industrial driven organizations who are okay are based on volunteers and then there are yeah communities like b-sides or others who are totally volunteering like yeah b-sides over and on so we have a very we have a lot of different people like the isaacs who get a lot of money every year they have professional staff working on topics and you have those larger communities which which get money

from sponsors and running it and then you have the total volunteer driven ones and i can tell you one thing in all those communities there's one thing which is common it's all about the people who are engaged with that so if it's if even they have a lot of money they are in the background people who really think this is the thing which we need to drive this is important for me so they spend their whole spare time in this community building it up and so on and that's always the same thing but why do you do that so the first thing why are doing that is learning so for me when i joined after this one year

thanks to a former manager of mine one of those communities i came there for five days and i can tell you i went back home and i was totally full of sugar and totally overengaged and so on because i gained so much knowledge during that week and that's mostly because the nice thing about this community is there are people there who are in the industry for way longer and who are willing to share what what um what say experience what say faults are so i grew up in the caesar community and the nice thing about the caesar community is they are always in the background so caesar people very don't talk about in the public how cool they

are they are not in not on on stages doing huge keynotes they are in the background and securing our network our internet and so on and when i went there there were all those people who were in the business for 30 years 20 years and when i talked with them and showed them i'm really happy to learn from them they gave back all the knowledge they have to me and i was lucky that two of sam which i call now friends took that opportunity up and and helped me support me to grow where i'm i am now but it's not always one way it's not always that the orders give back to the younger what i

learned is that as a young engineer i question topics and questions hey is that the correct way doing so is it really what we did as a best practice for 20 years something which should be done like that isn't there new technology out new new thinking in the uh in our society where we need to adapt our brushes and the exp a good experienced person in our community is seeing this potential that people are questioning and hearing and not just saying no no no we have done that for 20 years we know that's working and that's the wrong attitude to to tackle topics you need to listen to the people understand what you if it if that's

maybe something what you're doing for 20 years or wrong and accept that you may be wrong so the former team i was running it was always important for me that i listened to what they are doing what's it what say what say what's the experience they have and so on and for example when i was doing programming i was really bad in that and there's one person here here in the audience who need who i think hated to read my code overflow so and for me it was important that i was i was not telling him i'm the older one i know what i'm doing that i listened to him and said okay you're better than

that so please do so so learning is not always about one way orders giving the experience to young words it's also the other way that all the older people are listening to the younger people understanding hey maybe they are right or they're not or we are wrong and we should change what we are doing so that's an important very important question and i choose this picture by purpose because in our global society and especially what we are doing on the internet we also need to listen to other cultures what i've experienced in some communities is when there are people from other cultures since the western uh in the western culture approaching you they always say okay yeah

what are you talking about in africa i don't care what you're doing africa and this is also which we need to so uh yeah we need to improve because our cultures are so important the technology decisions which we are doing so it's not only learning between us it's learning listening to other cultures our societies and so on and with all the crisis we are kind of seeing that's getting more and more important so the next thing it's not about it's not only about sharing so for example the organization i was on the board first it was known for the huge conference so they had an annual huge conference people around the world came there but it was mostly only

known for this conference and i hated that because there are these c-shirts around there who really want to engage in on a daily on a daily business of working each other so if there's an attack on going there are structures in in blaze where ceasers can communicate to each other and say hey we have this problem can you step in so i don't know who if you are isn't are in the response but in zero response is not tied to one organization it's most of the time or you have a community which needs to work on that so this is so keen that you have ways in place when you need to work with each other that you say hey i have

a problem and then you step in and support those people if you go through management for example so i had this sometimes when you do my report to my manager and my managers that then to the next manager can we talk with this company it takes ages and when lawyers are involved then it takes forever so direct connections to each other are so important so building up a network in the community is so keen so you should not only go there and say great i will sit here and listen and then i go home you should be in there go out talk with the speakers talk with others build up a network because this network is so keen on daily operations

and it's not only being there at a conference it's also doing the conferences joining a mailing list give talks somewhere so it's it's keen because when you have this fire fighting happening you need to use this this network and help each other and there are various communities out there where you can do that but they're also something which is called trusted communities trusted groups and those people are those communities are sometimes called fight club so there are they have fight club rules i am not really huge fan of these fight club rules but there are communities who are individuals and these individuals share their knowledge share their operational work sometimes when i open my email box

and i've seen what the tags are ongoing in this main list i was like wow who should how should i process that all this information we have so you see this is ongoing this is something which is helping each other is so important so this is what you are able what is this about learning from each other sharing the knowledge supporting each other when there are things happening but as i said before you also need to think about okay how this community is working so you come here you listen to talks you already learned that there are here people volunteers who supp who are organizing all of that there are sponsors who are paying the

money says that we can be here to conference hotel and so on but imagine if there would be no volunteers doing that no sponsors and no speakers and so on nothing would happen so you would instead of going to munich and talk to each other you would need to go to the office or her home office and do your work so only because of volunteers as this is possible so what can you do that this can be improved the situation we have yeah first of all share your experience so i think all of you have on a daily basis um yeah something is happening in your office you build a tool you had an incident or you had a good

hack but you never want to talk about because as for example with me i always search for example oh i was really doing a bad job there so i should not talk about it or oh i don't have the time to prepare talk or whatever you find as a next as an rule that you don't do it and that's wrong so first of all what i learned is um talking about failure is often more important than showing off so i i'm for example now back in the academic field and i can tell you one of the worst things about the academic field is that you only share great great great papers in big conferences and then

you gain again another point that you're good talking about something but for example in the academic field we don't talk about failures even there should be those talks the industry is a bit better here but not as it could be so talking about failure especially in our field is so important because everyone can learn and leverage from that next when you work on something and you think okay i don't i'm not really sure should i publish that or not do it i don't know how your company policies are and so on but put it on github and share it on twitter talk about it that's very important give go to a conference talk about it then you get

feedback and maybe find people involved in that i have seen so many cool projects dying on a laptop's hard disk because people have not wanted to share it so for example my students when they come to me to write their final thesis i always say i don't want to have this source code on a usb stick or on a cd and not kidding still on cd sometimes put it on github tweet about it because that's maybe no one is interested in but maybe there are then you said then you build up your profile so what i want to encourage you is that everyone who is here now in the audience will submit the talk to next next year's

b sites even your manual can be selected [Applause] if you not get selected it's an experience you have you get feedback and so on so that's important share your experience then help others so you are here because you want to learn something but it's not only about learning it's also that we as people are very complex we may have our personal problems and so on and that's so keen in a community when you have a smaller community you build up friendship and therefore it's important that you listen to others maybe they say ask for help you don't understand that they're asking for help so you need to listen to them carefully understanding what the problems they

have and then they help helps them and this is something which i experienced quite often so i had personal problems i had problems in the shop and so on and i have always people i can ask out of the outside of my company within my company and so on and this important thing what i wanted to give or what i want to share with you is first of all if you need help if you have a complex problem for example you need to solve you should not be alone just ask just experience people or people you trust and tell them hey i have this problem can you help me or if you have personal problems because

you don't really know how your career path is ongoing there are a lot of communities outside supporting you on that so ask for help and more important be there if someone is asking for help so talk to them and and use this time i know we are in this industry we are all stressed out and we're doing a lot of work on a daily basis and then you oh and now it's 9 00 pm and i still need to talk with this one guy because he is asking for help and that's the total wrong the total wrong attitude to it it's something when you help someone you may later on when you need help this person

may help you as well and i can tell you i i was at the situation quite often and it was always very important for me that i had someone to talk a talk so now to the last point now i told you about what's all about these communities i talked about what you need to do or what you should do and so on but what do you really gain from that so first of all knowledge and as you know knowledge is keen in our society it's very important that you that you gain this knowledge and for i can tell you from my experience that's the most what you get from a community is exactly that knowledge

so what's what what kind of knowledge so i can give you now some examples um so i was working on a on something within my former company so i said we were able to detect some uh attacks and i had a problem with privacy so the way i wanted to do it was from a private perspective wrong and i knew that but privacy is important for all of us so i was really there for three months i have not found any solution so i went to a conference and discussed it with a friendly circle as it's a luxembourg czert and i talked to alexandre i had this program and just told him what problem i have

and he looked at me oh that's quite easy just use that in that algorithm and that is the data structure then everything is solved i was like wow this was a very easy answer so i went back home implemented that and it worked and yeah this knowledge that helped me on a daily basis and that's just one example another example is uh when i was when i became manager i never really were a good people manager and some of the audience can tell about that so i didn't want to discuss it internally and also the team i i was in there was totally different to other teams so i couldn't really ask for help within the organization

so i went out there and discussed with others who were running successfully teams like mine and said okay what are you doing what you how how how do you solve that in that problem or how do you tackle that uh and so and they gave me wonderful ideas about what you can do and that's also knowledge it's not only about technical knowledge it's also about how you yeah manage people how you how you grew your people and so on and last but not least building up friends so for example my best friend i i met uh at a conference or first of all i met him with on a mailing list and i thought who is this who is this person who is

asking so stupid questions but being biased and now he's my best friend because we found out that hey we have the same uh we have the same technical interest job interest but also private interest and i can tell you for sure that all of us are bit same not only only in the industry but also in our private field so you gain friends and like i i must say when i when i look into my friendship most of those people are within my community so these are the people i trust a lot i i talk on a daily basis and even i have private problems i have them because they knew me they know how i'm working

and so on so this is what you also gain a lot of friends which you can use the whole career path and later on hopefully also when you retire you can do nice trips with them and so on so that's also important that you just don't just look at the look at hey this is just a job because our job takes the majority of the time we have on a week so you should share that with friends and not with with people who you just in there for eight hours on a day so this is what i want to talk with or talk about um what i learned why for me communities aren't so important and so on

and i hope you get your yeah you gain something out of this talk that you learned about something and that you submitted a talk next year thank you very much i hope you enjoyed it

[Applause]

does anybody have a question for a wholeness it's a great opportunity some questions right now if you have please step forward

so first of all thank you for your talk when you mentioned uh we should talk more about failure uh it reminds me of something that i read i think in uh this book from uh i think it's called it and fragile but basically the guy makes this argument that we have this a big graveyard of failure behind us so when you see like say book authors uh we have like someone that releases a book and we have like i don't know ten thousand people that never released a book but tried to publish something uh could you please discuss a bit more how would you approach that how could we discuss more failure uh give maybe some ideas

so you you can approach that on on various levels i think there is a conference format out i can't be called the name where you just talk about failures uh i think that's one approach the second approach is within organization you what you can talk is that you do uh what you can do is on a weekly basis on monthly basis you have uh an evening where where you're only allowed to share failures you did and then on a daily on a daily job basis uh you may do a instead of doing a scrum or dailies and so on you also talk about in the dailies about what did what did not work and how

you solved it so you need to facilitate a culture where you discussed it and way more important is that from a mindset you need to allow your allow failures so it's quite often that people think okay if it if i fail i'm doing a bad job it's quite obvious it's quite obvious it's um uh opposite sorry obvious i said opposite it's you need to understand hey when i'm doing a failure it's totally fine that it's a failure and now i need to understand doing lessons learned about what i what i did wrong and honestly when i have a small child at home and she is doing the whole day failures i can shout at her

and say you're doing it wrong or can i can just show here why she did it wrong and that's like a mindset chef we need to we need to do and i can tell you our society unfortunately still doesn't allow failures so starting to change in our society that failure is allowed is the first thing which is important to do but a great question okay thank you very much and enjoy the rest of the day

do

oh both as well

test is

uh

hello

so

business

hello

okay so take your seats please

i guess we'll give you a minute

a few minutes okay

let's wait for the few hours

okay great so we are one minute in delay um yeah i'm very happy to introduce the next speaker and the first regular speaker connor morally who will talk about the mac os endpoint security framework what it is what it can do and i'm very much looking forward to this

thank you good morning back in 2019 apple at the worldwide developer conference announced they were making a dramatic change to the way they were dealing with the kernel space on their mac operating systems today i'm going to be dealing with one of the results of this namely the endpoint security framework or the esf i am a senior researcher with secure within their countercept team prior to being part of the research team was a threat hunter four years i have produced multiple white papers and proof of concept detection codes and presented at a number of international uh conventions so what are we going to be going over today we'll be going over what exactly the esa is

why the esf is important to the detection industry how can we in the industry utilize the esf some of the issues during implementation and some of the solutions that i found to these issues my device poc solution which i code named esfang and briefly going over a use case against the return pressure agent so first off what exactly is the esf so the esf is developed as a kernel-based telemetry acquisition system by apple and it's been developed as a way for a one-stop shop for telemetry needs from the security industry as you said it works in the kernel space and because of that it actually works fairly similar to the way the etw works inside the windows

operating system in many ways it succeeds the open bsm or the open basic security module by sun systems that was used primarily for auditing purposes prior to esf and one of the great results of esf is that it allows for real-time event acquisition for direct intervention on real-time interception but what does this exactly look like so the image on the right here gives a very basic overview of how this works from the user space at the bottom with a endpoint security application feeding into the kernel space and getting messages from the kernel space for the telemetry and on the left is a demonstration of that information which uh from every single event type includes multiple

low-level data points including the parent process id path environmental variables cd hashes and so forth but why is the usf important so during the presentation at the worldwide developer conference in 2019 they announced they were going to deprecate text or kernel extensions primarily this means that they were removing third-party current extensions from the kernel space meaning that uh third-party security vendors could not use their own kernel extensions for telemetry acquisition because of this they introduced something called system extensions which is their way of allowing kernel extension proprietary apple kernel extension access from the user space using something called system extensions equally before esf came along open bsm as i said was a primary auditing tool

but it was quite clunky to use and fairly difficult to integrate there's lots of documentation on the problems with open bsm so i won't go into them but needless to say earsafe is a much easier solution to getting these telemetry points so let's have a look at the old way of how monitoring worked so at the bottom we have the user space top kernel space on the left the logs so what would happen is is that third-party vendors would have their monitoring program hooked into a third-party kernel extension running in the kernel space and acquire telemetry via that point then using open bsm hooks they would hook into log files and acquire them through an auditing process run them

back to the monitoring program for detection uh processing the new way is slightly different so as we said the system extensions work in the user space and they work on frameworks that were introduced by apple to directly access kernel space using their proprietary kernel extensions these work out as the network extension framework endpoint security framework or esf and the driver kit framework and using a amalgamation of the three you have very high visibility over the low level telemetry of the operating system however we're focusing on the esf and from that it actually hooks into the endpoint security current extension getting developed by apple so it allows them to have this proprietary pipeline from the user space system extension to

their maintained kernel extension but why make the change so the one of the primary reasons that apple made the change to remove canon extensions was for increased stability and security so by removing third-party kernel extensions you avoid things like the black screens of death but equally you avoid third-party malicious kind of extensions from being introduced into the operating system this means that they have much greater control over what is happening in the kernel space and it just eliminates a lot of the overheads and a lot of the problems that are being encountered with less stable current extension that we're finding in the wild and it means that everything becomes very uniform one of the things to note about kernel

extensions is that even though they've been deprecated they can still be used they're just you have to dramatically reduce the security profile of a mac operating system in order to do so primarily you used to be able to introduce current extensions on the fly on demand however now you have to degrade the security profile that's running then you have to turn on canon extension and then reboot the system which means you can't just inject them as you would but how can we use the esf so this slide's a little bit messy so i'll just walk you through it so in the bottom right you'd have the main code running in the user space of your

program you would then create an endpoint security client object this would then subscribe to the endpoint security kernel extension with the event types that you want so you specify the event points you want like file creation or process creation or anything to that sort of thing the endpoint security kernel extension then crea populates an event message queue in the kernel space and then pipes that out to your endpoint security client which you have already subscribed which you use to subscribe to the kernel extension then on on each message a callback is generated which you can have a callback monitor and then each message that's received can be processed back into your main code for detection purposes

but what are some of the issues with the esf use and what are the solutions so during my experimentation with esf back in early 2020 there were a number of solutions that came out from multiple researchers and vendors but there were some very primary issues that came out with its utilization namely a bottlenecking issue uh system verbosity and a real parent process id issue i shall go through each individually first of all a bottlenecking issue so this was actually something that i was banging my head against for a couple of weeks even a month two months and effectively what was happening was is where you were doing exactly the same experiment you would get different data points

being populated and you'd have different numbers and it didn't really make any sense as to why so you'd conduct the same malicious experiment using the esf telemetry acquisition but you'd always get you always get mismatched data points and you'd have data points that were being missing for no particular reason and equally from a detection perspective this means that it's inconsistent and therefore the data points are unreliable and it was a very long-standing reason before i figured out exactly why this was primarily the issue was caused by the kernel level q so you may remember from the previous slide i said that the kernel extension inside for the esf would populate a queue within the kernel space but what was

happening was is this queue was being overloaded because the amount of data that you were subscribing to was simply too much so if you try to subscribe to too many event types the queue gets overloaded and suddenly drops data packets which then don't come back to the user space but there's no indication that this happens so although the data packets are dropped from the kernel side because the kernel side is obviously secure and it's isolated you don't get a notification that this has happened which is which is problematic two primary solutions to this are a development solution and a esf proprietary solution i'll go over the event muting first so event event muting is a function that was built into

esf and what it does is it allows for specification for events to be dropped inside the current extension space now what this means is that through process path or process token you can specify particular event types to be dropped on the kernel side and not published into the event message queue inside the kernel extension because of this it means that you can avoid overloading however it's quite a blunt tool back when i was experimenting with it because you either had to specify an entire process path again assuming it hasn't been hijacked by anything or you had to do it by a process token which only works for that instance of the process it was either too much of a sledgehammer

or it was too um unspecific it wasn't targeted enough so for a um viable solution event muting just wasn't specific enough for effective filtering multi-client system however did turn out to be very effective so whereas you would have a endpoint security client within a process subscribe to event types instead of having one client subscribe to multiple event types you could have multiple clients subscribed to individual event types which equally generated their own queue within the kernel space and by using this method you don't get overload and you get all the data you require and you can conglomerate them into one data set and then run it through your detection stack by crops referencing which was very

effective one point to note is that um the original development was on sdk 10.15 for catalina this was when esf was introduced but between 10.15 and 10.1 5.4 the sdk was suddenly updated one with a number of new event types they equate something called the sec num or sequence number in the es message so whereas i was saying before there's no notification from the kernel that silent drops have occurred that is still the case but by using the sequence number you can see whether there's a gap in those sequence numbers to see whether a message has been dropped due to overloading so it doesn't alert you per se but it does allow you to filter that

data and detect when it's happening and from that do diet things like dynamic rebalancing in order to prevent overload system verbosity um if you look on the image on the right you'll see that the original uid is zero this is the system user id on the on the system on the operating system um and because it works at the kernel level it ingests all the processing and all the file information for the system as well including system daemons and system level propriety management systems and this creates a problem because we've talked about the bottlenecking issue before the simple amount of data that's being processed by the esf when you take into account the system events is

extremely high and because of this you end up with a sort of overhead issue where you have to either filter out these system level events or you have to allow them but then do additional filtering for anomalous indications so one of the things again is that you could do it either before or after collection so before collection as we said would be by event muting but as i've said before this is quite a blunt tool and to use this for this sort of activity it means you'll be dropping the only way to do this would be to drop all processes running at the system level which can introduce ignorance of system level compromise which could be very dangerous

so instead you have to look at client-side filtering but then you look at the bottleneck issue that i mentioned before and even when you start to filter down the event types into individual clients you still run the risk of data events being dropped simply because the system level processing event generation is just so high so it's a becomes a balancing act between the two um there's no real one-size-fits-all solution for this so you have to sort of balance this dynamically as well as possible parent process id issue so this is an ongoing issue in the mac operating system primarily this relates back to identifying where the original process was spawned from so this actually goes

back to things like xpc proxy launch d and more recently something called running board so you may see at the top there's a real ppid of -1 this is actually by using a ppid solution um but as you can see because it was running on a newer operating system the result has actually come out as -1 which is incorrect data just to demonstrate that this is actually an ongoing issue that there isn't really a permanent solution for at the bottom you can see that there is launch d being launched um xbc proxy being launched uh launched by launch d but that doesn't necessarily indicate where an original process was called from the reason for this is because of

how cross-process communications work in mac os it's not really that easy to identify what process was initiating another because it jumps through these other programs so when you start to go back to the process chain that can be gapped due to these underlying system processes as i said there is no real future-proof solution the real reason for this is that apple keep changing the way that this cross-process communication works for process um spawning so any time a solution is made for one operating system version it can be made obsolete by the next update which as you can imagine is fairly frustrating the current solutions that i was looking at the time were true tree by john

bradley and launch xpc by patrick wall launch xvc was actually a solution that was implemented in my esfang solution but it doesn't take into account running board at the time which means that as was demonstrated by the minus one real ppid it is not a fully fledged or conclusive solution to this as i said this is an ongoing problem that there isn't a one-size-fits-all solution for yet um and it it was beyond my scope to to try and figure that one out so looking at the solution that i developed or my psc poc which i code named es fang so first of all i need to shout out to patrick wardle chris ross and omar ikram

they back in the early 2020 really set the stage for how esf could work the sort of telemetry acquisitions and really basic um fundamentals of how to ingest esf data and my plc was based primarily off of their work and then expanding on it and introducing capabilities and solutions that i mentioned before for some of the issues that i encountered my solution was actually developed early in 2021 um and it was based off of 52 event types back in sdk 10.15.4 the primary reason that i did this was that the old solutions had a fixed definition of what event types it would ingest and as i said before this meant that overloading could occur because it was a fixed number of event

types which you couldn't necessarily deviate from which if it was causing eventing you couldn't change from the rigid structure however the solution that i put in was a dynamic solution so this allows you that each boot you can specify which event types you want to subscribe to so you can have multiple espang instances subscribe to different event types or you can have them subscribe to multiple event types or group types like file file creation data or file metadata um uipc data um cross b cross process communication port data and things like that um using the solution you can avoid the overhead um the bottlenecking issue i when i started this had never coded in objective c before so doing

multi-threaded processes to achieve this as a standard solution was actually a bit outside the scope but it is it should be easy to do moving forward from any objective-c programmers out there one thing to note is that because this was not done with an apple developer id it is a plc code so sip has to be disabled or system integrity protection so esf cannot be accessed with sip enabled unless the program that you're using has been signed by an apple developer id so if you want to use this for testing purposes you have to disable sip on your system so this isn't viable for production level systems the other thing is that it was designed

for json output for upstream integration so all the output is put into json format so it can be easily piped up into a database for further detection processing and easy acquisition by um elastic or threatening teams to those data sets so let's go over the merterpretty use case use case was done on mac os 11.2.2 i used esvang to collect all the data and this is for the agent only so this isn't to do with the metasploit frameworks capability that can be piped through the interpreter agent it is only for the native interpreter agent capability this is only done on the post exploit phase so i'm not interested in how the agent got onto the system only what the

age you can do once it is on a system and this was done only against a single host so not against an enterprise interconnected sort of demonstration so it's not necessarily representative of a real life attack but it's to demonstrate what esf telemetry is capable of so the overall findings again the graph on the right is very small it's it's a representation of all the commands that were executed and the data that was generated as you can see from the top solution is actually webcam streaming which is not opsec obviously and as you can see it generates a massive amount of data but as you move further down you can see that some of the data some of the commands generated

seemingly none of very little event types but i will go through some of these to demonstrate some of the data values that can be found so during the install you can see that uh open read directory and protect were the highest event types that were generated through esf for merterpreter installation now memory protect and um memory mapping for new processes you expect this to occur and memory protect being that high is actually not that anonymous even though it's quite a high number but if you turn this against what a normal process would do on the mac estate it actually doesn't stand out as that anomalous especially when you look into the internal data the open event however is slightly more

anomalous alongside the fc ntl or file control events the reason for this is that during during deployment the amount of file that it accesses in order to enumerate number of system enumerate the system information and equally to change certain file access controls can be indicative of a temperature deployment so actually looking at these two specific event types does allow you to process profile interpreter installation on a mac operating system

so breaking down that slide a little bit further in total 259 data points are generated during an interpreter installation but 259 data points doesn't necessarily mean that they're great data points quantity is not always an indicator of quality but if you have enough data points with enough cross referencing you can generate more accurate processes profiles and higher fidelity as long as you know what you're looking for and esf during my experimentation showed that it had both quantity and it had quality event types so by filtering the two together you actually get a very accurate representation of when anomalous behavior has occurred but let's look at some of the other event types that are generated and

against what sort of commands so on this slide again so this is to do with file open events so on the far right you can see webcam stream of 478 and it's not operation secure so that stands out like a sore thumb process listing is slightly different so for each process that's running it opens an individual file for each process to see whether it is whether it is running um when you look at that event type individually you can actually filter out that scanning each individual processes file which is quite easy to profile if you already know what processes are running on the system you can just scan for is asus is a process scanning for all

the running processing files and detect against that the screen share and system information follows the same sort of process so in order to enumerate and gather information about how it's going to screen share it has to open up a number of system files and system information it does the same way as it does for process listing instead of using general system information commands from the bash terminal and instead accesses a number of system um files on the on the operating system in order to enumerate that information notify right this is a really interesting one so edit file when it targets an individual file to edit it writes the information the file and you get one writer then

upload is slightly different so when you upload using meterpreter instead as it's transferring the data from your attack machine to the compromised machine each packet is appended to the file routinely as it's received which generates this nump this huge number of write events to an individual file but the write a size is quite small and it actually corresponds to the size of network packets being received from the interpreter agent so from this you can actually target one processor targeting one other file in the right event and doing it in very small increments repeatedly especially if you filter out things like uh internet browsers and things like that you very quickly can isolate when this is occurring

io open is another interesting one so this is to do with driver or hardware access and again you'd expect to see webcam stream 220 because it's hijacking the webcam streams data feed but screen share again because it's hijacking the user interface uh display feed it equally generates io event types and again io kit is not something you see commonly it's quite a low generated event so finding any io kit events outside of system uid is quite anonymous and makes it very easy to identify when malicious behavior is occurring this is a list of some of the valued event types done against other that i've isolated from other investigations against things like uh max shell swift

sliver merlin and other attack frameworks one of the interesting ones uh one of the primary interesting ones is on the right there for pty grant uh the reason for this is that peter where guard is for pseudo-terminal generation so for a dynamic user session instead of piping the command through a bash terminal like you normally would instead max shell swift generates a pseudo terminal which is a master slave comparison and actually executes the commands that way which is slightly weird and you don't see that very often so in summary esf is extremely powerful and it's very clear that apple is taking the time to really look over the problems that are occurring during its use and equally to refine the

event types that are available during s d progression from sdk 10.15 to sdk 10.1 5.4 the number of event types increased but equally so did the capabilities of detection when the system wasn't necessarily doing what it was meant to because of this um it's actually quite nice it's actually quite reassuring knowing that apple have made this not really optional due to the deprecation of kernel extensions from third parties and the fact that they're actually maintaining this shows that they realize that they've taken that control away from the security industry but are actually taking steps to provide all the information and the stability that is required as we said some of the teething issues to begin with have been addressed but

there are still more that exist and during more and more experimentation they will rear their heads as we go forward as i said earlier um compared to prior solutions especially developing your internal extensions or using open bsn which is very clunky esf is very streamlined it's very easy to integrate into your existing stack and because of this it makes integration and development on it very easy from a security development standpoint looking at the use cases and some of the other experimentations that i've done in the past it does mean the detection capabilities that are available are very high and it means that moving forward we were able to create better detection profiles based off the esf telemetry

that is available at equally expanding event types that they're they're allowing the the final point to note is that as as i've mentioned the number of event points that are generated especially because the system level and even because of the low level temperature that's available is very very high and because you don't have control over the canon extension itself filtering is absolutely essential and it becomes not optional and it's one of those things that we're just going to have to deal with moving forward in when developing using the system that's my email uh that is the github with the esfang source code if anyone has any questions please feel free to email me afterwards um we do have some

time for q a so if anyone has any questions please

so if anybody has questions please come in frontier to the first mic just come to the mic yeah yeah thanks

okay uh so uh thank you for the talk uh i was wondering from a perspective from a blue team uh what kind of events i'm kind of i'm um i can find when i look at the telemetry because uh i'm more familiar with the windows events and sometimes we can find like credentials passwords what's not good so we have to remove them do you know if there is something in place that remove this pi data or do i have to do it with myself so yeah that's a great question so you'd have to do this all manually the events that are generated by the current extension themselves are raw so when you get them from the esf client side they can't be

tampered with from the current extension side so it would will have to be filtered on the client side itself and there's no way of doing this further up the chain okay so if i decide to ingest this in some like uh splunk so i have to take care of this yes okay okay cool thank you very much no problem yep you mentioned that there's the user space component that is registering for for the filters or for events sorry yeah uh two questions basically is there also a unregister element that i can for example call if i'm doing a red teaming assessment and i got i gain access to this user space component can i then

call an unregistered event that i basically disable monitoring for that uh so because the event monitoring has to be done together and it's done by an individual endpoint security client object inside the process there isn't a way to unregister them from an internal process unless you hijack it yourself and then basically destroy the object and recreate it you can't do it from external because it's an internal uh code object all right then the other question is also solved are there any other questions

yep we have time for another question nope okay thank you very much thanks for listening

ready

okay everybody take your seats please we'll continue in a few seconds

okay awesome so let's continue with the next talk i'm happy to introduce to you daniel feister who will tell us how to mess with edr systems okay [Applause] so good morning besides first of all thank you to the team from besides munich for the organization and for giving me the chance to speak here today welcome to master of puppets how to temper the edr my name is daniel i'm the founder of the company infosectihole with which i mainly focus on offensive security service on windows and i also spend a lot of my time in researching learning and in the area of antivirus products cdr products and the windows internals today we take a look at the mitre

subtechnic imper defenses motor disable or modifying tools and we focus on how can we disable the main functionalities from an edr by targeted control tempering of specific key components from them but we want to achieve this without relying on an uninstall password or token uninstall software uninstalling the product generally or by using the windows security center i would like to point out when i speak about edr systems in that presentation i always refer to products which are also in cloud including an antivirus module so an epp edr combination also i would like to point out it's also only my personal research my personal experience and the shown strategy or concept applies to multiple products on windows

in the first step we try to get a better understanding from the different components from edrs in user space and in kernel space we learned a little bit about their functionality and important house the relationship between the different components in user space and kernel space in the second step we want to use the gain knowledge to find a way or to find more or less the key element depending on the product to permanently disable the main functionality from an edr and permanently get rid of prevention by the antivirus module and permanently get rid of detections and telemetry footprint host isolation read time response and edr sensor recovery feature by the edr module so we have big plans

at the end of the presentation we should more or less able to depending on the product to disable the edr so this presentation is not about zero days it's more about learning a little bit about the windows internals and how do edr products work on windows so there can be some situations where it also be possible to do some activities in unprivileged user way but normally you need a privileged user in high integrity or system integrity level but despite um everybody which has fight around with eds knows that despite you have a privileged user most well-known edr products can still be very annoying and be a problem normally it is not possible to simply uninstall the product

because if the blue team has done his homework correctly um you have to know the uninstall password to uninstall the product and as mentioned in the intro we want to achieve this without relying on an uninstalled password so imagine the following scenario you have landed a successful fish and you were also able to escalate your local privileges my case i was able to use the print exploit to escalate to system integrity context and by having a look at the process structure of the compromised machine i saw that there is another interesting user session open so believe it or not in my case it was one from the domain admins and i knew okay could be maybe could be easy to get

credentials by dumping elsa's process or to impersonate as domain admin by token stealing but the problem was that i created a many alerts in the admin console from the product the blue team was informed and i get isolated from the machine and that was the starting point more or less from my journey digging deeper into the windows internals and edr products and find a way to disable the main functionality from them so we start with the user's base component of a media and have a look at edr processors normally edr or the processors from vdr products are executed as process protected light processes this means that even if you have achieved system integrity context it

shouldn't be possible to simply terminate the process but in the meantime there are a few ways how we can deal with our ppl processors from red team site so one possibility can be to use the more or less the concept of a driver with the with a null access vulnerability like the msc afterburner rtcor 64 driver so when we have a look at the picture the first step we try to escalate our unprivileged user to a local privileged user in high integrity or system integrity context and by this we should have the privilege to load a driver on windows and the by loading the rt core 64 driver we get also right access to kernel space

because of the null duct access vulnerability and um compared to user space in kernel space and windows there is no isolation between the different code sections this means that theoretically we would have whole access to the whole access from kernel space in this case we will use the one level driver to attach to the e-process structure from the ppl edr process and we temporarily patch the ppl flag and can then use uh different kind of tools to terminate the not longer protected process so for example ppl killer uses the rtcor 64 driver in his code or maybe cuts brings his own device driver also an interesting way depending on the product it can work if you are able to

execute process hacker in a privileged way depending on the product it is possible to directly terminate the ppl process in the system session without removing the ppl flag in the first step the conclusion of process tampering is that there are ways to temper the process to terminate the processes but from my observation this termination was always just temporary so if you terminate the process a few seconds later or at least a minute later the process gets restarted and ddr is is back there because of this in the next step we take a look at the user space component and a closer look at edr services so we have to identify the service which is connected to the protected

process and the protected service and the protected process more or less builds together the user's base component but similar to protected processes even if you have achieved system integrity on windows it's not simple possible to pause stop or disable a protected service but important in also important in our situation when we have a look at the recovery tab from the protected service we can see that the service is the component which is responsible for restarting the process the ppl process after terminating conclusion on services um we learned a little bit about the relationship between protective processes and protected services uh similar to processes is it is not possible also not possible to temporarily disable the service

but maybe we can use a find a way to still disable the service because of this in the next step we take a look at the user base component and edr registry keys so you have to identify the edrec key from the user's base component normally you can find them under control set current control set services and there are two interesting entries so um launch protected and start entry because time is a little bit short we will focus on the start entry and by the start entry we can have influence on the initialization behavior from the protected service this means if we would be able to change the end the value for the entry for the

start entry from the value 2 which is equal to autoload to the value 4 which is equal to disabled it should be possible to disable the protected service and furthermore the user space component from the edr but the problem similar to processes and services from the edr even in system integrity it is not possible with most products and depending on the product when you try to tamper the red key you will create an alert in the web console that's what this was the problem in my journey and the reason why i was isolated from the compromised machine so the interim status at the moment we are not are really able to permanently disable the edr or the user space

component but we learned a little bit about the relationship between the different components and user space and we see that direct key could be more or less the key element by changing the value for the start entry to permanently disable the user space component but at the moment it is not possible because the rec key or their keys are protected by a temper protection a mechanism by the edr because of this in the fourth step we make our first step into kernel land and have a look at kernel callback routines so since the introduction of kernel patch protection hk patchguard it is no longer possible uh officially possible for er vendors to set their hooks in kernel

space so they are forced in users base to use users based api hooking but despite patch guard they can use in kernel space a mechanism which is called callback routines and register different kind of callbacks to realize different kind of tasks in user space so for example they can use the process notify routine to register and realize telemetry collection in context of process creations also they can use the process notify routine to realize user space dln injection and furthermore realize user's base api hooking but more important in our situation now kernel callbacks routines by adr products can also be used to protect their own registry keys so this is more or less could be the the key element for

the register for the tempo protection for wreck for the registry keys um in this case the product is using the cm register callback function but we will see in our first pre-recorded demo because i have to blur every sensitive information we will see that not just the cm register callback can be used they also use all the callbacks to uh protect the red keys and in the first demo we try to patch the process notify callback temporary key disable the user space component and have a look at the impact

so for first plausible check because at least we want to dump okay

okay that looks better

uh for first possible check to see that the enterprise component and ddr is completely configured and at least we want to get credentials from the elses process we execute the pre-compiled version from mini cuts which you can find on github so we execute it and we should see that you get prevented by the anti-wireless module and the file gets deleted in the next step we make a short possible check in context of the temporal protection so at the beginning we try to terminate the protected process by executing process explorer and system integrity context but we are not allowed to do this also in case of the protected service even in system integrity we get an

access denied and if we try to tamper the value for the start entry to disable the user's base component to the value of four we also get an access tonight and depending on the product now we will have created an alert and you get with a high probability isolated from the machine by the blue team because of this we use a very nice pc which you can find on github it is called cheeky blinder it's not from my side i called it in this case pari.dxe and in the first step we will use that poc to load the driver with the availability the rtcor 64 driver to get access to kernel space so we load the driver the driver is

initialized and in the next step we list all the registered process notify routines on the machine and the blurred one on the lower side is our routine from the adr product in the next step we will use the poc to temporarily patch the callback

and after reopening the registry it should now be possible to change the value for the start entry and to disable the user space component because temple protection is now no longer active so we change the value to 4 which is equal to disabled and we have to reboot the machine

after the reboot we see at the lower right side that now the edr product is no longer registered in the windows security center and also by having a look at the structure from process explorer we now see no longer blurred sections because and there are no longer processes by the user-based component also if we check the status from the user's base component we will see that the star the service is now stopped so it looks very nice maybe we have until now reached all of our goals but this is not the case because after the reboot we have a few problems we again list the registered callbacks on the machine and we will see that all the previously

patched callbacks are re-registered again this means in case of prevention and detection based on kernel callback routines and furthermore windows uses user-based api hooking we can have the problem of prevention detection and especially telemetry footprinting so again when we execute mimi cuts we get again prevented and despite the user space component is disabled we still have the problem that the blue team can use the isolate function to isolate our compromi compromised machine so in a few seconds we will see that we lose connection to our compromised target

so what's the conclusion from the first demo we saw that we can more or less use a concept of the willable device driver to get access to kernel space remove or patch the respective callback temper direct key change the value for the start entry to 4 and by restarting the machine we can permanently disable the user space component but we also saw that only disabling the user space component do not really have a strong impact in reaching our previously defined goals and no matter after if you have rebooted the machine and you would again patch all the callbacks from the edr you would still have the pos the problem that um host isolation the recovery feature

and um the last one yeah the features which bluetooth can be used in the web console is still active so the biggest problem is that despite the user's base component system is disabled and you patch all the callbacks you still have the problem with the host isolation even if you would not do a reboot and you were um you just want to temporarily patch the callback um you still have the problem that um also your your host can still get isolated by the blue team so not really efficient from this point and we have to take our last step in the final step we take a look at the edr mini filter driver and the mini filter driver is the

component which is responsible for in general registering callbacks from the edr and that is always also the problem why even if the user space component is disabled the mini filter driver is a separate component is still active and but is after the reboot the callbacks get uh re-registered again but the good thing is that the mini filter has its own registry key and has a similar structure to the user space component this means that depending on the product the minifilter driver can be more or less the key element to permanently disable the main functionalities and get rid of prevention host isolation return response and edr recovery feature to check this out we will have a

look at our second demo where we try to tamper the mini filter driver and permanently get rid of prevention detection telemetry collection and so on

so we start at the point where we have stop remember we get isolated from the machine so we will lift the containment and get back connection to our compromised target and very important now we uh want to re-enable the user space component in the first step and only disable the minifilter driver and to and because we want to check what is the impact if we only disable the mini filter so we patched list the callbacks again we see that the process notify routine is still there we query the user space component service which is currently stopped and we check the status from the mini filter from ddr and we see that the mini filter is still

running so we open the registry and re-enable the user space component by changing the value from the start entry back from from fork to the value 2.

but we're not allowed why remember we have did we did a reboot so we have to to patch the callback one time again and reopen the registry

so we patch it again the process notify routine reopen the registry we now re-enable in the first step the user's base component set the value back to the value 2 and then we go to the mini filtered key and change the value to 4 which is equal to disabled reboot the machine and after the reboot we see that now the adr is still not registered in the windows security center but then when we have now um look at the process explorer we see again blood sections reasonably this is i have to learn because um there again uses base component processes active uh quality starts from the user space service we see that the service is now

again running but the mini filter driver is now stopped and at least we check the impact by disabling the minifilter we see that no longer callbacks are registered we try to isolate the machine again looks like the temple protection is no longer active so we can change the value however we want without creating a detection or creating a footprint based on telemetry and finally we can execute mimi cuts and get credentials very relaxed i would say isolation is still not happening so it looks like that isolating the machine is no longer working

okay so what's the conclusion from the second demo we saw that compared to disabling the user space component depending also on the product this mini filter has a much stronger impact in case of reaching our goals and to permanently disable main functionalities from the edr and permanently get rid of prevention by the antivirus module detection footprinting host isolation and so on at the end of the presentation i would like to point out that in my opinion this is not really based on availability more it's based on the concept from the windows architecture and i think every vendor excluding microsoft has to play on the same rules on windows so many thanks for your attention [Applause]

cool thank you very much um we have some time for questions any questions yeah please just line up at the mic and whoever is faster testing testing seems to be working your almost last sentence was every vendor except microsoft has to play by the same rules yeah can you go through the accept microsoft thing yeah i think the difference is compared to third-party vendors that microsoft is i think not really only forced to go in user space so because since the production of patch card officially it's not allowed also if even or maybe if you use a patch card bypass then you can also go in kernel space but i think that from my experiences

microsoft has a very deep or deeper visibility because it sits it adds more visibility into kernel space compared to other vendors so follow-up have you looked into cutting them off or their edr products do you need additional criminal space disability disabling capabilities or no at the moment i i have only researching in the area of mini filters also i also write a blog post about elon drivers so i had a look on different components but not on etvs when you mean this one etw that will be the topic for my next project looking forward to it thank you thank you so how do you think they would go about to fix this um do you think there's any

probability within the kernel space or from the isolation perspective and do you think windows 11 could fix this is there anything on the way with tpm 2.0 to fix this and what do you think how hard would it be to um for somebody interested in pen testing to learn this um do you think there you could make a lap about this or anybody learn this or is it just too oversimplified here no you can definitely learn it i also use it by myself for sure you have to be very sensitive because when you do a failure in the kernel space you will create a blue screen of death but if you know the product very well

you can use it depending on the machine so maybe when you are acting a very sensitive machine it is not maybe good when you do this what you can do about i could observe that well-known products are blacklisting uh begin to blacklist the drivers which have vulnerabilities this one way but it is possible to use different pocs to maybe flip some bytes and get bypass by that by that detections another possibility would be there is a software vendor on the market i will not say the name but they focus on a mechanism to get which is based on the web fiber weap variable and um the products have the possibility to include this code into

their code and by they will realize when there is something changed or tempered they can re-enable it or repair it in on the machine and do you think um windows 11 will fix something about that or with tpm is there any way uh vendor could facilitate it to protect um in in any kind it's a good question so at the moment i had another deeper look at windows 11 but i think generally with the drivers you only can use drivers which are released for july 2015 afterwards it is more it's harder but there are still drivers which you can be can be used um on windows 11 i can't say too much because i have too less experience at

the moment thank you thank you wonderful any more questions

okay cool thank you very much very interesting thank you

thank you

[Music]

a few last participants yes

okay welcome everybody back after the break we have now ethan and javier here on stage to i saw on the slides you're fans of kitties yes that is correct and you will tell us some details about automotive security so i think i just handed over to you because we are a tight schedule and you have a lot of awesome slides yep thank you thank you so let's get this started um hello everyone thanks very much for being here we hope that you're enjoying besides munich after such a long jog town so our talk is going to be obviously about hacking i mean you guys can see the picture there's two hackers on the on the slide so

let's start with the introduction so who are we i'm javier i'm a hardware security specialist on the embedded site i am from cadiz from the south of spain i enjoy reverse engineering products because quite often more than not i'm disappointed with the features that devices have and i always want to improve them myself right like why not if i can and i do like cake when it's not alive has happened sometimes and i do like barbecues i mean barbecues are good uh hi my name is ethan riggs i am from california in the u.s and i just graduated from college about a year ago and it's a mechanical engineering degree and a math degree but the math degree was more

for fun honestly but over this last year i've been able to work with javier a lot and i've been really lucky to work with him in automotive security engineering specifically in the hardware and a lot of the embedded systems electricity parts and i would really like to have that i'm looking forward to more so why are we here okay well current day car hack has become kind of mainstream you could see examples up there of people who post things like hacked on their dashboards like look at me i've done all this work or whatever uh a car is hacked um that's that and that's uh was mainly popularized in the 90s that's can injection injection it's not all that

difficult you can tell because you could probably look up a guide for it you could probably find something to do it um in about 15 minutes or so but further than that the reason why it's so simple is because a lot of the mechanisms for security on the ecu are pretty simple and they kind of uh they don't need to be but they are because the ecu's are so lightweight and they are so resistant to noise because they have to be they're in cars the cars are going to be jostling around there's going to be a lot of noise there has to be i might have said that already whoops but on the uh and this goes hand-in-hand

with diagnostics which are also pretty simple um once you understand what they're doing because they're talking exclusively codes once you memorize them it's you can read it like you read letters on this or like you read words um yeah what we'll be showing you specifically is three specific uh what are the hacks yeah some let's call them floss in the implementations of the state machine for diagnostics right and to add to what ethan was saying these two pictures i find them personally funny you cannot see the hacked here but it says hacked here so you have to believe that one typically these people start thinking that they can inject a cam frame and they put an ascii string right so it

shows up here now they're car hackers and that's the first and the last tweet you see about car hacking that they make because automotive security is actually not as easy as just injecting so let's talk a little bit about the automotive diagnostic protocols and this is just a heads up we are not going to waste waste too much time on that and so the the diagnostic protocols usually are based on a request response scheme where the tester or the device that you connect to the car is a client and the duty the device under test which is the ecu acts as a server right and then like a most things not everything unfortunately it requires like certain

authentication it has security measures for some functionality right so security access which is going to be the target of our talk uh which is uh the service id27 hex uh it is the the the tool that's used most commonly to retreat to restrict a functionality right so such as if you want to update the firmware on an ecu you will most likely hopefully need to perform a security access challenge response right the same thing if there are some things that could be critical such as the beam right it should not be easy it should not be easy to spoof the beam by just changing it right or there might be some other operations which are

restricted to the manufacturer such as adding a new key fob like sometimes when your car you have a key lost or you need a new key i mean you have to go to the dealer and then the dealer wants to charge you like 200 euros for adding a new key right that's because they have tooling that is specifically required to do that right i mean and security access is actually so relevant to automotive security that there's even a market for selling and buying security access algorithms okay so thank you um how the actual security access protocol works is you first you have the tester you have the thing that talks to the ecu or the diagnostic sports the dut

the device under test um and it starts off with the tester sending a uh request for security access which you can uh which means there it is request for security access which is that top left uh option right there the uh the dut recognizes that as you can see right here receives it and then it generates a 32-bit random seed this seed is random and it's once again four bytes which is important and will come into play later um then sends that seed back to the tester where the tester receives a c and both of them they calculate uh using an algorithm that is generally not known uh a key that you will have to use

to unlock it it then proceeds to calculate the key send it to the tester or send the tester sends it to the dut my apologies um and after it sends it to the duty the dut tests the key against the key that it calculated on its own and if it's correct it has two states that it goes into it go it says either security access is granted or on the other hand if it was incorrect they increment the wrong attempts counter and they don't give you access okay this is important because as i mentioned they're kind of random but more importantly that changes based on your security level um and most likely based on the dut so

if you're trying to get access to the very lowest level you can generally get access or you have one key for that and then if you get access to the next one um a higher level um you probably won't be able to get access with the same key and even then they're oftentimes random on top of that the security accesses often times change based on what level of security that you want which is a little bit different but if you're going in for a default diagnostic section uh hex 10 you are pro might not be able to do some things that require a a programming session finish which is which is hex 13 i believe

a diagnostic session is 10. so to add something and maybe this helps understand a little bit about the security access different levels think of a building right an office building so and all the doors have a key like a physical key like the one you guys most likely used to get in your house so now there's gonna be a let's say lower level by lower i mean like more more restricted so it would be the inverse if we're talking about actually reverse engineering so a higher level key which allows you access to a building so you get into the building that's a security access algorithm right usually that's the most simple uh so now you got that so you got into

the building now once you're into the building you can have access to different compartments offices right or rooms if you wish now the key for those rooms will be different and you can obviously not attempt to open the an office door if you're not already in the building right so that's what it means by switching different diagnostic modes diagnostics mods could be a standard session which is zero one right then you have you can perform certain security access levels which would be equivalent to getting inside the building then if you change to as a standard session or a programming session right which maybe you need to perform the this security access before it allows you to change to that session

then you could use a different uh more secure hopefully a security access algorithm right and i also mentioned before that um the wrong attempts counter increment and this is important because that prevents brute force attacks after you have three or so it depends on the manufacturer but then it locks down for generally 1 to 30 minutes just once again prevent proof forcing so just a for the for the ones of you that are familiar with uh coding a little bit i will explain it nonetheless this is if you can read it it might be a little bit too small but i'll walk you guys through the code this would be so the first part of the security access

uh challenge right response so one thing that we need to mention is that security access is divided into two different parts one is requesting the seed and the other one is providing the key they are separated usually you will see them one after the other but you could very well request a seed read the ecu bin and then send the key they do not need to be sequential so what does the request do this is the code from a duty the bystander test perspective so you just provide like the data which would be like the the raid uh contains the level so then the the duty checks for the level because as we said earlier ideally there would be

different uh keys different algorithms right now when it checks for the label the first check it does is see if that security access level is locked because the tester had too many wrong attempts right so if it's locked then it returns false it tells like yeah see you later like you've got to wait out the time if it's not locked then it will create a random number usually it's four bytes i mean there's different algorithms right but let's go with the standard it will create four random bytes and then it will just send those four random bytes so that's as far as it goes for the requesting the seed parts now the second part which is actually

now the tester uh cthan showed by before has calculated the the reply the response and sent it so the check key function the verify we call it here again we'll just use the seed that the ecu itself generated earlier perform its own calculation because obviously the ecu knows the algorithm right and then it would just compare if the result that it got is the same that the tester sent now if it was the same then yep oh good right there you go you got security access if it was not correct then it would increment the wrong attempts right because remember we got here because it wasn't yet locked so what happens when you up and you just like give the wrong

reply wrong attempts increase so hacker kitty knows something i don't think i personally noticed it but i trust him he has a cool hacker hoodie on he's attacked the planet behind him he seems like he knows what he's talking about yeah we'll we'll see what the kitty found probably in a little bit okay um the next thing is man-in-the-middle attacks specifically um man-elemental attacks similar to can injection they're they're not really new they were around in the 90s that they're used for a lot of attacks and it's oftentimes very effective but specifically because they were developed in the 90s at the same time kwp2000 and uds were also designed so you might think that they may take into

account this um this popular attack but obviously you probably guessed by the fact that we're presenting on it they they didn't it's still it's still very widely used in the current systems yeah which you can tell from security hijack which are um uh which was uh my apologies um presented on in 2016's black hat and defcon 24 by javier about this attack which uses a man in the middle on ecu's that are still being manufactured today and it's the standard man in the middle attack and it has three steps to it first of all you forward all traffic from the tester to the duty and from the du2 back to the tester until there was a

security access and then after that um the tester goes tester and the duty go through the steps that i mentioned before about getting security access and then it disconnects and suddenly without the tester there you have all the security access that the tester just gave you or would have had access to without them in the middle there so to make it like a perhaps easier to understand with letters that even i can barely see so i can't only imagine like what you guys can see there so let's imagine you can see them so here there's the tester side there would be the money in the middle attacker there would be the duty here right so tester sends the request for

seat right for security access the money in the middle forwards that doesn't touch the ac receives the request so it generates a seat that the money in the middle attacker forwards it doesn't touch now the tester gets the seed it does its calculations that we learned earlier the money the attacker forwards the result now the ecu verifies the seed and when the ecu sends a positive reply and only when that happens then the man in the middle attacker kicks the tester says goodbye and takes over diagnostic session so to the eyes of the duty this the money in the mail attacker and the tester are the same because there's like literally no validation there's no

authentication there's nothing so you end up having a secured a diagnostic session like having security access without needing to know the key or the algorithm so let's talk a little bit about the pros and cons this will be more obvious later so the pros of this attack where the the immediate one is that an attacker does not need to know the key or the security access algorithm like you just don't care you just like take a hijacker session that's the name right that's where name comes and the operation is easily repeatable with consistent timing i mean if you have a tool that let's say performs a firmware update right and you're going to use that

to hijack its session right then you can measure how long does it take for that tool since you press start on the flashing process until you can get a hijack the session right when the security access happens so it is easily replica you can do it like easily over and over consistently now the only con is that for every time you want to hijack a security access session you will need to have the tool so if you're going to be working on a project where you're going to attempt to hijack a session let's say twice a day for a month you're gonna need the tool for a month right so that's the only the only con

um now security hammer is a a programming attack that or software attack that javier once again developed and it explore exploits poor implementation project uh practices in state machine logic specifically for the diagnostics and before i really explain what exactly sec hammer does i kind of have to explain the uh implementation practices that it exploits um so i want you to first think about the tester it's set uh and the final step when the tester sends the key to the dut the dut gives you two options you can either have the right answer or the wrong answer but what if you didn't send that final answer then you didn't get a right answer or a wrong

answer this might be what hacker kitty was trying to point out earlier as well but back to set hammer if you instead of not sending anything you asked for another seed you weren't right and you weren't wrong but you still get another seed and then you do it again and again set hammer does it thousands of times until you have um thousands of seeds and with that you can look at uh randomize these randomized seeds that i mentioned earlier and you can see if they are really random you can try to look for some sort of pattern in them and just gather general information on them yeah so we are running a little bit tight on

time so we're going to have to speed it up nonetheless again another blurry image hopefully you guys can see some of it so basically this is an example of sega hammer on the first line we just start a standard diagnostic station well not a standard zero two so it says wait wait okay we got the diagnostic session then we request like seed level one yeah here's your seed seed level one here's your seed seed level one i mean on the wait time because we are using the standard but we could go way faster you can see that well you cannot probably but the ecu takes it from 855 microseconds to reply we're waiting 30 milliseconds we could narrow this down

to one millisecond so we could be getting around 500 seats per per second perhaps i mean like this is pretty fast and pretty powerful and like ethan said with this you can actually get a feeling of the randomness right in the security access the seed key algorithm which until now was not really possible now does it always work so they we tested more than 40 cus including my own car i mean like we have tested like issues on the bench different cars all the stuff of the 40 38 were were vulnerable and one of them we didn't consider it vulnerable because it was just repeating the same seat zero zero zero zero like no matter what you unplug

it we plug it always the same seat so we'll just say it's not vulnerable and now this is important some seeds actually they a little bit accounted for this not like a lot but they did so they actually if you request them the seed again they send you the same seed again now there's ways around that we need to remember that diagnostics is actually a state machine so if you are in a diagnostic session right and you ask for a seed then if you ask again and you get the same seed you need to change something now what can you change you can request perhaps security access for a different level because that would overwrite the seed value right again

we're thinking about poor implementations here so they have an embedded systems they have only one variable for seeds so when you replace it with a different security access then that gets replaced right or you could try switching to a different session right that diagnostic station because we also said that security access does not carry over two different diagnostic levels uh so or you could just like let's go now full here you could just disconnect and reconnect from the ecu right that takes longer but still that would work and the thing is that you don't get locked right like with typical security access if you send three wrong attempts you're gonna wait out from 10 seconds to 30 minutes

with this method you don't get locked out you just get like all the time all the seats and now sac puppet combines the sec hammer that i mentioned and the man in the middle attack um it's divided into three primary parts um first of all it analyzes the randomness it gets the randomness and it generates uh figures out what kind of seed or picks out a specific seed that it wants to use using sec hammer then you go over to the man in the middle part and it forces that seed once you're in the man in the middle attack so that it pulls out a specific one now once man in the middle has given

you uh that seed and you've using the process that we mentioned before found the key for that seed you can use set hammer again and force that same seed and now you have the key and you can get access to that level whenever you want without the tester using just sec hammer because you know exactly what the c that you can get and the key so to add a little something to what he has said so basically what we're going to do is we're going to use egg hammer we're going to figure out if the randomness is good we're going to figure out if there's a number that gets repeated more often than the others or the most

repeated number in the randomness or we're going to figure out if there's a pattern if there's like a counter that goes up and since reset then you know the initial value and then you can see it going up all these are true cases now if you find that value that you already know then you're going to use the man in the middle and what you're going to do he will explain in a second is you're gonna force your own seed right and then you're gonna get the pre-calculated uh reply from the tester and then you will use seghammer again to try to force the the ut to give you the same seed that seem to appear most commonly right or in

the pattern sorry okay sec puppet follows a very similar thank you um follows very similar patterns was before there's uh requesting to succeed for security uh for security access um goes uh the mail in the middle then more middle forwards it requests uh the duty request receives that seed request um and then it responds with the seed calculates it um before it uh receives that seed we force the seed force to seed very specifically

sorry so it forces the seed very specifically um as i mentioned earlier forces the seed for um that one seed that we want to have that one seed that we're going to recreate and reuse every single time and then once it gets that seed then it sends it to the tester and the tester then calculates yeah it calculates using um the algorithm that we don't know and then we save that key we save it so that we can use it whenever so really quick so here would be right when the duty sent say its own seed we would the money in the mid attacker would replace that seed with the one that we found out that gets

a mostly repeated right or the most common one or the one we want to find so then when we give that seed that we want to give to the tester the tester uses that seed and then it gives us the reply to that seed what does that mean the pros i again an attacker does not need to know or care about keys or algorithms right with this method uh this one unlike security hijack requires a one-time access to a testing equipment right just to do their money in the middle attack to get your seat precalculated and now the cons the the biggest one obviously is that the target needs to be vulnerable to safe hammer

right if you cannot analyze the randomness the source of randomness then that's not gonna work too well right or if you cannot get it to give you many seats right fast and then the target needs to have like poor or predictable randomness because it might be successful to a seg hammer but it might actually have good randomness and a few use cases for these attacks one would be this is more a rather fun test for testing than for an attacker it would be like testing the issue randomness source right like if you get a like an ecu front tire one and they are telling you or you having your requirements like yeah the the it should

be like truly random and then you perform this test and you see it's just sequential based on a timer on the microseconds that's the seed right or the ticks have seen then you can tell them like yeah this does not meet whatever another one would be like reading and writing memory or flash offsets that are like off limits in the sense of if you get like one of these tools from ebay that allows you to flash your ecu you could do security hijack or you could do a sec puppet or anything and then you could try to read into the ram range with that which that tool does not do by default right or any other tool

and then performing operations that are not supported by the testers like if you know like the payload to send to execute a specific action but the tool you have in your hands does not do that you could hijack the session and then send your authenticated command right and that was it we were a little bit over thank you very much for attending everyone we hope you enjoyed the talk and if you have any questions feel free to reach us out we're going to be around yeah so if you have any questions we have a microphone in front what would oh maybe you can walk up to the microphone in the center okay

hi good talk um given that the system would be vulnerable to the sec hammer could you just go ahead and do something like like rainbow tables just pre-computed uh the precomputation of all the seeds that you send um to the testing device and then simply save it and then basically use the one that that that you want to use just when it appears so that would be doable yeah the thing is that that would take too much time because let's go to this would be for the sec puppet attack right because on sega hammer you just get seats but you want to have a tester pre-calculate the key right i mean we're talking about the

scenario where you do not know the security access algorithm or you do not know the key so you will have the tester and the best you could do is like automate the testers starting the process whatever right or if the tester has literally a command to authenticate right uh then you could just like hammer the tester with your own seats yeah that would be an option i think i mean it is doable but it will be very slow that would be mostly useful if you have a subset of let's say 20 most repeated seats right then you want to create a table for all those 20 seats that is doable but if you want to

do the whole four bytes calculation that's going to take some time i mean if you're sitting in the middle uh you could be seeing all the traffic going on and never sending the response as you just did so even if that could overcome the good randomness um so you could work maybe with that so that was just i know yeah i mean that should be global yes thank you so it sounds like you have something to discuss afterwards off the stage so i have to kick you off the stage you kicked out the security no and we will prepare here for the next talk thank you very much everyone and here we will have our next talk

about usb fuzzing and give us a minute to build up okay thank you

yes

okay

let's wait until the other people are in and i think they're a bit late but that's okay lunch break thank you

uh

okay

okay then i see most of the people have arrived and changed so you are now listening to fuzzing usb from andrei konovalov and i wonder how you discovered this topic but i guess you will show us and what what your research will be about this okay no problem okay then i'll hand over to you and have a good talk thank you okay let's say you own an android device and an android phone and of course since you care about security your android phone is locked it's protected either with a password or a pin code or pattern now let's say you somehow misplaced your phone so for example maybe it was stolen from you maybe just lost it or maybe it

was even confiscated by law enforcement and now the question is can these people who now have your phone somehow unlock it and access the data that is stored inside and there are a few non-technical approaches they can take to achieve this but we're going to focus on a technical one and since each android device has a usb port maybe there is a way then they can somehow attack it over usb well normally if you just start plugging in like usual usb devices into an android phone you will unlikely to get any meaningful result to achieve this goal but if you manage to find a vulnerability inside the usb stack that is running inside android then

theoretically you could use this vulnerability to overtake the the android device and does unlock it and this was the focus of my talk so hi my name is andrei and i am a security researcher and i mainly work with the linux and android kernels and my talk is called fuzzing usb with raw gadget and essentially this talk is about trying to find a particular type of vulnerabilities in the android usb stack so i'm going to start with describing briefly describing the usb attack surface but the part that i'm gonna focus on are the usb drivers then we're gonna look in i'm gonna explain what fuzzing is and we're gonna we're gonna look into how to apply

fuzzing to usb after that we're going to check out a few ways to emulate usb devices but the way that i'm going to focus on is based on the linux usb gadget subsystem and the main topic of the talk is i'm going to show you how i used a new kernel module that i developed called raw gadget to do fuzzing usb to fuzz usb drivers within a virtual machine without any extra external dependencies so it can be used in any type of a virtual machine and it does not require any additional hardware and finally we're going to look at how to trigger usb box with raspberry pi zero and i actually have a raspberry pi

zero right here and hopefully we're gonna get to the demo where i will show you how to trigger one of the box that i managed to find all right all right let's start so first let's check out the architecture of a typical linux usb host i've split this architecture into three parts on the bottom part we have the hardware layer in the middle we have the kernel layer and on the top we have the eurospace layer and i'm going to describe this architecture from the bottom to the top so every linux device every linux host device has a usb port and this port is physically connected to a piece of hardware that's called the usb host controller

this cost controller knows how to communicate over the usb wire and since each hardware device in linux is supposed to have its own driver the host controller also has a driver in the kernel then this host controller driver is connected to an abstraction layer called usb core and behind this abstraction layer we have the drivers of different types of usb devices so for example we have the hid driver and hid stands for human interface device and this is a driver for keyboards mice and other types of input devices besides that we have drivers for my storage which is our which are flash drives and we also have drivers for video devices and other like there's

there's a lot of them so this abstraction layer the usb core abstraction layer allows a developer to write a usb driver regardless of what controller of what host controller driver is in use and finally on the very top on the user space we have some user space applications and user space daemons so for example when you plug in a keyboard and you start pressing the keys these keys are going to be shown in the text editor you're using and this text editor is part is also kind of considered a part of the attack surface and it's running in user space so here the goal our goal is that we control the usb device and we want to

somehow attack the linux usb host by controlling the device and we can actually attack each of the layers that we have we can attack the physical ports so for example there is a device that's called usb killer and when you connect it to a usb port it's going to charge itself from the usb power lines and then discharge 200 volts over the data lines and essentially just electrocute the usb port and maybe also the part of the boards maybe the host controller and then there are attacks that can be used like on the very top the user space level for example there was this old windows attack where you plug in a flash drive into a device and there is this

autorun file that is on this flash drive and windows will just run it and this way you could execute arbitrary code on windows so since i work with the linux kernel i targeted the kernel layer and the parts i was interested in are the usb core and the usb drivers the reason i didn't really care about the host controller driver is because different use different linux devices might have different host controllers and i wanted to find bugs in the linux kernel that would affect every linux device there is so i wanted to find bugs in the part of the kernel that are present on every linux device all right to find this box i wanted to

use an approach that's called fuzzing and fuzzing is a way to find bugs and computer kernels in computer programs and can also it can also be applied to the kernel so i have this definition for fuzzing fuzzing is you is fitting in random inputs into the program until the program crashes and it works like this you generate a random input you execute the program and feed this input in and then you check did the program crash for example the program could have seg-faulted and if the program i mean if the program didn't crash you just generate a new input and you keep going and if some point the program crashed that means great we have found a bug

because we generated an input that triggered a crash in a particular program that is a definition of a bug at least one of the definitions so for example if we have an xml parser we would be generating random xml files we would be feeding them into the parser and would be checking for parser crashes and at some point if the parser crashes then perfectly we actually have found a bug both a bug and a reproducer for this bug in the form of an xml file right how would we apply this to usb if we look at the definition again so first we replace the word program with the word kernel because we're trying to attack the kernel but then there are two

parts of this definition that we need to address so first we need to figure out how do we feed data into usb do we use any special hardware maybe we use some virtual machines maybe we use some other stuff and the second thing is we need to figure out is what kind of inputs does usb accept in general like what data with do we feed in what's the protocol and i want to start with the second part and talk a little bit about the usb protocol and the most important part about the usb is that it is cost driven that means the device when when the device is connected into a host the device cannot

just send data by itself it needs to wait until a request from the host so here on the right i have a device and on the left i have a host and let's say we plugged in a device into the host so at this point device cannot start communicating the first communication is happening from the host site and the host has the device like what kind of device are you and the device for example responds i'm a keyboard and this response is actually done i mean the device sends a few structures to the host and these structures are called usb descriptors these descriptors encode information about the device so for example they decode the class id which is a keyboard

and they encode the device manufacturer and so on and there's there's quite a lot of stuff so now again the device cannot just start communicating and now the host asks the host knows how to handle keyboards and now the host asks the device like what kind of keys do you have on what what purpose do they serve and now the device responds i have this many keys intended for this purpose and the device and now the host has alright you're now connected to which the device will respond okay this is didn't fit into the slide all right so the key part here that the device only responded to requests from the host and this is actually happening so for

example if you plugged in a keyboard and you started typing in keys you started pressing on keys the keyboard cannot just tell the host by itself that this key particular key is now pressed the host need to ask for it and this is it works like this so the host just keeps asking the keyboard like which keys do you have pressed and it's asked like any keys pressed here the keyboard says no any keys for executive versus no and at some point you press the key and the keyboard the keyboard is going to respond now i have this particular key pressed all right so the key part about the protocol is we cannot just send data right

so the way a usb file would work is first it would wait until a request from the host then it would generate a response to this request and send the response back and keep doing this in a loop until at some point in time it the fuzzer also needs to monitor the host for crashes and if at some point in time the host crashes that means we managed to find a bug okay this is like a high level higher level overview of how usb fuzzing would work now let's go back to challenge number one so we need to figure out a way how do we fit data into the usb and i started looking for ways of doing

that and the first way that i found is to use the face sensor board and i actually have it right here so this board allows emulating arbitrary usb devices and essentially this board has two usb ports one on the left one on the right and one of them is plugged into a computer that you control and another one is plugged into the computer you're trying to attack and then on on the computer that you control you can run a python library that allows you to receive usb requests and provide responses to them and this way you can emulate devices and of course since we can emulate devices we can use these these boards for fuzzing

the problem with this approach is that it requires hardware and fuzzing with hardware is challenging so for example we need to monitor our host we're trying to attack for crashes and if it crashed we need to reboot it and doing this with hardware is is difficult since that it would be great to have some kind of approach that works inside the virtual machine so i kept looking and the next thing i found is the usb redire module in common and qam is a type of a virtual machine and this module allows the host to inject arbitrary usb traffic for the guest kernel that is running inside camo and this can also be used for pausing

and there is actually other it's called virtual usb fuzzer and yeah it uses this qmo usb reader module so the great thing about this approach is that we don't need any hardware we only need camo but this is also a problem because this approach depends on qmo so we cannot use other types of virtual machines to do this kind of fuzzing and this was also a problem for me and the reason was that i wanted to use a fuzzer called syscaller and i'm going to talk a bit more about it later but the key thing about this caller is it it mainly runs on google cloud engine and google cloud engine is another type of a

virtual machine and that google cloud engine does not have the usb reader module so this this approach is great but it was not quite what i was looking for so i kept looking and another thing that i found was the linux usb gadget subsystem and this subsystem essentially allows you to take a linux based board like a raspberry pi zero that i have right here and turn it into a usb device so there is a way to configure this board so whenever you plug it in over usb it pretends to be a device and the logic of the device is controlled by the software that is running on the board right and most of the linux based

workplace most of the popular boards that exist they have the support for this so the beaglebone black also tested and i had an android board that also worked and the raspberry pi boards also work and theoretically if we if we have this kind of device we can also use it for fuzzing probably because it allows you to define usb devices and the problem here is that if you're using an actual linux based board we still require hardware but maybe maybe we can figure out a way to get rid of this hardware requirement so let's try to take a look into the architecture of the gadget site so this this this particular feature of these boards

is called this board pretends to be a usb gadget so let's take a look at the gadget architecture and try to figure out a way maybe we can do something about it so on the left i previously shown you the architecture of a usb host and now we're going to look into the architecture of usb gadget and here we're going to start from the upper part so from user space and the logic of the usb device that is being emulated by linux based board is driven by a user space application then there is a gadget driver in the kernel and this gadget driver defines the type of the usb device so for example there is a gadget drivers for

emulating keyboards there is a gadget driver for emulating video devices and so on then similar to the host we have an abstraction layer called usb gadget core and behind this abstraction layer we have the device controller driver unlike the host controller driver this driver is for the device controller that knows how to communicate over the usb wire while being a device not being a host right and then we have the usb port which is supposed to be plugged into usb host and this part on the left allows us to use a linux based board to emulate usb devices so now the question is if we want to use this approach for fuzzing we need to

figure out which gadget driver do we use and if you use it just the driver for keyboards there is of course this of course means that we can only connect keyboards which is not very useful when when we're fuzzing we want to connect all kinds of devices so i started looking and i found a driver that's called gadgetfs and it allows emulating almost arbitrary usb devices and the keyword here is almost the way gadget device works is when it starts it asks the user space applications for the usb descriptors of the device it's going to emulate and then whenever this this gadget stack receives a usb descriptor request from the host the gadget fs is going to respond to this

request by itself without asking user space if it receives other types of requests it's going to pass them to user space and ask them for response and then give it back to the host there are two problems with gadgetfs the first problem is that gadgetfs does not pass all usb requests to user space it responds to them on their own to some of them on their own and that means that if the same usb descriptor is requested twice we cannot provide different responses and providing different responses would be very useful for fuzzing because this would be likely to trigger bugs in the host and another problem which is actually a bigger problem is that gadgetfest has

validity checks for the usb descriptors that we provide to it and that means we cannot send mall form descriptors to the host we have to we have to send the correct ones and we want to send malformed descriptors because we want to trigger bugs so initially i wanted to remove these countries from gadgetfs i started trying to change the gadgetfs code but it has a lot of legacy code in there also the interface is a bit unusual so it was hard to adapt gadgetfs instead i just implemented my own gadget driver this took some effort but i managed to do it and this driver it works similarly to gadgetfs but it allows you to emulate

arbitrary usb devices and the key differences are that all usb requests that these gadget drive received they're passed to user space to get the response and then there are no sanity checks on usb descriptors and i actually mailed this request to upstream so i i mailed the linux kernel patches and they were accepted in version 0.7 i think it was released it was over a year ago right and i have a github repository in case you want to check it out with some examples and instructions on how to run raw gadget all right at this point we figured out how to we have our own new gadget driver that allows us to emulate arbitrary usb devices we still

have that problem that we need hardware to do that so what we could do here is that this part of the stack the gadget part of the stack is going to be running on our linux based board we can take a separately physical usb host we could plug this board into the host and then run the fuzzer as a user space application and this is a way to use the linux usb gadget subsystem for fuzzing for physical fuzzing of usb hosts now the problem is that we need hardware right and is there any way we can get rid of the hardware as it turns out there is linux actually provides a solution to this and this solution is called the

damage drivers so essentially linux has virtual host and device controllers that are connected to each other within the same kernel so previously we had the gadget part of the stack running on the linux based board and the host part of the stack running on a separate host device but here we can actually run both parts of the stack on the same device or like within the same kernel so this all would be running within the same environment and this is actually the key slide of my talk so we have our fuzzer running as a eurospace application we use row gadget to emulate arbitrary usb devices we use the damage drivers so that the usb device devices this fuzzer

emulates they're going to be connecting to the very same kernel the fuzzer is running on and this way you can do fuzzing inside the virtual machine or inside you can take an android phone and you can just run raw gadget and virtual damage drivers inside and you don't need any external hardware you're going to be fuzzing on an android phone internally and you're going to be reaching usb drivers from from within the kernel right all right and yeah this this was the approach that they ended up using and i actually integrated this approach into syscaller syscaller is a kernel fuzzer that is developed by google and it was initially developed to target the linux kernel but right now it targets a bunch

of different kernels and it has a dashboard you can check out the dashboard the dashboard shows thousands of bugs that this fuzzer has found and all of them are public you can check out debug you can check the buck report you can check out the zero producers so essentially dc programs that you can run to trigger a particular bug and i integrated usb fusing into cisco and so far it is found over 300 bucks in different parts of the linux usb subsystem some of them are in drivers and a few of them are in the usb core layer and the only thing that i failed to do so far is to actually exploit a linux or

android device over usb so there are 300 bucks and these bugs affect some of the drivers but not all all of these drivers are present on an actual linux on for example in ubuntu and when it comes to android android also restricts the set of enabled drivers severely so mainly maybe only a few of the bugs they actually affect android but the ones that do affect android i failed to find a way to exploit them at least so far now i want to get to the final part of the talk which is i think this is the most the most awesome part so first when using his caller to fuss the kernel whenever c scholar finds a buck it

generates a reproducer for this block and a reproducer is essentially a sequence of actions the father can do or like the the the father can do to trigger a particular bug so syscaller has a utility a tool to run these reproducers it's called csx proc so essentially when this caller generates reproducer you can run csx proc and trigger the bug and it works with usb as well because we're doing the fuzzing in vm we take in the reproducer with fuzz we run it in the semi-vm and we can trigger the debug that the father has found but the key thing here is that the interface between user space and the kernel is stable and it is stable in the

way that we can use any type of a device controller or device controller driver so which means we can take raspberry pi zero so we can take the reproducer that was generated inside the virtual machine we can copy it onto raspberry pi zero and we can plug this raspberry pi zero into the physical host and this way we can replay the usb communications we generated in a virtual machine with physical hardware right okay actually let me let me try to show you do you still have five minutes this should be enough for the demo so here i have a separate linux laptop all right and i also have raspberry pi zero so what i'm going to do is i'm going to

plug this raspberry pi zero into the laptop and let it boot

so what i'm going to do now is it's it's this demo is kind of weird because i need to show you the screens of both my laptop and the laptop i'm trying to attack so i'm going to be using a web camera to record the screen of the laptop i'm trying to attack

okay let's see if this works

okay we can see something all right so this laptop is running essentially it's ubuntu kernel and the kernel version is quite new we see it it's from april 22nd so essentially it's one month old all right so what i'm going to do is on this laptop where i plugged in the raspberry pi 0 i'm going to run the dmsh minus w command it's having trouble to focus the message minus w command which is going to show us the kernel lock in real time right and i'm not actually so for this demo i'm not running the simple the the most basic version of raspberry pi zero i'm running the version with wi-fi and it's called raspberry pi zero w

and i configure this board in a way that when i plug it in it boots and it sets up a wi-fi point so instead of using a bunch of wires to connect to that by zero i'm going to be using a wi-fi connection i'm just gonna ssh onto the board okay now let's see if we get if we have the wi-fi connection so on some technical conferences actually this demo has a problem uh we do have it because wi-fi is being attacked all the time it's very hard to connect but it worked all right now we have ssh onto raspberry pi zero so on the left we're gonna see the part of the screen that is recording the kernel lock of the

device we're trying to attack and on the right we have the connection onto the raspberry pi zero that is plugged into this device so first we need to insert the raw gadget module and now what we're going to do i have collected the reproducers for the usb bugs that's this color found and i have them right here so this this this is a special syntax that cisco uses to describe reproducers so essentially csusb connect means that this particular line is going to emulate a type of a usb device and the type of the device is described by all of these arguments so what i'm going to do is i have this run script and this route script

essentially runs sees exact proc that i showed here so i'm gonna do run reprose csbot and it takes a little bit of time to initialize so essentially what is this is going to do i have about 300 391's to be exact 391 different usb devices that will be replayed one after each other and this raspberry pi 0 is going to be connecting these devices to our physical host and let's hope that this works

okay now it's started so as you can see on the right we have the log from the from the part of the code that is emulating usb devices and on the left we can actually see in the d message that these new devices are being connected and if we wait for some time i'm not sure yeah at some point i think this particular this particular sequence of crashes so at this point you can see that the the messages stopped that usually means that we managed to crash the usb stack that is running on the linux laptop and if you could look look i don't have the mouse here okay okay now we got something so

actually the mouse is broken right now i mean the the touchpad broke because touchpad is connected over usb and will likely kill the usb stack on the machine so you can see a part of the stack trace that is that was triggered and yeah this is just some generic stuff so probably something something something bad happened the problem with running reproducers like this is that some of them corrupt memory and without running a memory corrupt detector a memory corruption detector on the device it's hard to tell what exactly happened all right we're running very long time so let me let me quickly finish the presentation yeah this was the demo so it worked all right so the summary is that i try

to attack android and linux devices over usb and i tried to find bugs in linux kernel drivers so the result of the work is that i used row gadget and i use the dummy virtual controllers to that allow fuzzing either or in a virtual machine and on device and this was integrated in this color and we managed to i managed to find a way to run this reproducers by hardware without any additional hassle and yeah by using raspberry pi 0 and row gadget we get a very cheap tool for usb attacks so essentially phase density costs about 80 bucks this tool costs about larger pi zero only is only for ten dollars you can buy

and finally the one final not very important that cisco is hiring so just in case you want to work on linux kernel fuzzing or linux kernel security in general there is a team and at google and they are here in the munich office and they are looking for a new team member so if you want to work on cisco this is the place you can apply so with this thank you and we have no time for questions so just in case you can catch me at the break and ask qualifications

i'm yeah no it's working um thank you andre so um we don't really have time for questions on the audience but um maybe you can yeah answer a few questions outside yeah yeah i'm going to be right outside right now and so you can't ask questions and our next talk here will be regarding building an ica ics firing range within your home kitchen with moritz and nico so stay tuned in five minutes we'll be back cool everything works yeah it's you can take this off now right around a little bit low on time

thank you

is

um

um

okay

okay um so we closed the door i think we can start nico and moritz we'll be talking about the kitchen within series or something no let's see let's see um thanks everybody for for being here and thanks for having us um we're going to present in the next 30 minutes a project that have been we've been working on moritz and i for the last year and a half and i was building a firing range for industrial control systems um we did that in our kitchen because of corona so there was no way to do it in the office and that's why that's that's that's why um we're going to you can see we didn't bring so we actually created a

model we didn't bring that but we're going to show some some videos of it later and then for everybody to take away as well the lessons that we learned so if you're planning to build something similar in your kitchen then you know what to do and not to do hopefully um so roughly that's the the agenda for the next 30 minutes um we try to get everybody on the same page what we understand is firing ranges what we understand as ot or operational uh technology then mods will talk us through how the building of the ics fire english actually um unfolded we're going to demonstrate it and then the lessons learned and then at the end the

questions about us i'm niko i'm the red team lead for nvizo i've had my fair share in ig security and i focus on ot security as well in the past two years and most of you my name is summers i'm part of the software and security team and r d team at nvizo and i'm an iot and ics enthusiast and i try and get a hold of as many devices as i can try and break them um a few words about nvizo it's it's a company that's roughly nine years old originally founded in in brussels a noun office in frankfurt in munich um we have 150 colleagues at the moment so it's um it was just a very small subset

obviously um but what's interesting is that in viso it's dedicated to invest 10 of the revenue into r d and that is how we can fund this project and the time that went into it um so that's um that's how good that that happens now firing ranges um what do we understand as fine range of course it's it's somewhat a controlled interactive environment um it's an abstraction of a real environment think of a toy train for example instead of having the real train of course to play with it should be as realistic as possible and that's what we try to do as well so you will see that we built a a model of a basket

bridge so a bridge that has leaves that are lifting so we tried to move letters as closely to the real thing as possible it should be reusable so we want to use it for trying out stuff um in our context it's i.t security so we want to try out attacks for example or defense mechanisms and of course if we run an attack we should be able to do it again and again and it should have that playground factor so it should be something where we can approach things from different angles and just see what happens if we try that way and so on why do we need a fire range so the first that comes to mind and that

was our motivation as well at the beginning is for training purposes so we've done a lot of id security ot security was fairly new to us back then so what we wanted to do is to train our people as well and ourselves on how to approach ot next networks and how to do security assessments on those the the lab itself the way it turned out is very suitable for awareness trainings as well because it's very visual so if you run an attack for example then you can make some some lights stop blinking or the bridge leaves are not moving anymore something else is happening there's a buzzer going off so for awareness trainings for example um very

useful for testing if you had any o2 components for example pll plc's that you want to try out that you've never played with before you just plug it into the lab or the firing range configured and then make it work and then last but not least for further development of tools for example for forensics so we actually used that lab for forensic training as well which you'll see then at the end as well very briefly now ot and ics operational technology are computing systems that are used to manage industrial operations for example the manufacturing process and the subset of that uh industrial control system so the integration of hardware and software components to control processes for

automation and humanitarian institute mentalization we do see two candidates here on the slides on the right hand side you see a programmer programmable logic controller it's very difficult for me to say that plcs which is essentially the core unit if you will of some automation process so it's it's the cpu that that runs was connected to to sensors and actuators that drives for example conveyor belts and so on and then on the other side that's an hmi human machine interface where the operators normally oversee the process they can start stop processes or react on errors and and so on so it's just the interaction with the machinery behind it now um i bet you're all familiar with it

the what many might not be familiar with is that there's a difference between security requirements for operational technology and i.t whereas in iit normally you would probably argue that confidentiality is has high priority so you don't want a customer database for example to leak to the internet that would be very severe of course if you shut down that database i um in regard to availability then that can have some impact but it's probably not on that same level whereas in ot it's very different confidentiality usually doesn't play a big factor but availability does so if you think about power plants for example if they just switch off or steal a steel mill if that is corrupted somehow the process and you

need to restart that might take days so there's a financial impact obviously but also the the production behind just the delay it takes to get every everything back up and starting so availability is key now attacks against ics this is the sun's ics cyber kill chain it usually comes in what as far as we know it comes in two stages the first stage usually goes against the it network so the way we know it there's a resource development phase at the beginning then delivery where for example phishing emails are sent to somebody working in the office they click the attachment the the payload installs on the workstation ensures persistence and establish c2 communication and then

the the adversary can act through the c2 channel and um yeah their objectives whatever it may be now if the attack goes against ot then this is also true so we are that that stage where we have a foothold on the the enterprise um we then recon where is ot located we identify possible jump servers or dmz that we need to pass through in order to get there once we're there we locate the targets we develop capabilities testing them so that might be payloads that we want to download on plc's for example then we download them install and execute the attack which could be the disruption of some manufacturing process for example the objectives of those attackers are commonly either

the the permanent loss of view of control for the operator so that they can't see the process they can't interact with it anymore it could be temporary only for vm control again but also safety which is a very important factor and the same of course for manipulation and the safety aspect is also very different to what we have in i.t there's rarely at least i don't know of any attacks against i.t infrastructure where you would say safety was a concern safety of people specifically and we had a case in 2017 um which happened in saudi arabia i guess a power plant where they found a malware that targeted safety equipment it didn't succeed but the assumption is

that if it would have compromised those safety systems it would have then disabled some of the procedures that would have triggered in course of emergencies for example if gas is leaking um into the outside then those safety systems would prevent this now if you modify or manipulate that then you couldn't or you wouldn't prevent it so this would not only be then of course um an issue for the um the the provider here but also for everybody that lives around the power plant if hazardous gases are leaked to the outside now morris is going to talk us through our ics-5 range please right thanks so how did we actually go about building this ics firing range well let me maybe

first start with uh what we had in mind what were our driving motivation behind building this so as niko pointed out we have a difference in security requirements in ot comparing to i.t right and we then have to adapt those differences obviously into our approaches to performing security assessments right so most importantly what we have to do to pay attention to is of course the safety aspect we can never know really what is the safety impact of of hacking a ot device if we don't have accurate depictions of the documentation of what actually those devices do so what we want to do and what we then succeeded to do is to build a playground where we can

perform internal training and to then perform research and development on this so we developed the concept of a water treatment plant which was comprised of three uh water filtration units and then as actuators pumping stations which pumped water from one filtration unit to the next one and to back this up and to represent a more or less realistic environment that drives this we had also scada and an enterprise network behind this fully virtualized so when we worked on this we were then contacted by a client who said well we really like what you're doing we would like to have this for us as well because we are actually operating critical infrastructure and as part of

this they operated bridges pasco bridges and they wanted to have this such a playground for their own digital forensics and incident response teams to find out okay what happened post-compromise and this should be also a mobile solution so they can then do those trainings at different sites so we adapted our concept to to do basketball bridges and this was our very first concept so here you can see it's basically an aluminum frame on wheels in the very top there you have a scale model of bascule bridges then in the lower half there's a front and a back side in the front side you see rot equipment such as plc's power distribution power supplies and the back there the

the area that you can't really look into because there are black panels there should be the virtualization server so uh we were familiar with building virtual environments uh we were somewhat familiar with assembling stuff but not very much with 3d printing and um yeah just uh designing this this bridge for example so that's what we started with you can see in the center there's our um our cad model of the bridge um you saw there were two bridges right so the other side is just a flip version and we 3d printed those so what i would like to point out here in particular are the most prominent parts so here you can see the bridge leaves

which are lower and race there are switches that are our sensors which detect whether our bridge is now in a fully opened or fully closed state and of course we also have traffic lights and a road barrier that goes up and down and when all of this works it's synced so that there's a process where the bridge opens and then the traffic barriers are down and the lights indicate that no traffic should happen here and it should all hold and then they go back down again the road barriers go up again and then traffic should commands there as we said as part of our title we built this in our kitchen this is my kitchen

and that's my oven actually much to the dismay of my wife we did this in my kitchen just to keep those prototyping intervals very very short so typically i would design a model in the evening kick off the print this would take hours and hours and hours and sometime in the afternoon it would be done on the next day and i could take it out of the printer and try and assemble it and see if it works or if i have to adapt anything about it so at that point we got a model but it couldn't really do anything on its own so we needed to do more to it we first assembled it then we got a hold of those aluminum

profile beams and built the frame and the back plate for the electro cabinet inside we put all the ot equipment in there wired everything together installed the bridges and wired them those are roughly 50 individual connections that we then put down to the ot equipments we put panels to it because they look nice and they of course hinder you from actually reaching into it while it's operating you shouldn't do that some sweet backlighting we put in there so you could actually see something because it was getting dark and we did a lot of testing an awful lot of testing many hours were put into debugging all of this and we at some point even had to recite

using an oscilloscope as you can see here to debug the motor control signals because they were not really in sync and they were off so we had the physical build but this only gets you so far what we were really interested in was actually powering it with a fully virtual environment in the back right so let's get to the network infrastructure that was then used for example for kill chains this should again be somewhat realistic and should be extendable so here's our network infrastructure those are four networks the networks in the dotted rectangle they're mostly virtual while the ot network down there is mostly physical let's go through all of those they were implemented in an e6i server

and they all use opn sensors for segmentation and segregation so just for routing and for the firewalls now let's let's see so in the enterprise we had an enterprise network we had a domain controller and a main joint office workstation which we heavily abused for initial foothold in our kill chains so there would be for example the case where we had a phishing mail and this the attachment of this phishing mail got opened and then it was compromised there was also a web ui that would allow office workers to then have a look at what the process is currently like in the uh at the bridge then in the skydive network we had an operator workstation which would allow

you to more closely monitor what's actually going on in the process taking place like what are the individual pieces of the bridge doing there was an historian which actually grabbed production data and stored it in a time series database and a jump post to which we will get back later the dmz honestly stays very empty for most of the time and still is we plan to put a windows update server may be there but we didn't and then in the ot network we have of course all our rt equipment so there are four plc's one rather new model so there was an s7 1500 they were all siemens plc's that was the area supervision that one would

orchestrate the other ones then there are two more or less budget models s7 1200s they were the lifting substations and they drove the leaves and the barriers and then lastly we got a later scene model the s7 300 which is if i recall correctly close to 30 years old and that drives actually the lights and of course we have the manual override hmi so where you can actually tap and change stuff around physically we also built in a cctv system using raspberry pi's and pen tilt cameras and here's one last virtualized station so that's the engineering workstation that will be used to actually program the controllers and that would also be more or less the last step in our kill

chain where which we could compromise to then effectively compromise the plcs because this one could program those right so knowing about this let me show you what it really looks like when we all assembled it let's see if the video works ah nice okay so here you see the model on top and it will just start its cycle of lowering the barriers and here we can see in the hmi that what you see on screen is actually sync to what's going on in the bridge model so now for example they are raising again taking a look inside we can see a whole bunch of wiring stepper motors uh and actually also active cooling we had to do this a whole bunch um over

the whole firing range because it got really really hot then there on the back you can see a desktop case so that's our esxi server there's the big switch and an ap for wireless and easier debugging and that's pretty much it and that's what it does all the time and now at that point we can go there and try and compromise it then we have some backup slides in case the video wouldn't work you can skip those now for the lessons learned what did we learn we learned a whole bunch first of all regarding the ics lab setup we learned that especially with the hardware compatibilities and dependencies that you've got assembling such a fire range can get

somewhat complicated when you're not a domain expert we are not ics engineers we are not mechanical engineers so those were things that were rather new to us you have to take several things into account right so power supply what kind of power source do you need what kind of protocols do those devices support what kind of protocols are relevant to your cases to your scenarios maybe so this can get uh this can be quite a bit uh then the the software licenses are quite pricey we'll get into this later and a practical thing that we figured out was that the stepper motors they tended to overheat and that was actually very bad for 3d printing

again we didn't really do this prior to this project so this was all new to us we are not mechanical engineers so especially the part with the bridge leaves opening and closing this was quite challenging for us to develop to design and then to implement and to make sure that it doesn't break because it broke a whole lot and then factor in the fact that printing is very time consuming as i said i would kick off a print at night and it would maybe be done in the afternoon next day we had prints we had individual parts that took up to three days to print so 72 hours and if anything goes wrong you're done it's bad

and also if you're not an expert in 3d printing you will run into a whole lot of problems ranging from from a to z so we had for example problems with the prints not sticking to the print bed so the print would just go and the filament would go all over the place but it wouldn't be there where you actually needed it and then to have models that you can actually adapt to your needs and to to modify further you would do this in cad again we didn't know cads prior to this so this was also required a whole bunch to learn but also there were practical challenges so uh this is this is the firing range

it was 80 centimeters by 120 centimeters in size uh only the frame was in meter in in in height and if you factor in the wheels and the models it's more like one and a half meters so every time we wanted to move it anywhere we needed to get a very very big van which was mostly empty but we needed it to put it there upright so we needed the vertical space and then also we had problems with the models right so here you would see that the the road barrier would collide with the lights up there it wasn't very nice the stepper motors were heating up there was the wiring which was all over the place

and there was very little clearance for the moving parts this was iteration one so we iterated one more time and their iteration two we addressed all of those problems here you can see those lights are set back so they don't collide anymore there are actually fans that we built into it so there's active cooling and you can just let it run for hours now it won't every heat which is nice if you want to do maintenance or if you have to do it there's a central back plane where all the wiring is just going which is nice and of course the clearance we have way more clearance here and then no more parts scraping on

other parts so that's great what's the bottom line though we started this in january 2021 we put more than a thousand hours of manual work into it more than 900 hours additional to the first 1000 hours which is net 3d printing time and a whole lot of this time was overnight and over the course of the whole project we burned through more than eight kilograms of filament we spent roughly eighteen hundred eighteen thousand us dollars on the whole project most of this for hardware but as you can see three and a half thousand euros are use dollars just for licenses which i hadn't expected in the beginning but here we go and we actually destroyed some stuff

over the course of the project so there were two stepper motors that we actually burned through one plc that's died presumably from faulty wiring uh and one motor driver that also died due to 40 wiring and at times we lost parts of our sanity to be honest and at that point i would like to hand over to nico again about what's next what is next thanks for watching okay so you if you paid attention i'm sure everybody did um the video differed the video that we showed differed from the 3d model for example um so if it wasn't clear we we actually created two iterations the first one where the problems the rules that that most mentioned and then to

make it better we created a second model so there's actually two models what we should be doing is creating a third iteration to address some of those points that are still not not that great so for example mobility is an issue you need still to rent a van to transport it and it was supposed to be very mobile it really isn't at this moment um so it could be for example that the disassembling of the entire um firing range is somewhat easier um maybe yeah that parts could be taken out of it um also the modularization is something that we we had in mind for example so we talk about basket bridges now but maybe we want to have a small

town or toy train or something running on top without changing the hardware underneath because it's most likely going to be very similar so we just would replace the the top on the model i.e the bridges with something else and then download new software on the plcs and then it should be done so this is something that that could be done in third iteration whether or not we're doing this we have to see um it's not clear at this moment what we certainly will be doing is develop more scenarios so this the intention of this firing range was to use it for training and we've used it for forensics training in ot for example so we had a two today

workshop on this which were quite great um but we currently don't have scenarios for example for penetration testing or red teaming in ot so this is something we want to to work on as well um similarly uh monitoring and detection ot i think is a topic that's very interesting and and many people are interested in interlooking that as well even in our company so for for this developing a scenario would be very worthwhile now if if you're interested in ot or want to know more about this project we're here to answer questions of course we're going to be outside you know after this talk you can reach us via these email addresses um if you want

to know what nviso does for ics there's a page for that as well and otherwise we have two twitter handles and a block that you can look up on the block there is a series um about this fire range as well so parts of what we presented a bit more detail as well in some in some areas and just check it out and see for yourself good any questions thank you thank you thank you so do we have questions i'm sure for the audience the front microphone if you want to take any questions so okay i see chen please come up telecool so thank you very much for this um i was wondering what other materials did you consider

when putting this model together like did you did you think about legos or something like this or did you go directly to printing i think we lego was in talks there was some talks about using a bit more wood as well to to get some parts done i'm not sure what we did not decide on lego in the end to be honest not sure i think it just looks cooler if it's 3d printed maybe it was occupied by children or something yeah as well yeah i think the probably the possibilities are it's just a lot more possibilities to create stuff if we have a 3d printer then we're fully in control of how it should look like and

especially the smaller parts than they are needed and i'm a long time no lego player but i remember there were no real bridges moving bridges so you only had trains and stuff okay um do you have any similar projects like other companies doing similar test ranges firing ranges that you exchanged ideas with honestly we no we didn't we know that not for bridges but but of course siemens for example they they have developed um also test environments for for traffic controls for example but it's a different price range as well and i'm not sure if they they would disclose too much about their projects towards us but no actually don't know maybe over you find somebody from

siemens over a coffee so do we have any other questions one coming up so most of the ics or the ot environment will have some physical control over the process not only that some malware can shut down the process while building your fire range did you also consider physical controls like you detect some attack or something and then there is somebody who is putting out switch or like why triton did not succeed and also scenarios like that so that's the hmi if you meant that so that's the hmi photo control so for example uh if you uh consider the titan thing it tried to uh shut down the safety system but of course it did not succeed there were

also physical controls on site and all to avoid this were you considering this to build in the firing range because that would be more realistic in an ot or ics environment right yeah so we can't put everything in that model that's i think um probably that's difficult um so we we um restrict ourselves to the couple of plcs that we have in the hmi um to and build the stars around that so additional hardware equipment we did not plan for at least in these situations so far yeah yeah this is why because it would not be practical to imagine that okay only through an enterprise network you get into the ot network because there would always be some physical control

over the processes as well sure so you're saying that the the attacker like they so they skip the it part and they go straight to ot and and uh carry out the attack from this is that no i meant like there would be some uh physical control that cannot be reached by attackers to avoid such scenarios and maybe in these kind of projects we can also consider this it could be it could be implemented in this show yeah i mean it's it's also extendable if this is a project for fuji i think it's a good idea to include thanks and are you going to put this on the internet for us to test as well

i mean you need that i guess you need data for forensic analysis so you just just put it on the internet and wait for us to test it maybe we'll think about it yeah okay okay then thank you very much thank you and before everybody heads out for lunch break we have lunch outside and you can also use the upper floor so to have a bit more space and i think we will continue here in this room at two o'clock so after one hour of break and see you then [Applause]

[Music]

[Music]

um i don't think that we're gonna magic one up in the next [Music] yeah and i think we have time [Music]

hello oh too many things

so it is two o'clock so she we should get started with the next talk in the where are we in the ballroom in the ballroom um today we are here as a community we're making connections we're exchanging information and guess what else wants to make connections and exchange information good answer blockchain so today uh our speaker sebastian baniskew is going to talk to us about not quite water under the bridge a review of cross chain bridge hacks and so sebastian take it away thanks so much for the introduction this talk is put together by some of my amazing colleagues listed here and i'm just sort of distributing the knowledge so a bit about myself i've been working

in security since 2011. held various roles at various companies and i've been in the blockchain smart contract auditing scene since uh 2018 and it's been a hell of a ride who i work for i work for quan stamp which is the company the logo there in the middle uh we're one of the leaders in smart contract and blockchain security we've secured more than 200 billion in assets in smart contracts we have a lot of clients a lot of revenue and most importantly we're hiring so stick around until my last slide that one has a very interesting piece of information on it so why is this talk relevant um there's been a bunch of bridge hacks

over the past year so just in 2022 alone there's been more than uh one billion worth of crypto assets stolen from such systems here are at least just a few of them the ones with the ones which are like most important and we're going to go through more details as to why these kind of things happen and why this kind of thing is so hard to get right from a security perspective so here's a bridge it's pretty simple like you'd say like why the heck is this thing so hard to get right you have an actor on the left side you have a smart contract system like ethereum that it interacts with and it basically wants to send assets to

a different chain and on the other end this actor or someone else might want to receive those assets denominated in something else on the other chain now this is like the essence of it the going into a bit more detail you see like when someone deposits like this actor on the left deposits maybe some ether it interacts with a smart contract called the custodian and there's a communicator in the middle which is an off chain component it could be a traditional whatever server application that is watching what is happening on the chain once it sees that someone has deposited funds in this contract it initiates a debt issuance call to the debt issuer contract on a different chain and this

debt issuer here on the right hand side it means some tokens which should be the equivalent of the ether that the first actor put in right so again pretty simple the opposite path so starting from right to left the actor wants to redeem back the ether deposited after having used it for some purpose so it burns these tokens that it got called ace it sends them to this depth issuer contract the communicator is also watching the debt issuer and when it sees that someone has burned these tokens it initiates the fund release on the custodian side so the actor should receive the corresponding amount of eth on the ethereum side so this is depositing and

withdrawing now you notice from the previous slides the bridge structure contains like three main components two contracts which are the asset custodian the data source and the depth issuer the oracle contract here and a communicator which is an offline off chain component which we call the oracle and there's therefore a huge attack surface when it comes to like the uh web free space so you have the custodian the depth issuer the communicator any interfaces that they use and the network itself they could all be subject to attacks and in the next slides we're going to go through examples of attacks for each of these layers and bear in mind that some of these attacks i won't name names because it's

not professional but some of them are uh referring to the hacks that happen in the in the past the ones that are on the one of the previous slides so for the custodian we have something which is called the call relay attack and here depending on the bridge you might have this asset custody smart contract which is accessible by some privileged addresses because they need to manage them somehow and maybe maximize the the profit of the project or something so what the attacker goal in this case was was to take over this asset custody contract and be able to take puns out of it so the way in which they want they did that was that there

was this function that was in the ascity asset custody smart contract called change custody address there at the arrow at the bottom and this function should not be callable from the outside so there should be some access control in place that prevents anyone from calling it but what the attack or like what any user would have would be allowed to do was that they were allowed to call functions of this form right there was a function name three parameters two byte strings and one unsigned integer on 64 bit so this was something that was deemed allowable from the access control policy because there were just one or two functions which were fine to be called by anyone

now the nasty part is that the attacker found a way found a hash collision on some random name that they put together using this signature here so they just randomly compiled a function name and some parameter values which had a hash collision with this function name the way in which this contract in the middle here works this cross chain communication contract it whenever it says call this function it basically hashes the signature and looks it up in a hash table and due to this hash collision it was the attacker was able to figure out okay if i have this function well it's just a random name there but xx123 and some parameter values is going to

have a hash collision and i'm going to be able to call this function which i'm not supposed to call and change the owner of the asset custody to my own address and then i can just take the funds out so yeah that's that's one of the attacks moving on to another attack transaction replay it's again on the custodian and depending on again the custodian the bridge that is implementing you need to present proofs when withdrawing assets so this kind of markle proofs they get issued so let's let's go step by step the first step is that you deposit an asset to the custodian and in the second step the debt issuer sends you the debt token we've seen that

on the original slide with depositing then withdrawing you do burning the depth token to the depth issuer and in this case the depth issuer gives you a proof that you can then go and send to the custodian and say here's the proof give me my money give me my tokens the the mistake in this case was that the verification of the proof uh didn't take into account one of the fields so the attacker was able to change one of the fields and resend the proof again and again it was a valid proof but it was different from the previous ones so for each replay the attacker got the same amount until it drained the custodian

going on to the depth issuer so in case of the depth issuer the attacker goal would be to bypass the signature verification right and arbitrarily mint these tokens right so the attacker what what they did in this case it was again a very specific implementation of a bridge which received as an input parameter over here not sure you see my mouse yeah it received the signature verification contract address right so this was an input parameter but it should have been pretty fixed but this is the way in which the system worked they were just receiving this as an input parameter and of course this signature verification contract was unique it was a trusted component what the

attacker did they deployed a fake verification contract and they passed that as an input parameter so this fake verification contract said it's always good whatever you do you passed the signature verification and this way the attacker was able to basically initiate these kind of uh withdrawals without or like this disminting without deposits without actually depositing any money in the custodian moving on to the communicator layer there the attacker has the goal of tricking the communicator into forwarding invalid messages and minting tokens so for this particular attack the communicator was watching the custodian and was watching if someone deposits funds in the custodian there's an event which is emitted i'm going to check how much i should emit i'm going to

check the address of the custodian is correct unfortunately for multiple withdrawals or like yeah for multiple it was looking only at the first event for the address of the contract which emitted it what uh an attacker did was they deployed a malicious contract which emits the same events it looked the events look the same except for the fact that it's emitted by a different address the problem with the communicator it was only checking the address of the first event and then only looking for the other events that came afterwards so the attacker basically issued a bunch of fake events from these from this malicious contract and was able to uh drain yeah basically get a bunch of

funds that they were not entitled to moving on to the interface level here there's a bunch of attacks which we're going to look on the first is um you have this situation where a legitimate user here at the bottom has approved a contract to allow them to to basically execute some kind of function call for a particular token and this is quite broad so what an attacker did there was that they just crafted out of this execute call data generic function they just crafted okay transfer from user to me right so they were able just to easily craft this function call inside of the call data which led to of the legitimate users tokens being transferred to the attacker

so the attacker could do this for multiple users who have approved um their wallet uh this this interface spending money from or funds from their wallet in a similar attack where again the user has approved a third-party contract for allowing people to deposit with permission basically here you're allowing someone to spend money in your name it should only be meant for tokens which implement this feature but there are tokens which do not implement this feature and do not give any uh warning as to the fact that this function called failed so if someone calls this function deposit with permit into this contract it's just not going to say anything it's not going to return anything and

the error was in this third-party contract where they did not check the return value so they didn't check that the return value is explicitly successful they just assumed that if it doesn't complain then it's successful right so missing uh return value validation and this way the attacker was just able to say like hey i'm going to deposit all of these funds from the legitimate user into my account and that's how they got away with all the funds from legitimate users who approved finally the network layer this is something that hasn't yet happened but might happen in the future there is this thing called the 51 attack which allows you to reorganize blocks and what you can do is launch such an attack

on a level one chain then you deposit assets to that custodian contract and you receive the debt token afterwards you take that debt token and withdraw those funds out of an exchange or something you re withdraw them transform them into fiat or dollars whatever and finally you cancel that deposit transaction in step two because you now can reorganize the chain right and depending on how long these steps this this withdrawal process takes you could afford to pay for such an attack because one hour of such a 51 attack on the ethereum main net is somewhere between 1.5 and 2 million if there are sufficient funds an attacker might be incentivized to actually pay this for a certain amount of hours depending

on how fast the bridge enables someone to transfer funds to deposit and withdraw so in summary there's a lot of bad things that could happen with bridges we saw things on the custodian layer we saw things on the depth issuer layer i'm not going to go into all of these but basically you might have recognized that these are not totally different from your classical security mistakes like missing input validation not checking return values and so on right the communicator layer is this off chain component which is basically watching and triggering smart contracts on both sides and there's a many things that could go wrong we only went into one or two things in this presentation but

there's a longer list and probably it's not exhaustive there's the interface where basically it's not necessarily bridge specific but there's a lot of things that could go wrong with allowing people to deposit in someone else's name or allowing external contract calls and then finally the network layer there's the 51 attack but there could be other problems as well finally just want to say we're hiring we offer great packages and if you're interested in this kind of stuff finding bugs and smart contracts we'd be happy to to get in touch with you thanks so much [Applause] so do we have any questions for sebastian that we have a runner coming forward so it looks like a lot of stuff you mentioned

is like you said input validation stuff like that that could be remediated with like good programming practices um is my assumption correct that if you do all these correct then the problem is the custodian and you there's no way to do that one trustless because it is off chain so can we ever get rid of and not the communicator because that is option is there ever way to make that component trustless as well or will that always be a linchpin in the entire setup how and there's no way to secure it even if all the rest is perfect yeah that's a very good question thanks so this architecture which we've mainly looked at during this talk

is something which is being used generally right now there are projects out there which are taking a totally different approach they don't use this architecture they use something else where they don't have this kind of trusted off chain component and well those kind of things are not yet in production but they are working towards a solution which does not does not have this kind of um trusted third party which is off chain all right if you want we could i can give you more more details uh in the break and another question if i may um sure for example in in terms of web applications we have stuff like ports right academy do you have equivalents for like if you want to

learn about smart contract blockchain is there any like playgrounds that are easy to get into for someone that is not familiar yet totally yeah there are multiple actually there's things like well um call them offline ctfs because they are always going on right you just have like a playground where there are some contracts that you can just exploit toy contracts and there's like multiple difficulty challenges right and there's like things like you know damn vulnerable linux sure yeah there's something like that for smart contracts or google damn vulnerable ethereum or yeah damn vulnerable defy it's called yeah so um there's there's all these kind of things and also there are training programs free of charge

because there's so much interest in um getting more security experts in this field that people are no longer charging for training everything is sponsored by all these projects which have suffered such hacks and and like the amount that they need to sponsor these academies is nothing compared to the amount that they could lose so everything is like sort of free of charge so you can pick it up really easily okay thanks thank you please come up with your question

hello for someone that is completely new to this type of hacking i have a question how much of hacking smart contracts and and blockchain and cryptos is about actually cryptography and how much is about more of this type of logic books that's a good question um i think probably more than 95 percent is about this okay and probably like five percent is about the crypto because um the actual cryptography which is being used that is being actively researched by many research groups at top universities and many of the world's leading cryptographers so the actual hash functions encryption um proofs and so on everything is based on many many years of research from universities and now we're just like working at

implementing applications on top of that okay so we can say something like if you'll invest more time there learning about solidity you are doing a very better investment that by learning i don't know cryptology something like that um depends right um if you're a brilliant cryptographer and you might invent something amazing i'm not then it might be worth doing that but i think like if you don't want to do that then definitely learning something like the solidity or other programming languages being used for smart contracts is a very good return on investment okay thank you thank you

any other questions

yeah so i have a bit of a controversial question so your company is like doing these audits for the contracts so you're making money by doing the audits and stuff but like i've read this idea that it's something like a bit like a reverse but bounty scheme because in the end there is an incentive for you to keep like some kind of bug in the contract and then in a blackhead form exploited so you know what i mean yeah i do i do totally and so it gets philosophical in everything but like you're part of the industry and what is your thinking about this how this could improve in a way where it's better for everyone

like your auditors can do your job others can do their job and everybody can trust the thing in the end yeah no that's a very good question thanks so i think it's always this kind of trust issue when you're working with a security service provider it's not specific to the space it could be like when you're pen testing a web application right for any company right you might withhold some kind of attack that you can then use to exploit of course like maybe in web 2 the damage you do is not directly financial but it's reputation damage or so on but basically the idea is that what we're doing if we were doing that if we were would

be withholding and exploiting there would be a huge reputation damage to the company itself and it would cease to exist right we are well in relative terms a young company like we've been around for five years and we have a reputation for being one of the best so if you have a like the longer your time span is and just keep in mind the oldest auditing companies in this space are like five six years old so there's no uh there's no others which are like have been around for longer doing this specific thing there's been there's companies who have pivoted from doing traditional security to smart contract audits and they have been around for longer they have a longer track record

but definitely this kind of track record and knowing that the company has been around for longer helps with this kind of trust issue because if let's say you somehow managed to um trick someone in trusting you and you do this kind of thing you you withhold an export you exploited probably in a few months to a year your company is no longer going to be trusted by anyone because your audits always get hacked you know what i mean so that's what i think the way to balance this

okay any other questions okay i have a curiosity question so suppose i wanted to exchange a large sum of money and i knew it was going to be a cross blocking kind of exchange um should i just avoid that altogether or what can i do as a consumer of this entire process definitely get advice from professionals and don't just do it um sort of on your own because there are way more problems sort of attacks economic attacks that someone could use to do an arbitrage on something that you're doing right so if you're exchanging one token for another you have a lot of parameters which might be like the margin that you're willing to lose in the exchange and if

you don't properly set that margin you might end up uh being arbitraged by there's an army of bots out there just looking for these kind of opportunities so definitely ask someone for these kind of settings right and then like this is in addition to all the security issues that might be there right so i'm not saying there are definitely ways to do that i've done well i don't i didn't transfer large sums of money but i've done that myself with no let's say issues afterwards right so it's possible but you really need to to uh be careful because like we've just presented let's say i don't know six seven hacks but keep in mind there are like dozens and dozens of

bridges and some of them have never been hacked yeah exchanges exchanges on their set themselves are bridges as well they're just a different type of bridge they're not from chain to chain they're from bank account to chain super any other questions then all right thank you so much sebastian thank you

so

don't worry you have plenty of time yeah

wrong paper

foreign

okay

is everything under control okay all right so we're up with our next talk um so our conference is called the layers one of the subtitles is layer six and this is really referring to the osi model which is the presentation layer so we thought it would be fun to call the conference layer six it's our sixth year layer six presentation yeah okay so you get the joke um but layer six is really concerned with protocol conversion encryption decryption so on and so forth and when we saw this talk the talk is mime is broken we thought this is a definitely a layer six talk and we have to have it so today i'm going to introduce to you stephan ulrich

he is a software engineer security researcher likes jogging and biking and he's going to tell us why mime is broken and what we should do about it okay so on okay yeah uh welcome uh yeah my talk is today about uh why mime is broken and okay the screen is a little bit off let's see um so uh about me i'm yeah i'm the security engineer security researcher fellow i'm working uh 20 years plus at genoa gmbr near munich my focus of the work is not on breaking things but on protecting what's already broken and i did a lot of firewall development focusing on the application layer i did eight years of collaboration with academia and research projects

focusing on defending against attacks via meal and web and i'm currently involved more in product and research strategies uh yeah i'm back in again it's a company exactly 30 years old now with 360 plus employees in various locations inside germany we are a subsidiary of the bundesliga and we build security solutions for iit and ot and our focus is on sectors with higher security requirements so public sector critical infrastructure structures regulated industry e-health etc and yeah of course if you want to help you're welcome uh the motivation of my research like i said i did a lot of firewall development at the application layer so i had to implement application protocols and the best way or the preferred way to

implement application protocols is to actually look at the standards not many do this actually when implementing these protocols yeah but when looking at the standards and trying to find out how to implement these uh the problem is that these standards are typically very flexible and very complex and they are unnecessarily unnecessary flexible and complex they leave way too much room for creative interpretation so there are lots of edge cases with with no clearly defined behavior there's no defined behavior of protocol errors there's a lot of short instead of must be implemented and they are partly conflicting with previous standards of course all of this conflicts kind of the security as a result are that we have different

implementations which interpret uh especially edge cases in different ways and this can of course be used by attackers so if the analysis system like a firewall interprets the content differently then the end user system like in our case a male user agent uh then we can pass attacks through the firewall so the focus of my research in this case is it's mime mime is kind of a standard for rich meal so what we have today structured male binary attachments non-ski characters uh and uh what i explore here is how to use different interpretations of mime to bypass security systems so these are analysis in mail filters firewalls intrusion detection systems antivirus versus the interpretation by mail user

agents or web frontends i'm looking into bypassing milva detection by content using the icar test viewers and uh bypassing detection or attachment filtering by file name so if we can make the firewall belief that we have a different file name it's not a dot access not a zip then the firewall will pass through the mail there's similar research for http one there's all this ports figure stuff about http dsync attacks which are targeting the server side uh i did a lot of research on the http site too but for targeting clients so sending malicious responses or unexpected responses by the server and by passing firewalls this way this research here uh was primarily done in 2015 to 2018

during a research project but i freshly updated it to make sure that all the attacks still work and they do so um yeah to explore this topic um i created a lot of different mails with different test cases so nearly a few hundred males for continental assists or nearly 200 meals for bypassing extension blocking um these males if exported as a files or mail directory and also as uh packet capture file to check against intrusion action systems and then i checked against several systems like various mail user agents various antivirus and male feather products uh some intuition detection symptoms in firewall i won't mention here because it's not a free one and some libraries okay

a short introduction into what mime actually is in the beginning there was about uh before 1994 five there was only esky there were only ski males these had a line length limit of thousand thousand bytes and there was no kind of structure no attachment or similar inside 1996 we got these mime rfc which defined different aspects of how to serialize um structured information and non-sk information into the original limits so um yeah binary data uh structures data like attachments and of course like i said uh this is a standard which is flexible which is complex underspecified and has lots of room for creative interpretation a year later there were some additional standards one is called

disposition how to specify a file name is how to specify if the attachment is inline or external and there is another standard for encoding of non-ascii characters into structured data like file names or so for some reason they didn't include this in the original one so they needed yet another one which implemented a totally different encoding for some reason okay if you look at the source code of a mail we see the different standards applied here so we have the subject which is an unstructured header field and there we see error of c 2047 so this one which is about encoding characters in the header what we have here is with q is a quoted printable encoding which means

that the characters the non-ascii characters are encoded in some kind of hex value so this equals c dry a c3 means that it's a character hex c three and then we can look in the utf art utf-8 and calling what the second actually means okay uh then we have this kind of mine preamble here which is hidden in modern male clients so basically all male clients we have today it's only for the old male clients so this is everything before the first part because we have here a multi-part male which is defined in rfc 2046 and we have a boundary here defined to split the male into different parts with the final boundary here uh

yeah then we have for some parts we have a name so we can have a name given in this way we can have name given in this way this is actually rc2231 which has this kind of encoding for long file names which we can split over multiple parameters because like i said the uh limit for line lengths in male is a thousand characters and which has this kind of hex encoding here which looks similar to what we have in quoted principle but again it's totally different because we have uh here we have an equal sign here we percent percents and so and yeah for some reason they needed to make everything differently and then we have quoted

printable encoding here for the content and base 64 encoding here uh for the content of the attachment okay i show some selected examples of how this can be used to create edge cases uh which are theoretically allowed or especially not as they are not especially forbidden by the standard but where the interpretation is ambitious so uh for example in this case you can define two content transfer encodings this is similar like attacks in http where we have two content length sellers with different values and because we have two values here it's not clear how this part so this is basic before how this gets interpreted and depending on the mail client we use it's uh gets interpreted in a different way

and depends on the analysis software we use or the library some manage to see this and some don't see this so sorry carter ids doesn't see the attachments the right attachment in this case oops slight variation of this we have one field only but we have multiple values inside again some male clients understand this and just take the base64 because it's the first one no male client actually uh unders takes it it takes the last one but uh outlook and apple mail don't understand this at all and just assume there's no encoding done um we can play similar games uh with the content uh with multi-part male so we have a boundary c and we can

have two boundary definitions and uh the correct one would be a bar here so this is the first one and here's this is as the last one and again depending on the major client and depending on the analysis software we have different results same thing again one field multiple values and here again it switches first what was on this side and apple mill was on this side he has a node switch and it's the same with the various software so we see here see a majesty new which behaves the same as pearl mime tools because it's used inside uh yeah and then uh different area uh the area of encodings there's a b64 encoding uh base64 basically means that

we have three bytes binary data mapped to four byte ascii data so it's already it's taken the first six bit then the next six bit and so on until we have uh have 18 20 24 bit three bytes uh binary and mapses to 32 bits which uh are playing ascii and if you have less than three bytes and we have a padding so in this case we have four bytes here which means we have two bytes padding because we have to fill in uh the last two bytes you get six bytes and here in this case we have two bytes this means you have one byte padding and this padding is done with the equal sign here

and the standard or the rc is not clear in this case if what we have should be encoded in one piece so that we have only a single padding at most a single padding at the end or if you can actually have multiple pieces it suggests that was maybe done this way but there is no short or even better a must and that's why we have different implementations so we have thunderbolt apple mail which just accepts that it's okay to do so uh we have mode just take the first part and you have outlook which i don't know it takes the first part and then says garbage and i don't actually know where this garbage comes

from um yeah and if you have a look at the analysis software also the libraries we see that most of these plainly fail to see what major male clients actually can do but we have more encodings we have quotable printable which is in the standard tool but we have also encodings which are not in the standard but with a which are actually uh supported by some male clients there is this why ink encoding and this is an encoding which comes from the area of youth net news i don't know if somebody still remembers it's some 15 20 years ago this time was it yes very efficient encoding more efficient than base64 and that's why it was used on usenet to

transfer transfer binary data and because thunderbird can still function as a news reader it implements these encoding but nobody else does and no analysis software is able to see this other is interesting encoding is uun code this predates mime this predates why ink too uh it was used in former times uh in plain text meals to include some binary data uh so we had some kind of file name here inside the part and then you had this encoding and then there was an end and this encoding is very similar to base64 only it's mapped slightly different but the idea is the same uh yeah and this is actually a widely supported in coding so we have major

milk clients which can do it and we actually have major antivirus products which can do it too but there are lots of variations so we can have different transfer encoding times vm sometimes we need the begin and then sometimes you can skip it etc etc and uh they all work slightly different so there's lots of room for passing bypassing analysis software and then we have another strange feature of mail which is comments in mail header in mail fields so the standard actually says that one can place combat this is this part with the parentheses around it in several places freely inserted and outlook takes this to the next level it's basically nearly everywhere this stuff

can be inserted and it simply gets ignored so outlook sees in this case this i bar as a boundary all other implementations don't see this bar and yeah clam can see it and there's this firewall which can see us and there is a python library but everybody else cannot see this okay this is about uh content filtering and then we have the specification of file lamps uh two few uh two small examples uh like i said we have this uh rfc 2023 one which defined a new encoding for file names and which defined how that these can be split into multiple parameters uh so there is an index for the parameter and uh yeah i

can split it over multiple source indexes the order actually does not matter which is explicitly specified as a standard and yeah um like once see i can see thunderbird apple mail not all supports this kind of outlook has no idea what the standard is at all and the funny thing if i use a microsoft exchange it even transforms this encoding this write encoding into a wrong encoding because outlook does not support it but i have this another standard this rfc 2047 which was defined for encoding of characters into unstructured male fields and it specifically says that these encoding stuff should not be quoted and should not be used inside of content type of content disposition but

funny thing is most supports this and yeah many analysis software supports us too but there are several variations of the topic so if i use some strange encodings like otf16 it gets weird and i have again ways to bypass firewalls or analysis systems okay so how to apply this knowledge in the practice a small thing i create a mail and i check this mail with a virus total so this mail has inside here base64 encoding uh a zip file and the zip file contains the ica test virus and like we can see there are so the ground truth 38 products in a virus total can actually parse this male and can detect the virus inside so

nice let's see if we make a small change we add another transfer encoding here and we can see the number goes slightly down if we switch the order it's not much different and if we specify a content transfer encoding which is something which doesn't exist like the x66 then yeah it goes down but there are still a lot of scanners which will simply have a heuristic and see this is base64 encoding and will try to analyze it okay small step uh we don't use a single chunk for our base64 but we are actually mult