Automating SAP Security Patches

good afternoon I hope I'm not gonna talk in a way that gets annoying because of this sound I'm glad to see so many of you back after the lunch I hope you guys can stay awake otherwise I will throw some stuff into the audience first of all a really short question because I'm used to speak to audiences who are quite familiar with a zippe already but I have the feeling that not everybody knows as ap maybe can I see some hands who knows as ap as a as a vendor okay that's already quite comforting any of those people are they are you working with sa P also in your company can I see some hands of the people who work with

as you P in our company that's already less alright I want to make this as interactive as possible so guys if you have any questions just feel free to shout or scream or whatever yeah well I'm yours a short introduction I work with a zippe for quite some time already I did an implementation for the Ministry of Defense like 11 years ago and by then I started doing more into security specifically related to s AP and I also do SP security related research and found like 70 over 70 filner abilities in SP software that sounds like a lot and it is if you know how simple they are to find you would be amazed yeah

first of all the head off to the Luxembourg hackers they already captured our national flag like I don't know two hundred years ago so that's some next-level that capture the flag well done guys okay a short introduction about a zippy it's one of those monolithic big enterprise applications that in general people don't want to touch because if it breaks it might break your company quite often it stores all your business critical data it's used for automating those processes that are really in the core of your enterprise like logistics like seals like production like whatever their range of functionality is really broad they have over three hundred thousand customers worldwide and they say that of every yeah 74 percent of the

world's financial transactions touching as ap system somewhere so this stuff is huge it's not that visible always within companies but the bottom line is it's an interesting target because it stores everything about everyone about every thing the company does in one database often a short history introduction on sa P they have a lot of background in the financial world and in big corporate environments they are therefore already like four years and big parts of the code base that they use nowadays where the current systems is already being developed back in the 80s and the 90s in a time where security was not that relevant or important yet they did something about that so over the past

couple of ten years they really really improve purity for customers that do you installations of sa P but all the customers that are already running on as a B for like 10 20 30 years they tend to have all these old legacy and backlog in their systems that they just can't rid of quite simple the thing is like I mentioned as we did a lot to improve security but quite late that's because they they missed the boat so to say in the whole internet connected world like you know the big companies like Google already picked up quite early and and they also worked a lot on security like Microsoft and Google there are years ahead of sa p as we try to catch up but

they are quite late we do a lot of ESPE security related research we also present about it there are some other specialized companies in the world to do that if you search for security conferences where as u P topics are discussed that's quite a lot these are just some examples maybe some people here have been to troopers before in Germany Klaus knows that it's a security conference with a dedicated SP security track which is quite unique this was my presentation lot here at HEC a lure also about sa B so there are researchers working on it and SP itself is also doing a lot of code scanning and stop on their code to improve things this

resulted in so-called recipe security notes and as ap security notes are let's say the same as your Microsoft security updates these are small bug fixes small patches to solve a small specific as it be vulnerability there's over 4,000 ah better and this number is increasing each month as ap releases like 20 to 30 s AP security related notes on the same Patch Tuesday as micro school as Microsoft does Microsoft and as it be as SP has aligned to patch Tuesday to the Microsoft release date something about the current state many s AP customers are now aware they should do something but that awareness does not always lead to action many SP running companies are still

really really lazy and slow in applying patches because this SP system is often so critical for them that they are quite reluctant and afraid to touch it the companies that do work on their SP security are often a really big fortune 2,000 ones that do have a lot of money and people and knowledge to do so yeah and I want to mention that shortly because the GD P R was already mentioned by Elena before I think also for hospitals that will have quite some impact the same goes for these SE p running companies because they often process privacy specific data as well so a lot will be needed to do to work on in that area this is a list of 12 of the

most common vulnerabilities that we found over the years we did as ap security related assessments already I think we already do that for six or seven years and there's quite some high risk stuff that we keep seeing keep keeps see coming back all the time I will not go through it in detail because not everybody is aware of it but a lot of these are really specific to sa P so you can see this as you can compare it a bit with let's say the wasp top 10 but then specifically for sa P and I guess a lot of those things will not say too much to people from outside the SI p world but but what's the most important one

which come keeps coming back on number one is missing ASAP security notes some missing patches and that's what I want that's the thing I want to zoom in on today yeah this is it this is related to a question I get a lot because people say yeah well we don't hear too often that sa P running companies are in the news because they got hacked well there are there are definitely examples the thing is I think in in in real life the number is much higher because by default as a PE running customers have all the logging switched off so most customers wouldn't even know if they got hacked and this is a this is an example from maybe two

years ago where you sis which is a private background investigative company in the u.s. they do background checks for the American government and they were hacked as the story goes by the Chinese it's not confirmed by the way but yeah you know how those things go they got hacked anyway and a lot of important data about employees working for the American government those records were stolen which ultimately led to the United States government to to stop the contract with this company with this particular company which ended up in this company going bankrupt and they had like a three billion dollar contract with the government so you can imagine that this was killing for them and it

got them out of business so this is quite a good example of things that can go wrong yeah well in general I we see this more is that in general typically patches that are provided by fenders are not always applied soon enough a good example is to Equifax heck of a couple of months ago which could easily be prevented by just applying patches the same goes for sa P systems they don't apply patches for several reasons very important one is that they don't apply patches because that those systems are so business critical that they need to touch it test it thoroughly and if it breaks anything it might really bring your your complete business to a stop so

what we see typically is that they just do a large upgrade every maybe year or every two years and in between they do nothing nothing at all so it's like it's like your Windows operating system for example it's it's similar if you would say well I have Windows 8 let's wait until Windows 10 is out then I'll do an upgrade and in between I'm not going to install any security patches that's not good but it happens a lot we see that typically and another important thing to mention is that as if you releases those security patches on a regular basis on the patch Tuesday the code is quite easily reversed can be engineer a reverse engineer the code is open it's

it's on there marketplace you can download the patches and from there you can easily reverse engineer what the vulnerability is so that makes it extra important to make sure that you patch well those were the problems I want to go and look quickly to some solutions what we what we hear a lot from customers why they don't apply patches also is because it needs to be done manually those patches need to be one-by-one selected you need to click on them you need to approve them you need to activate it it's a manual process that takes you a couple of minutes per batch per note the thing is if a zippe releases like twenty to thirty of those notes per month and

you have like twenty systems which is quite common then you talk about 400 batches that you have to apply manually this is really be a big pain for people so what would really really help here is if that process could be automated to a certain level it's also good to mention that those as ap security notes are they have a rating like a sieve ESS score so you don't always have to apply them all you can also only pick the ones that have a really high risk yeah as an example that's what I want to show because what we typically see is that customers have like hundreds of those notes missing even one one of those

notes already one of those notes could look do a full compromise of your system I do have a short demo of in this case a privilege escalation via such an SCP security note what we will see in the demo is that you can get ASAP all rights which is the equivalent of of route online acts and we will do so via sequel injection

hmm it's running yeah what we see here typically is an irregular user which only has specific rights to test function modules like API is in SI p system and what we will do here is inject sequel sequel code and this will give us a reference to the dduk user the Didache user is like the root user in and this can be easily achieved by and doing an update in the u.s. our user a first table and by then afterwards you will see that our user is connected to the dduk user which means that we take over the rights from this dduk user which means we have full control over the SE p system and by then you can do

anything you can download all the tables you can create your own users you can do whatever you want and this is this can be achieved easily by just one missing su P security nodes where most customers have tens or hundreds of them missing as mentioned this is a lot of manual work this is also stated by the official documentation of se P itself where they state well you have to implement every security node manually in every death test and production transport landscape and if you have many deaf systems you need to do it in every def system what we did for a specific customer is we we were asked if there wasn't an easier way

to do so as ap itself does not provide an easy way to do so so what we figured out is that we could automate 50 to 75% of the implementation of these nodes there are several varieties of those notes you have really simple notes and you have also notes that have a dependency of other notes all the ones with dependencies we cannot implement automatically with the single notes we can and because it's a quite straightforward easy process what we did because as if he does not provide any api's or any function modules or any tools to help you with that so we had to fall back to a really old mechanism in SP systems to basically do some sort of

macro recording where we record transactions and then replay them over and over again with the specific notes that you want to implement I do have a demo of that so to automate that we created an above program which is the programming language of si P it's hard to say but it will probably not not say too much to you anyway the thing is these are the screen recordings we captured by first applying a note recording all the steps that we did and then replay it afterwards again for each and every single security note that you want to apply so when we execute that program all you need to provide is a transport number which is a mechanism of a GP to

move changes throughout your landscape so all the notes that you apply will be put in a transport and you need to provide the list of notes in this case just nine but we did tests with hundreds of notes as well and it works it works really well it will only run for quite some time but the thing is you can do other work while this process runs in this particular case it's hard to see with there's out of the nine I think six are implemented automatically which is 2/3 of the work that you don't have to do manually and in this case it's just six nodes which you can imagine if it's like a couple of

hundreds and you have a lot of systems that will definitely and this will definitely save you a lot of work yeah the business benefits yeah it's obvious it saves you a lot of manual and very boring work and how buddy wants to do this believe me yeah and also what we saw discussed at this customer is that they raised the frequency where they applied patches because it was it was not so so much work anymore they could quite easily say okay instead of once every three months we will do it every month it saves you time and it's good for compliance here as well yeah concluding well especially with and that's also what we see a lot is more

and more customers tend to connect RSVP systems to the internet because they want to you know communicate with a GP using partners vendors they have interfaces to the government for example employees need to access those systems from home so they are being internet connected way more than in the past so risk is getting higher GDP is coming so it's really up to customers now to start working on the security of the receipe systems and tooling like this might help yeah and I said even one single missing node can lead to a fully compromised system so they really need to step up and yeah the good thing is a lot of this work can be automated that's the that's

the the biggest part of the presentation I hope it was quite clear although maybe the tooling itself wasn't that clear yet it's good to mention the tooling itself it's not we cannot publish it like open source tooling because it's as ap specific coding however if there are people interested in using the tooling I can quite easily help you setting up something similar so if there's interest for the tooling please let me know afterwards and I'll send you some documentation that the thing is we cannot publish everything online because as a piece not always too happy about it and to make it work you need to do some it's not it's not like SP has its own propriety

protocols and stuff so it can be quite a pain to set up but if there's people interested in this stuff I'd be happy to tell you more about it and give you some good pointers and parts of the code yeah if there's any questions please do let me know I'm gonna grab the mic thank you very much have you been in touch with the ASAP to fix the remaining patches that are not installed automatically

yeah thanks yeah I'm in touch with a GP all the time about a lot of things I'm not sure if I understand your question specifically because you are talking about patches that are not installed automatically yes yes the the thing is that we can apply the ones that are quite simple the security patches there are security patches that have dependencies with other patches and then you can have a whole chain of patches that you need to apply it can start with one single patch it has a dependency with one that has a dependency with three others and then it can end up like having to implement 20 or 25 notes starting with this one single note so

for us at the moment it's not yet possible to automate that because the things we can automate are predictable but if you have all these dependencies for us it's it's hard to predict how many notes will come out of that so it's it's just not possible yet the thing is we were hoping that sa P would come up with dueling as a vendor itself to make this process more easy they don't they won't I don't know for what reason I guess it has to do with legal stuff because what I see P says is you have to read thoroughly each and every single note and if you apply it you have to really click a button where you say

yes I have read this note and it's like a legal disclaimer that you have read a note and don't hold a zit be liable whatever blah blah blah if you automate that stuff then you're not seeing that anymore so I guess it's a legal thing as you P itself does not provide any tooling that they leave customers out there struggling with this because every customer I speak with they have the same issue they're not applying these patches because it's just too much work and for now it's just the simple ones maybe in the future we can also automate the more complex ones but that's a challenge we have for the future to find a way to do

that in that case I think that they say P might be easier to serve is something like an XML schema that they will have drawn up all the dependencies and they can update once they add new patches I think that's something even even micro systems as a vendor has so s AP being around for for long enough it's soon it won't come it won't compromise on anything on their sides giving a map of dependencies but that's of course big bold words but just throwing some ideas there

can be mapped but that's a lot of work and and yeah you say well even Microsoft has that yes but you cannot compare those two all always it's quite hard and si P does a lot of efforts I must admit that they do a lot of efforts to improve things but the hard thing for damage that they have to do it is this backwards compatibility which goes back 20 30 40 years and that's quite hard for them so but yeah that shouldn't be an excuse to not do anything that's the point and yes dependencies can be met we are working on something like that but that's that's still under investigation but thanks for the feedback

anyone else with question I think you are faster than your second need to need to make order thank you for the presentation when you are talking about your customers about the risk but they are answering you because typically those SP systems the open fully open on the lan to be able to connect so we can imagine and even with one infected pc this could lead to big issue including trademarks that are sometimes protected in acp systems thanks SP systems they store so many business critical data and it ranges from intellectual property to HR data to vendor data prices or stock relevant information stuff like that it's all energy P systems and yes even ok let me put it differently as a P

systems used to be connected only internally in the lung for employees that was the that was the case in the past and what we also see typically is that the ports to connect to SP systems for example with an SP GUI which is like a big fat client connecting through the server those ports are almost always open from the user Leung to the server on so there's not too much defence there the thing is and I mentioned that earlier to some other people here as well many big companies they do have security teams general security teams with what we see in real life is they look at everything except as ap because SP has their own

propriety protocols like the SP GUI protocol and RFC protocols to communicate between each other so your regular pen test team most probably will not touch an NCP system most of the times and it's like a separated world and it's really strange because SP systems are really business critical but nobody's watching at it so yeah there's a lot of work to do and you're right yes often they are really exposed in the network sometimes also from the internet but a lot of times also from the local law and if you have a big company where tens of thousands of people are working worldwide yet they can often quite easily get access to an SP system in an

intended ways yes thank you I was curious your experience even once you've put in the patch you go from development test testing and for production environments or oil and gas specifically that would from my experience we would do that for a long time before you been moving to production what is the average lag time you see before they actually implement it on the production side you have things that's a good question yeah the customers that do patching yeah yeah what's the window of exposure because before something is patched in production it's typically quite long let's make a distinction between I mean there's a lot of customers then don't even apply those patches regularly so they can be vulnerable for months or

years the customers that do apply patches what we see there what is quite normal is that as soon as the patches are released on Patch Tuesday they analyze them the ones with high vulnerability they apply they put them in a transport they move them to a test environment and what we see often is that it's not really tested explicitly but more tested implicitly so they just leave it there for let's say two weeks if it doesn't break anything in the test environment they just move it to production so it could take several weeks in that case before it is in production you know anybody else question so related to that one how often is that you apply a patch and it

breaks something Plus how is it how easy it is to roll back from that state when you just broke something and okay you quickly need to roll back because it's a critical system there yes another great question and very relevant because you don't want to break stuff in my experience it doesn't happen too often maybe just to give a number maybe one patch out of a thousand will break something so it's really uncommon for that to happen in case it happens and you send it to production and it breaks something what you're not gonna do is restore your whole as ap system because then you go back in time and you lose business transactions and money and that's never

gonna happen so what you can do in that case is go back to your development system d implement the note there so roll it back only this specific change in the code for that specific patch put it in a transport and move it throughout your landscape in that case you will fix it in production you will go back to the old let's say vulnerable state but at least your system is working again and then you can find other ways to fix it yeah well what we won last edition what we also sometimes see although it's very rare but people fixing it directly in production with all kind of crazy code don't do that because that's a no-go

putting some oil on the fire so ok if you're prepared then it's fairly easy to well restore a state where your system still gonna be working properly ok so in that case based on what you said and if you roll out automated just it's really just like this legal fear or you don't really have any information on by sa P is really shying away from doing this more automated ok apart from dependencies but the ones that you can just deploy and it doesn't occur because even now with with Windows patches okay we have sometimes problems with it blue screen of that is really not that company anymore so it's hard to hide behind the plots if something is if

something is broken that how fast it turns out that it's broken you don't need to wait for a week so what you explained with the test system that they wait for a week for nothing or two weeks for nothing does it even make sense so do you need to wait that long because they are like if you have a business process that is only tried touching certain transactions you need to wait long for for figure out that that one is broken so this does this because in in most Windows environment they do the same thing they wait for two weeks or three weeks and it doesn't make any sense because yeah if it should turn out

pretty fast that you have issues there yeah to answer to that there are other strategies as ap did release on tooling that you can use to analyze what functionality is hit by a specific security patch if you do use that you could implement the notes test those specific functionalities only those and move on to production the thing is to set something up like that I'm not sure what the exact name is for that functionality but they do have tooling that can see where what functionality is impacted setting that up also takes a lot of time so I don't see that too much being used in the world ok time for one last question I think a lot of these security problems

around sapa are connected to the complexity of the total system there's so many different platforms on s AP applications and so on front end middle end and back end servers and web facing services I read a lot of blogs about ACP security a published by the company ERP scan and they also do a lot like you like work in in finding s AP vulnerabilities and writing about them and honestly it seems to me that every company that has s AP just need to get fished once and their total totally hacked they are literally wide open for that for a movement and resistance so if they were Oracle they would claim that they find and remove most of the

vulnerabilities themselves and that it's not really a risk to their customers but they're not Oracle so they have not really claimed this but but how can that just the customers justify the risk of course they can't live without sa P so it's a business critical mission mission critical application but shouldn't it be a way to do that more securely

what what we see is that the companies that do work on patching their systems hardening their SP platform on on you know mitigating stuff that's that's really a small group that's not common but it's it's the group of most big as you be using customers that have money that have resources but don't forget there's I mean I don't know the exact numbers but let's say it's maybe 20 80 percent there's another 80 percent of customers that don't really do that stuff those are the ones that are vulnerable and and yes it is like you described um as a people not say and will not make a statement like like you mentioned and and honestly they do a lot but it's

really a shared responsibility the customers also need to work on this and as long as they don't pick it up I mean SP can create patches as you P can make white papers as if he can do everything they want if that's not being implemented by customers it's not gonna happen so I think we will see in coming years more as you be related blogs and stuff like this and hopefully not too many public hacks but yeah it's already happening and I think it will need to get worse before it needs to get better yeah yeah yeah well there's not sure I discussed this before I did some investigation on the Internet where I used masks and to just scan the whole

ipv4 range worldwide to see what a zippe systems are connected and what ports are they're open and what versions are they using and yeah there are so many that like literally thousands of azp systems I mean I didn't even try out the default credentials because it's really really Annette achill I guess but I think if you would do something like that and it's easily crippled you you could you could hack a lot of lot of companies fortunately we ran out of time thank you yours for your presentation give him some clap [Applause]