Vendors Weighed and Some Found Wanting: DIY Supply Chain Security

the second entry this is a great event you get the practice coming to the stage okay I'm Yan Kella I will be talking about supply chain security for the next 30 minutes or so uh security can be annoying every once in a while especially to the people when when security people bring bad news to somebody else quite often they are not happy if you're are bringing bad news to someone who didn't even ask for it it could be that they are even less happy so how to turn this into positive so when you do vendor or supply chain security checking if you want to do it in scale there is no time to ask permissions so basically you are doing

it and you're not getting the permission then if you deliver bad news you might be worried that how will they react so this presentation is a story about how to turn this kind of work into something positive and it turns out that it pays off to kick your vendos on the security issues quite many will actually appreciate it the most mature wos have these hall of fames they make they'll make a big deal out of the things that they have had vulnerabilities and they have fixed them and they are thankful for you for helping them to do so I really love how the scene has changed 10 years back nobody ever had any vulnerabilities so so the scene has

changed quite a lot the context for this presentation is that uh in Finland 150 companies a little bit more did supply chain checking for their vendors uh this was a government sponsored project uh it's a finnished word getto uh Tonto is kind of like Santa's Little Helper and ketu means chain so this was Santa's little little helpers for supply chain and companies want to participate in this kind of activities because it's a really easy way for them to check the security of the supply chain and government likes it because the this is a really straightforward way to make your National critical infrastructure more resilient few words about me so I'm a security researchers mostly during some rare nights nights and I'm a

security entrepreneur during the days and I like dull vulnerabilities and I like dull security issues it's because if they are dull it probably means that there are a lot of them it's something that is quite common and when it's quite common it means that there is impact to be made if you can eradicate these kind of dull vulnerabilities you have a chance if you are doing that you have a chance to make a big difference in the world so uh I'm not trying to trap you there a QR code if you don't want to ask any questions today please do show in the linked INF for example so why should you care about this presentation so let me check uh do

we have any bounty hunters on the audience today okay a few do we have security and it people who um are worried about the supply chain of their own company or thinking things like that okay quite a few all right then one more random technical check uh how many of you know what is um subdomain take over okay thank you now I can calibrate my presentation okay so this is what your kind of people and lastly if there are anybody who has a bigger ecosystem more things to worry about rather than just own company just then this presentation is for you also because you might learn how to improve Security in scale so what did we do

uh there are ways to check the security of the vendors or take care of the security of the vendors most of them I'm been told and I've heard are annoying for the vendors you can buy vendor risk score Services they will scan the vendor and rate the vendor based on uh what they see how many vulnerabilities and stuff like that uh if you don't like that kind of approach then you can do other stuff like you can push vendor questionnaires to the vendors you can have 100 question how are you doing this and that and that drives people mad it's very labor Ro and then of course a little bit easier approach is that you just push

security requirements to your windows but what there's a problem have let me ask this this question have you ever seen a company who says on the paper that we are not doing security we are not that interested about it no we don't care about audits yes everybody takes security very seriously on paper but when you start testing them then you start to see a little bit difference between different companies so we wanted to make something that scales something that you can do in a big scale but it would be sensible it would not be annoying hopefully the vendors would actually even found it useful so I will go through few steps how you can do it this yourself maybe

you are a buck Bounty Hunter you want to do things in scale maybe you need to do this for the supply chain of your own company and so forth so step number one map your own external attack surface and one thing what is quite often forgotten is that vendors are also your external attack surface not just your own infra so you need to consider them as well so there are non technical methods list the ones that you remember you are working in a company you are using different systems you already know a list of vendors then go and ask around go to the procurement what stuff have we bought in the last one year if you are a

bigger company and so on and run around different business units what are you using and stuff like that this is a really easy way to involve people in security they can help you and maybe the security people are not the ones who are always blocking and preventing stuff now now you are doing stuff together so involve the people in the company and then there are of course technical me ways to do it just list your domains uh start checking the txt records from the DNS uh what have we authorized to operate under our name stuff like that and do a external attack surface mapping for your own company and then once you have done that then check that okay

these IPS where are these These are these cloud services hosting companies so on and so forth and then you will do external attack surface mapping for your windows so this is like external external attack surface mapping so you can use your favorite OS Recon tools sub finders amas whatever you like there are tons of ways to do this nowadays so if you don't know any with a quick Googling you will find plenty of those if you don't like tools if you want to understand what kind of actual things you need to do uh you need to find subdomains of the uh companies whatever they are using you can use for that certificate transparency logs if

you have passive DNS at your def disposal you can use that and so

forth then it's time to find the vulnerabilities uh at this point you need to be nice also in this method if you want to do it fast if you want to do it for a big amount of companies you don't you cannot be asking permission for it because permissions take a lot of time and quite often you don't get the permission so you need to find ways to do this check in a way that you can be ethical but still have some relevant results so you can do your list of things this is what we have done so we check for example take over about subdomains so no not many hands were raised when I asked so I'll quickly

explain what that means so this happens usually when you start using some cloud service you set up a service to the cloud and you will Point your own domain name to that cloud service or maybe marketing does that or whatever and then at someday in the future that cloud service is the commiss and the uh DNS record is not removed so at that point an attacker can go to the cloud service and set up his own service there and ask for that same name you used to use and at this point the criminal can operate under your domain name they will use it quite often to something like sharing harmful content to the internet or uh I've seen uh

internet casinos being set up to a domain of a company and so forth but they could also if they are smart they could circumvent the browser protection by injecting JavaScript and attack the services inside that domain so that's take uh suban takeover problem then you can look for end of life servers I'll talk a little bit more about that in a moment and you can check for expost services for example example uh only the worst ones I would advise and what is surprising that for a trivial checks like these one out of seven companies will typically have something to fix so that's quite a lot for even for a simple check like this so as a

result uh with this approach for the vendors of 150 companies approximately about 2,300 vendors uh we reported something like 840 issues of these categories today something like 25% of them are fixed 7% of them uh the vendor said that okay we have other kind of protection this is not the relevant issue for us and 68% is not fixed at the

moment okay a few words about the end of life services so how do you do that pretty straightforward you do the same thing all the vulnerability scanners do but at the first part don't do the second part so you check the version strings but ignore the typical thing that the Verner abilities can do they check the version string and then they uh check that what CV is what vulnerabilities this version has and uh that creates a lot of false positives so if you would be sending reports about these you would get mad uh recipients because you would be sending something like 11 vulnerabilities to them and then they need to go through them and then

they figure out that okay none of this is true so skip the cve part but look for the strings that tell you that the server is already so old that it's not getting any security patches anymore again simple check like this you will find quite a lot of them you will find old uh Linux distributions you will find Windows systems that are older than my teenage daughters and so on then a few words about the export services so 1.6 trillion connections later uh you find quite a lot of open databases so there's almost like a 100 if you count the my sqls and Maria DBS and postgress and so on and so forth so that's quite a lot but again

again how far can you go without permission what we did is we just check that is this database visible because the visibility of the database really often is a mistake already we do not try to lock in and steal the data that in our opinion requires permission so you find quite a lot of these then you find different remote Services RDP is the way way most uh common one and I was wondering that do people keep RDP open on purpose but uh looks like in most of the cases it's not because they do close the ports when you report them uh then there are bunch of other something I would like to highlight is 40 gate management that this is also

something that we find quite often and it's not a good idea to expose 4ate management to the internet because they have had quite a lot of vulnerabilities uh in the past and it's quite likely that they probably will have more of them in the future so even though they are patched quickly uh you just cannot share it to the internet Because the Internet will will break in quite fast all right that's about finding the vulnerabilities the vulnerabilities are not the main point in this work work it's really good that you find vulnerabilities and they will get fixed but actually the main purpose of the work is to get the conversation going so now it's time to get the

conversation going why do you want to get the conversation going if you want to measure what's the maturity of that vendor in cyber security you want to understand that how do they deal with security issues when you report one vulnerab this come and go what's more important in my opinion is that when you inform a vendor about the vulnerability what will they do will they ignore it will they fix it fast will they fix it slow and stuff like that so you need to have the conversation so let's make it happen here's the problem very few of the vendors are actually prepared for receiving security reports uh in our case 7% of the vendors did have Security txt on Place informing

that this is this is our policy and this is where you should be reporting and stuff like that a little bit more had the web page uh about the security policy so let's be kind let's say 15% are prepared to receive a report everybody else will be quite confused when they get the report uh about the security issues people will think that you are back Bounty Hunter you will be extorting them in no time uh people will think that you are a salesperson you are trying to get uh the customer to a sales call oh you are some kind of scammer anyway I don't know what's going on but this is some kind of scam oh it's a bot

I don't need to care oh whatever reason to think you are irrelevant so we need to avoid this so there are few ways you can minimize this annoying fact there are some ways that you can easy more easily get through with your reports so uh if you use email templates for your reports it's strongly recommended that you do for the efficiency uh also have a hint of human when you send the email so like for example we we have this standard automation creates this part uh but there's always the person who is sending the report there's some kind of human analysis of the situation few words doesn't need to be much but something that uh people will understand that

there's actual human sending it if I would see just a report like that my brain will just shortcut it's a marketing email and I will just skip it then the second thing is you need to explain the context because people we think that you want money and stuff like that so you need to explain that why you don't need the money okay you could be lying yes but fun fact people are interesting if you are on a big line on the airport if you just cut everybody off people will be more mad than if you cut them off and you say I'm sorry I'm in a hurry then it's totally fine so you need to be explaining

yourself that helps a lot another thing is don't withhold any information withholding information will put you in the same bucket with the uh bu Bounty be sorry be bounty hunters and the salese I have a lots of people complaining about I I it's 2023 uh salespeople have used like to my knowledge at least 13 years this tactic that when will dig up a vulnerability and then then we'll contact the potential customer and and we get them to the call where we really wield the vulnerability and maybe they need to pay a little bit for a product to get to know that vulnerability so that just drives people mad I don't know anybody ever has gotten any sales with that

approach so if you are setting up a startup or something like that and you have the same idea uh let's inform even if you give out free the vulnerability uh to the company it's really hard to get to the sales conversation afterwards so you need to find other ways contacting people then try to be helpful be available for questions uh you can say that you are available for questions you can say even that you can call me 0.00001% will actually call you so it's a easy promise but it will change the tone of the report quite a lot and then after you have reported you can be helpful so 95% of the time that's just

being a little bit annoying hello I did a routine checkup and Uh I notic that you haven't fixed the issues yet can I help in some way because quite many will forget the first report they some of them won't see the first report some of them will see it but they forget some of them will see it they will forward it to somebody else and that somebody else will forget and we people are funny we just forget stuff so some followups will raise the likelihood that the issues actually will get fixed then if you actually get to help there are things like this vendor they don't have any security or it people themselves they are relying heavily on

their vendors so you need to be talking to their vendors how what is this about and so forth and so on and sometimes you need to help with the risk assessment uh I will give you one anecdote about that in a moment uh after a few slides then if you're doing this for yourself you don't need grading but if you need to communicate about your work to somebody else to your boss uh to the procurement department or what whoever is the uh person you need to convey a message then it's a good day to good idea to create the companies you can have your own grading this is what we use so we have a simple

ABC uh sometimes I wish we had more categories but then it will get because this is not exact science so having more exact numbers is not a good idea after all so we do a means that the vendor fixes issues fast and communicates properly then B uh takes the issue seriously but the fixes take time we will take a look why that is in a moment and then see no answer and no fixes Sometimes some organizations end up into this category C while they in principle they wouldn't deserve it maybe it's a false positive maybe your report is a false positive but then on the other hand uh we have had vendors who thought that our

report was false positive and when they responded we were able to work that out and the vendor realized that it's not the F false positive so I would think that uh a good vendor does communicate with the reporter so I don't feel that bad putting them into the C category as well so right now the numbers uh of that getto uh campaign uh in a category they are 80 B 59 C 57 so as you can see they are pretty even uh if you are mathematician and you calculate this all together you will notice that it's not 2,300 which is the am amount of vendors we have been checking we have few of those still uh unrated and they will go

to the BNC category and based on experience uh vendors will go pretty much 1/3 to each category and now it's really hard for me to decide that is this good or bad is um on the other hand A and B care about security and on then on the other hand B and C are not able to fix the issues quickly which is sad so I don't know if the class is 2/3 full or 2/3 empty some anecdotes and stereotypes in the ABC categories of companies so in the a category we have the pro not much to say about this fixes the issues really fast communicates clearly basically DS we are going to fix this in three days

then they fix it in three days and that's it then we have something I love a Relentless Hunter so especially with the big organization it's sometimes difficult for them to find who actually owns the system that needs fixing so there was this one company for example in Finland really large company uh they had an open database on a service it was hosted on some service provider but it was under their domain and they couldn't find the owner for it they the security people they went to different departments and do you own this and do you own this and nothing happened and then they went to the financial department and asked that uh uh who is who is paying the bills for

this hosting service tried to find the owner that way and then at the end of the day they went and went through some really old ticketing systems and find the owner from really old tickets so they are Relentless Hunters like that they do not give up until the case is properly dealt with then I have a Estonian bonus do it in scale first time I uh I saw this in Estonia uh I was doing a security audit it's already I guess something like 15 years ago so I probably can say that it was for xroad and uh we we had to do a lot of work to find vulnerabilities but we did find for example

one and I was prepared to fight that yes this is a vulnerability and stuff like that uh in the meeting when we started talking about that vulnerability the person who had been implementing the xroad said that yeah yes yes uh I already fixed this it's the last night's build I went through the whole code base and I found s similar vulnerabilities from seven different places I have fixed them also so I was like I've been a fan of estonians ever since uh then category B the slowest vendor there are many many reasons vendors are slow one reason is what I just told you they just cannot find who owns this system marketing department somewhere uh has bought something uh

this is another uh region of this multinational company and so on and so forth sometimes uh I have heard many times we have forwarded this report to our sock in Canada they will be dealing this and that almost mean that's the place where all the vulnerabilities go to die it it just never get fixed but uh for example this example I'm thinking about very much later on those got fixed so so let's not be too hard on them then we have the live in the moment vendors these vendors are building things fast they're coming up with new services but nobody actually stops and thinks that what's the life cycle of this service especially they don't think that

what happens when this is the uh this nobody uses this anymore so what's the end result is that the services will be run indefinitely so you find something like inter some service built on internet information server version six on Windows that has uh long time ago it didn't get any patches anymore and so forth and now you have a service running on IIs 6 you need to move it to the completely different platform you have no idea who built it and stuff like that so it will take time then on the C category we have silent treatment this is probably 90% of the C category no answers no fixes that's quite straightforward then we have there is no

problem most common one we inform that you have an exposed database and they said no no no it's not a problem we are not using that database anymore and then we got but you did use to use it right yes so there might be some data in it I guess so I don't know what's the thought process behind that this is not a problem so this is the most typical one then there was one for the subdomain Takeover for example uh this was really painful for me it was a vendor who is doing lots of stuff for the Finish critical infra and uh they had obious subdomain take over vulnerabilities we had seen that the domain where they were pointing

to were actually being taken over in practice so we knew that this is a real problem and then how to fix the subdomain Takeover problem you go and remove the broken DNS entry from your DNS server it takes something like 5 minutes so really easy to fix and a problem that you have seen that uh is being exploited so then you get this message that we have done a risk assessment okay what was the result of that risk assment then you don't hear anything from them anymore the issues won't get fixed months after months they stay there and nothing gets fixed only after the customers of that vendor here about the same issue that they have then it takes two days to fix

the problem so somewhat there is a bunch of there is no problem vendors out there and then the last but also the F worst so our our it is so broken that we cannot even take your report in you will get from the email system something like quota is full or or something like that so I guess the subdomain Takeover problem is least of the worries at that point so how much time this takes if you're planning to do something like this it's not that bad uh again if you're doing this for your own company you can here we have actually two categories so 120 minutes per company who you are doing this so if

you're doing this for one company your own then count two hours for listing the vorss then rest of them depends on how many vors you are going to take a look so validating vendor about 5 minutes um what's validated the vendor if you have bunch of these vendors already you might have some vendors with different names multiple times so you need to check that uh do I have duplicates is this actually a vendor or is this just some brand and the vendor behind it is something else so you need to consider that kind of things and so on and so forth 5 minutes for that for each vendor then the external external attack surface mapping finding the

vulnerabilities that depends on your automation I don't have a number for that we we on our case the automation does all of that so it's close to zero minutes then the sending the report so you need to find the contact who to send the report to uh you need to verify that the report actually makes sense and stuff like that so 50 minutes for that at first it will be longer 30 minutes or something like that but once you get the hang of it it will be faster and then facilitating the a grade vendor well that's pretty simple send the email out uh they will say that we fixed this and then you congratulate them they great job so not not that big

of a dusk but for the B and C Windows you need to be following up uh you need to be checking that have they fixed they haven't communicated anything but have they fixed and that takes a little bit of time so in summary you can check your vendors in scale uh and still make it so that the work is sensible you just need to pay attention to some small details what we have gone through then finding the issues is actually the easy part uh you will have content with what whatever checks you will made you will just do or 50 vendors and a bunch of them will have some issues so this is not a humongous amount of work but yes

you need the automation part for that and then finally remember if you do this your work will be appreciated so please go out there and help your vendors to fix the security problems thank [Applause] you so amazing any questions oh here we go how many honeypots have you found during the vulnerability scanning Etc zero that we know of I thought we found one honeybot because uh there was this one reputational security company and we have found a um old service from them and they were saying that no we don't have any old services and we said that yes do and then they said that no we don't and I started wondering that hm maybe this is a honeybot and then I one

final follow up uh and uh this is what we see when we connect the SSH to your server and this is the version string and then they come at oh oops actually we do have some old services that uh they had had they actually were running a upto-date deban but they had been bringing some own custom SSH from old Debian and they had brought the binary as well so it was running but I was this close to giving up that this must be a honey but uh yep in the back there and then there is one more question this lunch was good I say yes please uh in the reports I saw you grading the issues as low medium and

high yeah um if you complaining about the C great companies I think it was uh that uh people don't act on the issues as it's uh like not in the risk assessment uh should we be like not grading issues in our reports to say these are all issues please fix them yeah that's the main thing to do I I agree with with you about that uh and the stuff we send uh to the vendor there's no grading in that we give the grade afterwards on how they're performed so we don't say that please fix this this is a c type of issue we sometimes we separate uh again we want to keep the vendors happy and

positive because we don't want to cause any pie of vendors uh so we do say that we try to be honest yes this is a quite small risk right now but why wouldn't you fix this it takes 5 minutes to fix it and then you don't have to worry about it so so we do sometimes say that this is not such a big issue but the grading actually happens after they have acted so so

yeah uh could we get the microphone to the loudly lady next to the camera I wonder how many times out of the cases where you get no response your report email has just gotten stuck in some Microsoft spam filter yeah that's a good question I'm quite tempted to put some tracking inside the emails but then again there's nowadays there's also tracking evasion by the Microsoft so it's a very good question we sometimes we if we have time we try multiple different contacts also but if it's from the same organization maybe we hit the same spam filter all the time so hey but what um sometimes the last follow-up email we sent that we haven't heard anything from you uh and you

haven't fixed the issues this means that you will be getting the worst possible grade C then the spam filters actually fix themselves magically yeah sometimes not all the

time y there's a question right there in the corner ah hello thank you for an insightful talk I have an like suggestion not so much a question uh in in the data privacy sphere there is more or less an panan requirement that companies must have a working contact in case somebody wants to have some issue with them should it actually be the same with security the companies are mandated to have a security contact which is responsive that requests that would be great uh I fully agree and actually we sometimes use the data privacy officer so if you cannot find any sensible contact from the companies we go to the gdpr paperwork and see if we can find

any names from there and then we say that hey you have export database this might be a uh data privacy issue so can you forward this but for the security that would be lovely and it will make sense also uh I have a little bit of time quick story We once reported the vulnerability for a large organization uh and we had been telling them that you should have a security contact in the past they hadn't implemented that yet uh back then we wrote explo also to demonstrate the vulnerability we don't do that anymore but long time ago so that report got got passed inside the organization from some people to another and uh at some point

it ended up on the table of some unfortunate PE person in us the person didn't have a clue what to do with the report so he posted it into some news group that I got this kind of report that can you help that add it content exploit for the remote explod for the firewall they were doing so it makes sense for the companies to have a security contact who can handle this in coordinated

manner all right any other questions there is indeed the traffic advis those 10050 plus small and medium Enterprises to drop those ATC these vendors um the jury is still out uh on that we haven't completed the project completely so let's see what happens on the C graders sorry I don't have an answer for that but um um owners of the the companies of those 150 they will learn which of the vendors are great C and then they will have constructive discussions with the windows and that's actually a good closing Point what I really love about this it's the it's actually on the meta level so it's not that much about the vulnerabilities uh it's a little bit

more about how do they fix it but the most important thing in my opinion is that we are keeping the vendors on their to they will be they recognize that somebody actually cares about the security uh somebody watches for the security and somebody expects that the security is in good order in companies so looks like my time is up I thank you very much for the questions lovely audience hope to be here sometime again thank [Applause] you fantastic so as you can see ladies and gentlemen uh the previous sessions have helped organizers to unlock a secret that people who make a presentation get a present don't worry everybody got one just not everybody on the stage but we're going to try to

continue with that uh thinking about informing people it always reminds me about the lovely story from Estonia where they were sending an email but they didn't know to whom the email address belonged so we found out too late that vaccin should not have been kept in a too warm of an environment because it might be bad for them but emails got sent now we're still looking for the guy who owns the email address it kind reminds me of registering my first email with hotp h you can deduce the first part yourself I was 12 so it's not not that difficult it was hot and heavy as one might say yeah I still used it in law school

though they were very happy to send the seminar materials to sex at hot it's still in work if anybody's interested but moving along