Who Watches The Watchmen

Welcome! Thank you! Welcome! One second. Something didn't work. It's been waiting too long. Maybe now it will be better. What am I going to talk about today? I will not tell you anything about cyber, AI or machine learning. I will also not tell you anything about red-teaming in Poland. I will try to tell you a few words about privacy, bugs and bounty related to them, and a little about Responsible Disclosure. Who am I? My name is Marek Szustak, I work in a company called Eskai. I've been working in IT-seq for the last few years. You can find me here on Twitter. What I'm going to talk about today is just my opinion of my employer.

So, who is this title guard from my presentation today? Well, he is basically a leitmotif that will only be seen as a background today. I will talk about these services or service providers of some kind of specific analytical tools that advertise themselves with such passwords.

What does it have in common? It has a unique functionality that Google Analytics doesn't have, which is most commonly known as Session Replay. As you can see, this market is quite densely packed. There are many providers of this type of functionality. Some of you may know at least the biggest ones. In this market, there is even IBM with the TLEAF service. The most popular is, it seems, in terms of online appearances, Hotjar, but others may also be known as Yandex.Metric. Where can you find such analytical services? I don't know if it will be visible on the stream, There is a project called Web Transparency Project, which showed in 2017 that in Polish Internet Yandex scripts were for Cineo, Hotjar was for Quake,

there are players like Monet, Demotywatory, Spiders Web Orange, Rzeczpospolita, BZWBK. To refresh the data from last year, I asked about the HTTP archive in BigQuery. And it turns out that the hotjar, which is the most popular in Poland, is in the Polish Internet. You can check it out in the source code of these services. static_hotjar.com is loaded there, which means that the Hotjar is probably running on these services. Here are a few screenshots from yesterday.

I don't know why, but Senat, Biedronka, Wośp and Home are using this service. What is unique about this kind of services? They offer an exceptional feature called Session Replay, which I will show you in a moment. But why did I even get interested in this topic? One day, one of my colleagues told me that something strange was happening with our application on his developer version. For some reason, his browser started digging coins. The source of the problem was CoinHive. Most of you are probably well aware of it. The source was the mouse that we used in our service. However, we have implemented in the transaction service, i.e. where you buy our airline tickets, well, not our airline lines, but if you use our services, we have

implemented Content Security Policy, which prevented loading any scripts that we did not include in the whitelist. This is how it looked from the perspective of our analyst, who collects reports about anything that tries to load on our websites, which is not allowed to be loaded. This peak meant that this script appeared in the 15-20 minute window, and then it disappeared. We don't want such things to happen in our services, so we decided to take care of it. What you see here are signals, because most of the AVs are blocking CoinHive. The next day, in the AV reports, we noticed that most of our employees who walk on our websites also got a kick on our website. We contacted the authors

of Maustac, asked what was going on. It turned out that we were not able to break down much, but they confirmed some kind of security breach, they confirmed that they had a mine in their scripts for a while and it appeared because there was an unprecedented push of change in their production system. We don't like to cooperate, and even more so, we don't like to integrate anything in our systems that is not of basic maturity, when it comes to SDLC, so we decided to turn them off. And look for alternatives. which would allow us to integrate directly with the use of content security policy, so we wouldn't have to do reverse engineering of their scripts to determine what resources

they may charge, and they will immediately send us ready-made rules. And if possible, we would like to use Subway Service Integrity, i.e. a technology that allows you to sign scripts loaded from the server's supplier with a cryptographic hash. Thanks to this, in the event that the script on its server changes, for example, by byte, this hash checked by the browser would refuse to load anything. Another thing we decided to do is to audit such systems. And that's what I would like to tell you about today. The first thing we decided to do was to verify how these systems work. The integration with them is very simple. After registering with them, we get a piece of the script that we install on our

website, as with Google Analytics, which is loaded into the viewer of our client. What does this script do? It registers the entire structure of the house, or the entire HTML, which is loaded into the client's browser, notes down all events taking place in the house, monitors the mouse position, records it and all the events. However, from the perspective of the owner of such a script, the owner can open such a session later. And how it looks in practice, we will try to show you now. It looks more or less like this. So the owner of the page has the opportunity to watch what the user has seen on this page. It could be said that the owner of the page is standing behind the back

Okay, not to prolong it, it is probably due to some consequences associated with privacy and how these services try to cope with this privacy, if at all they try to cope. Here is another demo, in which it is shown, let's say, what happens when we visit a service where we can leave some potentially sensitive data. Here is a specific example of using the FullStory service. On the right side we see what the owner of the page sees during the preview almost live. On the left side is the monitored page on which this script is installed. So the user is typing on the left side and on the right side we see literally everything. A careful

observer will notice that the credit card number is not reproduced here, because as you can see, the owners of this website, that is, the owners of FullStory, try to ensure at least a minimum of privacy and security, so that data about credit card numbers do not flow to their servers, etc. However, not all providers are doing their best, which does not mean that as a website owner, when you would like to integrate something like this, you cannot do it in a way that provides basic privacy security for your users. When you install the scripts you get on your websites, the users try to filter out the data that they can leave in the selected form fields. Some do better, others do worse. So,

as part of the audit, we decided to ask ourselves the following question: What if a person who is angry comes to my page and before the monitoring scripts start, he will slightly modify my page by adding something from himself. Will this something be registered from himself? And if so, will the owner of the page see the same thing while watching this session? So the question arises: if the owner of the page, while watching this session,

will be exposed to the risk of losing cookies due to the fact that it is verified in the Hotjar, SessionCam or other service. We decided to check it. The simplest thing that came to our mind was to add something in our service. At this point, we start Bulb. Before loading the page, we paste one tag "ing", which in "on error" starts a piece of JavaScript. And what happens? It turns out that these services were not designed with such problems in attack vectors. As part of this demo, I try to show that if I insert a piece of JavaScript into the page and display it, something may happen to the Hotjar service. The question is, will this thing be launched in the viewer's

browser? If you noticed it in the payload I put there, I add a piece of JavaScript that corresponds to reading cookies, local storage from the browser, serialization of this data and setting up an additional resource that goes to the third server. When the owner of the site decides to reopen the session, The attacker receives a request in which the exfiltration of the data I mentioned, i.e. the session cookies and the basic data from the local storage, which from the attacker's position allow the session to be taken, which I will try to demonstrate here. The appropriate session cookie is set, where I copy the value, set the information from local storage, which for some reason turned out to

be important, so that this session could be taken over. It was in the base, so you had to decode, refresh the page and I'm logged in. Let's move on. So what are the limitations we are talking about here? Of course, it is a XSS storage. The simplest payload that could ever come to mind works as much as possible. Business Impact is also a session takeover. Because the provider runs, as later turned out, Bug Bounty, we passed our observations to Hotjar. They invited me specifically to their Bug Bounty program and within this cycle on the Hacker Ones platform we solved this problem together. However, Hodjar knew about this problem, because, as they admitted, they had recently carried out a security audit

of their system. The reward was slightly reduced in terms of the program, but it was, which is a pleasant gesture on their part. Another question was whether other features related to registration may have similar problems. As it turns out, Hotjar had a feature that enabled installing a budget responsible for collecting feedback on your server. from users visiting our website, which allows us to show the elements we like or not. And as part of this feedback, I don't know why, although I know that in a moment we will talk about it, they send the entire HTML of the page on which the user was. And again, I had an idea, I had an idea what would happen

if I added something from myself in this HTML. And as it turns out, there are interesting consequences. Well, adding a piece of JavaScript Finally, on the side of the Hotjar panel itself, there was a reflection in the image that the user was watching. Because this HTML was not displayed here, but the image. However, this JavaScript that I typed, it was displayed in such a way as if a regular browser rendered it. As a result, it turned out that it was possible to find out the address of the resource in the browser. Later, it turned out that the resource was also publicly available. And when I went to the address that was in the screenshot, it turned out that I

was getting the same HTML that I was sending there. What does this mean in practice? Well, you can formulate the HTML you want, send it to their server and it turns out to be hosted under their domain. Fascinating. because it allows you to do exactly the same as I showed you on the previous demo, i.e. to print a piece of any HTML, we can host any page on their domain and run the same payloads that I showed you before. Again, the same cycle, as you can see, it is considered a duplicate by them, okay, I will not discuss it, sometimes it happens. From the perspective of the viewer, it was possible to notice that when a piece of HTML is sent as feedback, the

user watches JPEG. How does it happen? As you can easily guess, there is some Chrome Headless or other solution below, which loads this HTML and renders it so that the website can be displayed in the service. What can you do with this Fanta? If you discover that such a service is running on Amazon, if we look at the Amazon documentation, where we discover that there is something like a unique URL address, under which metadata is available, and the service is activated, then you can do interesting things with it. If we insert a piece of JavaScript into the payload, redirecting the browser to the address we want, it will turn out that the browser will render something it has seen. And going on,

you can learn about this machine many interesting things.

Among others, you can find out what are Security Credentials, security tokens that allow you to present before Amazon as an application running on their service and get access to all resources to which this application has access. So we are dealing here with Information Disclosure using SSRF type of liability. and safety keys. As you can see, I was not the first to show this error, but due to the quality of this report, I also received thanks. As I showed at the very beginning of the presentation, the number of providers providing similar solutions is large, so we had the question in our minds: were so considerate to solve these problems. As it turned out, not especially, exactly the same trivial payloads

in their services were triggered and not in all, but mostly with an analogous business impact, because there are solutions that allow you to do what they want to do in a safe way. However, we do not meet here just to show that in subsequent services it was possible to run exactly the same payloads. I would like to show you something more interesting. Well, if you think about it, when you would like to copy HTML from any page and display it to a third person, then often this HTML in the viewer of this person will not be displayed well, because third resources may not be available at the time of viewing such a session. And in some way, such services must solve

this kind of problems. And they can do it in two ways: to load the same resources from the original servers from which they were loaded or simply do the processing or caching, download and save these resources within their services at the time of recording right after them or at the time of display. And what does it mean? It means that most of these services They somehow download resources such as CSS, JPEG and all other resources that your website potentially links. They often build services such as proxy, in which they take into account the parameter of what resource it is, where to download it, What I'm going to do, I suspect most of you already know. Well, as it turns out, it was possible to

use this type of proxy services to, in the case of services that have a slightly larger number of security solutions, such as content security policy, to serve any content from their domain. In this particular case, it shows that when the resources were being loaded from their servers, The client session was secured with Content Security Policy technology, where they allowed to load resources from their domain only, not from my domain. It turned out that using such a proxy endpoint I could load any content type, here in this case TextJS. A bit not very good content type, but the browsers are doing great with this, because there is a so-called legacy code sniffing, which is run by most browsers if they have no idea

what content type the server is basically sending them. Therefore, it was possible to run JavaScript within a full story replay session. And here it turned out that ... Yes? One small question. Did you install the PHP version there? Yes. I can tell you about it in a moment. Anyway, what you can see here is that this endpoint proxy transmits a set of headers from my server. I thought that somewhere in their network there was something like that. No, at some point I started experimenting with the fact that I can. Because making such a request meant that this request was sent to their server. Their server asked my server, downloaded content and passed it on to the server. I could start experimenting with these

headings. Here, the PHP is not very fresh, but I could use this observation in later tests. Once again, I had the pleasure of talking to professionals, because they had the Responsible Disclosure policy. I sent them my observations by e-mail. They thanked me, here is our Responsible Disclosure Program, these are the rules of the game. They considered this error as a high priority and thanked me. The solution to my problem was to close CSP rules and introduce X-Content Type Options NoSnave, which should prevent me from using the proxy endpoint as an attacker. send anything to the browser because their endpoint was default filtering the most obvious types of documents. That is, it did not allow javascript to be processed, it did

not allow HTML to be processed so that it was not possible to smuggle potentially malicious things through this process. And this is how the timeline looked like. As you can see, it can be solved even high priority quickly. So full respect. So what does Pentester do when he gets the information after two days that the problem is solved? He gets in and his goal is a retest. He would like to confirm that he solved the problem. But what I observed was that not all It was good to filter the content type that came from my server. In specific cases, they introduced the X-ContentTypeOptions, which in browsers turns off the functionality that is responsible for the browser to guess what kind of payload is sent

there. However, as you can see, I was still able to send HTML text. I could formulate any payload. So I started experimenting with what I could do with the no-sniff that was added. As it turned out after many, many attempts, when my server responded with a large value of the same header, i.e. it looked more or less like my server, the end point proxy suddenly He rejected the header and said: "I don't know it, I won't proxy it because it's too big". And in the same way I got rid of, I turned off the mitigation that they implemented, I got rid of the header, thanks to which I could once again serve any JavaScript. Another

problem was that I couldn't import any HTML file to my browser. to the module related to the playback, because they had implemented proper sanitization, filtering of malicious things that could be triggered. So, for example, the iframe could not be injected, the script could not be injected, which is obvious. However, when during recording I changed the size of the SRC attribute and the sign. Everything worked. And thus the evil payload coming from their server. Because I used the procs endpoint, which I had to do due to CSP. Again, so the CSP bypass had place. It turned out that everything can be started again. A report sent to FullStory, omitting CSP using ContentType, the correct one, turning off XContentTypeOptionsNoSniff and another bug

and another bounty. The timeline is similar, but even faster, because I send a report one day, the next day the correction is already in production. So what do we do next? Next time when the problem is solved, we check if all problems are solved. We start to combine everything that comes to mind, whether it is a characteristic and only working in Chrome Link Rel Iimport, which allows you to load resources from any server, iframe, objects, embeds, everything was filtered. One variant of embed was not filtered. According to the W3C standard, the old tag embed for setting resources on the page should always have the attribute type. This type of filtering was done in FullStory, i.e. it checked whether the

element that is inserted into the DOM was an Embed object, whether it had a type attribute and whether it was HTML text. could be opened, but for some reason the viewers have no problem with this type of element. And again, another error, another bounty.

In this particular case, something was wrong with the communication, because after sending the report, they thanked me, promised to take care of the case. After a week or a little more, I asked myself if something was already known about it. They answered that it was a work in progress and after another two weeks, the correction was in production. After another check-up, I was not able to find anything at all. Another service that we talked about earlier was Lucky2Orange, where the proxy endpoint was located and using it you could ask for the content the same as I showed in Hotjar, i.e. the metadata endpoint in AWS, where, as you can see, you could read the contents of

this metadata. And once again, access tokens, security keys, which are often used in metadata, sometimes scripts responsible for the EC2 machine setup on AWS, can also be found. And using such a proxy endpoint, In the U parameter, we can assign the address of the page we would like the endpoint to proxy. Here we have 169.254.169.254. Typical problems with SSRF, because the Lucky-to-orange case is so cool that in subsequent iterations, the report, the solved problem, the report, the solved problem, I showed them how to implement mitigation. When I showed them that the IP address 169.254 should not be read, they made a blacklist and added it to it. It turned out that a simple eight-digit notation allows you to bypass it. domain demand, which is

solved by DNS, for exactly the same address. Also, later, the use of redirecting, i.e. indicating the endpoint that makes the directing of the usual HTTP 301 to the same endpoint, worked again. And likewise, we did about 4 or 5 tours, mitigation report, mitigation report. which was distributed in a few weeks. Is that all? When it comes to the type of endpoints that are processing, and which use mitigation or security based on blacklists, or content type blacklists that are ready to be processed, I think not. What I discovered relatively recently is that for some reason one browser, Firefox, has a support for a very interesting content type that no other browser knows or understands. Content type is multi-part mixed. So

something that usually appears in email clients. For some reason Firefox understands it. So for some reason, it can display one content as HTML and the other as a link. So entering one page means that the browser will display something and suggest the collection of additional resources. If such endpoints are using the technique blacklists, they probably won't have anything like that on the blacklist. However, within the document itself, I can redefine the content type of the individual resources that can be added here. So maybe there will be more discoveries related to these endpoints. Something for developers that I would like to bring from such a presentation. Namely, if

If you will have to display content from another server to your user, use sandboxes if you want to do something like that. iFrames have a mechanism that makes it impossible to run any active content even if it is in the iFrame. Use the Content Security Policy. It is a perfect tool to solve problems with third-party resources, but it is a decent tool. Remember to use sanitizers or filters. I do not recommend writing them yourself. It is best to use what is well known and comes with the framework from which you use. I would like to invite bug hunters to present their presentation. Our friend from the bank presented a very nice presentation on Friday. He talked about Responsible Disclosure, and Bug Bounty.

If you would like to start a bug hunter competition on HackerOne or any other platform, Back Bounty is a very popular game nowadays. You can test it and do what you want. Before you start any test, it is worth to spend 5 minutes to get to know the content that the authors of these programs, the owners of these services, provide and proceed ethically according to what they expect from you. Do not touch the best data of other users, do not hurt anyone. However, from technical advice, in the case of XSS and reporting anything in general, remember to emphasize the business impact of this error in your report, because you will therefore put on the plate of the person serving on the other side of

the table the dish in which he will be able to consume, what it means to me, what the problem is, In the case of SSRFs, it is worth taking care not to expose your metadata, especially from the cloud, or even more from the internal network, because this type of vulnerability is very nice to use for scanning, for example, the machine on which something works, or the machine standing around and services related to it. If you want to start playing on the HackerOne platform, you can find dozens of programs publicly available. After registering and learning the rules of the game for a given program, you can start testing many services. As part of pentests or other

activities, if you find problems with providers, which are sometimes more difficult to reach than because they do not provide such an open bug bounty, you may find support from Trend Micro, which is running its Zero Day initiative, where they will focus on almost all, well, not all, but a large part of from researchers and the mediators in solving these errors with vendors. Their business, Trend Micro, aims to gain insight into potential security errors in the world today. creating certain signatures for their products such as IDS, IPS and boasting that their solutions can mitigate problems that vendors have not yet solved. Let's say there is a Polish side, so I would like to ask you, can you

drop the camera? The Polish side is not here, the Polish side? The question was where we can report a security problem if we find something like that on the Polish side of some office, maybe honestly I have no idea. If I were you, I would probably start by looking for contacts with administrators of the domain, somewhere in FHUIS or some other channels. I found technical people standing probably somewhere behind the service where you found the error. From my own experience and from the attempts I have made here, it turned out several times that making such a contact is not a trivial task. Finding It is difficult to find competent people on the other side. In the

case of Western providers who have developed marketing activities, it is possible that a good idea is to look for someone on Twitter, Facebook, For example, you can contact some of the public channels and ask them to give you some information. Sometimes it works better, sometimes worse. But from my own experience, I can say that it's better than worse. Typical methods of using email are often thrown into the deep. As I mentioned, it is worth being ethical during testing and not harming anyone, follow the rules of the game. It is worth doing good notes. If the tests take a few days, for example, and it often happens, it is worth doing these notes, because ultimately when writing

a report about the quality that is worth taking care of, It is worth taking screenshots, collecting logs, making notes, anything that can be helpful to the person standing on the other side, who will be responsible for solving this problem. When it comes to reporting, it is worth being patient. Many mistakes that I had the pleasure to report were solved within weeks, months. You should not be greedy. Many services that do not run bug bounty may have responsible disclosure or may not have it. However, if they are ethical, they should report their observations and should not expect that You will be necessarily rewarded with thousands of dollars that Google or Facebook boasts in their bug bounty.

Because many providers, many internet services are modest enterprises sometimes, where such an enterprise is one or two people, they do not have large budgets, and even more so, as you can see, on security, so you should not be greedy. You should appreciate every gesture, as often as in the world of backhunters there is a t-shirt or anything else, a cup. These are gestures that indicate that, or like a diploma that the bank gives you, can give you. These are gestures that indicate that there is a second-hand understanding of the whole situation, but there is no budget to thank you enough. And if you get some dollars, you have to settle it with the tax office. Are there any questions? No, it's

not difficult. Even without having an economic activity, only one declaration is made. You don't even have to do it right after receiving the money, only once a year with the sum of the amounts you earned. If you have economic activity, it is included as an additional income. Often, if In the case of FullStory, which you may have noticed, there was a point where I showed that they asked for an invoice. Yes, they asked for an invoice for each bug. I run the business, so I put out such an invoice, I put it into the accounting, I calculated. This is not a "ropiet science". If you have additional income, there is one additional declaration, which you have

to calculate once a year with the treasury.

Do we have any more questions? OK, thank you. I invite you to the SKI website. We recruit. Thank you.