DevOps/SysOps Security

Mateusz, whom you will probably be able to hear tomorrow at the presentation... let's see what I will come up with for your presentation, Mondrala. 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3. Okay. I'll be talking about security... my name is Borys Łącki and I'll be talking about security. Who knows who DevOps is anyway? It's a great sentence anyway. Hand up, who's gonna admit it? You can lie, that's fine. Wait, 1/5th of the room, 3 or 4 people lied. When you type DevOps and security, image like this appears in search engine - the most popular one. I'll be talking about administrators in a huge simplification, but nowadays you can't say administrator to DevOps because they get very nervous. It's the biggest faux pas in the world anyway.

So in a huge simplification we'll be talking about non-programmers, only about those who provide solutions, provide the environment for developers. At the beginning, has anyone heard about such a site "Sprawdź PESEL peel"? Did you see it? You did. Okay. It was our idea that we did when the PESEL affair appeared. In a huge simplification... can't you hear back there? Good, no problem. There's a slight echo, and that's why I wanted to be a little quieter. We launched this site as a test, a 2-hour quick action such as buying a domain, setting up a site. "Check if your PESEL number was stolen". Of course, when you clicked here, you typed PESEL, then first of all we didn't take this PESEL, or at least that's our official version.

However, in JavaScript we checked if this PESEL validates, so theoretically did someone type a correct PESEL or not. Well guess how many out of several tens of thousands of people that visited this site... With most traffic coming from a trusted third party, meaning people who are a bit more clever... how many people typed something that validated? In percentage. Shoot... 90... 90 is a nice number, but it's not that bad yet. I think if we launched it somewhere like gazeta peel or Onet it would be 90. There was almost 20% in the peak point. Such an interesting story. Some additional materials, those you can watch later if you want to regarding the topic I'll be talking about.

Talking about DevOps I prepared this presentation a bit from a perspective of being very advanced at the beginning. I thought about showing you how to configure Dockers with PKI infrastructure of your certificates, how to put it in the cloud to keep these certificates safe. But when I start talking to people, and based on pen tests we've been doing for years, it turns out that we still lack basics. We lack basics of security and we encounter these problems most often. Talking about advanced things has zero sense today while we're failing at things we've been talking about for 10 years. I will want to point out a few times today, when you're doing this administrative or SysOps or DevOps work as you prefer...

Remember that nowadays the internal network is almost the same, should be treated the same as the internet network. Because to get to an internal network it's really enough to send one simple... oh, Michał Gawryluk published a post in the Słowianie group, thanks. To get to internal network today, we see on such penetration tests where we send phishing, where we send malware, where we scatter USB around companies, that getting to a company is really... Switch off the network maybe? Switch off the network? I won't even know how to switch off the network on Windows. Okay. No, you definitely must help me with that. I'd rather not touch such things, it's your computer, I'll break something. Daily I generally use Windows usually for installing updates and showing presentations because these are two things that work great regarding Windows.

Advanc... I would never get to such a window. Thank you very much. And I will want to remind you to think about security from different sides. To think about security more broadly, to treat these examples I will be showing you here... Not only from a perspective of what you see - I don't know - when you see an image with Nagios... well that means I don't have Nagios, right, I have some other software so I don't have to worry. Translate these examples to your software, to your environments, to everything you have at your companies. These are really things that concern DevOps and that concern SysOps and which in my opinion are most important in a very big summary and simplification.

Today the motivation of attackers... I have this tradition that for years every time I show a few slides about cyber criminals. It's already such a tradition that we all laugh at it. Therefore I didn't hesitate to add a few of these slides here. Wait, something's wrong anyway because we agreed that before the lecture we should drink vodka. Because we as Logical Trust company, I haven't said too much about myself probably. We are Logical Trust company, blah blah blah, my name is Borys Łącki. There's no point in getting into it. Apart from the fact that we sponsored somewhere because we believe it's a great... a great experience to be here with you, and for years basically we've been active at lectures.

We wanted to add something from ourselves. And we thought that this year we'll put a few bottles and speakers if they want to, of course they're 18, can drink a shot before the lecture. This is at a few conferences abroad, we wanted to put something from ourselves and I'm just waiting for that. I wanted to tell you if the vodka is not sweet, but it's too late now. Is it cold? Oh my god... if it's warm then... she's probably not cold. Oh my god, it's gonna be fun. Cheers! Now it's going great. Today the motivation... if I faint halfway, you know, or I want more, let me know. Or when I start lisping and talking nonsense then throw something at me, right.

Motivation of attackers today is that they just want to make money. It's quite simple and I hope you have that awareness. In all charts, in all reports it's said very simply, it's getting worse and it really won't get better in the coming years. I won't show you charts and reports. It tells you like it is today. A few Polish examples. I think you're aware that Polish companies get it too, and they're really getting it on hefty sums. 3 million here, half a million PLN there, as a result of simple, really simple scams. Remember Plus Bank? Everyone associates Plus Bank? I hope so, or rather who doesn't... you can't ask who doesn't associate because nobody will admit, right? You all associate Plus Bank, right? Okay.

You all associate the law firm that was attacked and blackmailed so that they paid money in some cryptocurrency of course, so not to reveal this information. We have this type of incidents. We also as a company that helps clients often, it happens. These are examples you see in news but there are more and more such incidents really. Bank accounts stolen on forums on various Tor and so on, I hope you check such things... to realize that statistically we have 2-3 people in Polish cyber underground who post such ads, who publish these things. So if it's your screenshot you can thank later, no problem. Various databases of various websites, sometimes for sale, sometimes just to show that we broke in, that our ranking grew.

I hope you as people involved and more technical people have this awareness, but I always try to throw these several slides in to surprise. POCKET here, you probably heard about POCKET that it was caught. How it was caught, Adam will tell probably somewhere on some slide on Saturday or Sunday. What are the costs of defects today in my opinion? This sentence shows perfectly what the problem of security is. A 15-year old person exploited some SQL injection bug. They incurred costs in the amount of 60 million pounds, lost data of 101,000 of their users. Well security begins to look like this today, especially in other countries where you must obligatorily inform about incident for example, where you must have insurance for your clients.

And now a few examples related to configuration. A few examples from Bug Bounty, from these programs where you can legally hack and break into various institutions. Some of these companies publish info about it. And here you have such an example, a company creating an internet shop, at some point someone, some of programmers puts their private GitHub repo on a public GitHub repo. With all keys, with all passwords, with all sensitive data that were in this repo apart from source code. Well for such a bug 1,500$ for the fact that someone pointed it out, such things happen, one must be careful. Second bug in a monitoring server, also of the same company. It turned out they configured Google OAuth for themselves and they were supposed to allow logging in only from accounts in this one domain, but allowed all domains.

Someone didn't finish configuring this service, well it seemed okay - we have security, we have tokens etc., but really anyone could register there and log in to administrative part of the service. 3,000$ for this type of again simple configuration bug. One of people searching in SNAPCHAT panel, you associate probably... about those funny videos where everyone shows some weird body parts. This person first of all found access to Django administrative panel, which really should be cut off, access should be blocked. Then decided to try 10,000 most popular passwords for login admin. Well here is a question what the password was according to you? Shoot a few, we'll see if anyone gets it. ADMIN... DUPA... DUPA KROPKA 8... okay, I asked only to find out what passwords you use. Well thanks.

Here the password was RESEARCH, very complicated. Still, it fell in the first 10,000 very popular passwords. 1,000$ for such a bug. Here's a bug from slightly different direction because these few domains of UBER company... The one I took a taxi today and talked with driver, many Warsaw residents use it. These domains were redirected in DNS configuration to external domain that wasn't bought. And the guy simply bought this domain for 3$ and it turned out that he obtained control over domains in which he could put any malicious code. Because somewhere someone forgot about DNS configuration, about updating it, or that order with domains is needed in processes. Over 2,000$ for this type of bug. ATLASSIAN company has such solutions as JIRA, maybe you associate, CONFLUENCE etc., very popular nowadays, used by programmers. Similarly UBER company.

Person who found this bug showed that on some very minor site like NEWSROOM.UBER.COM where there was a WordPress. This WordPress was not updated and he took over this WordPress. Well UBER said at the beginning "but really it's not some critical bug, basically you could take over only these our newsrooms and they don't have special meaning". But it turned out... When he took over this their infrastructure, it turned out that this JAVASCRIPT code from this domain is used in all those their internal ATLASSIAN software. And de facto he could also take sessions of those users who logged in there through that taken domain. So it's not always so that at the beginning when you assess that a bug is trivial, it actually is. It might turn out that as a result of more complicated escalation one can take over for example user accounts.

Here also a bug related to the fact that first this person found additional IP addresses that were used by company while properly combining and iterating. Then he saw such a result, he found such a domain DOCKER.WHINEUP.COM. Well he combined there as he was interested about this message here and he started combining what it even is, doesn't really have a clue about it, started being interested in Docker. Well as a result of a few hours of searching... It turned out that this company exposed instances of its Docker images. And one could just simply download whole Docker images they used by providing appropriate query in URL address. Of course with passwords inside and so on. Here another configuration bug for 1,000$, bit similar to that with Google earlier with OAuth, but here it concerned OPEN ID and UBIQUITI company, maybe you associate with such devices as radio access points mostly, but mainly network devices.

Well they did something similar, configured access to their administrative resources from domains of all users and it was enough to register any Google account and one could totally take over their infrastructure on this server. Well this type of revealing info also appears more and more often, 1,000$ for someone noticing their GitHub was configured so it allows reading their private repos and source code of their domains. Well my favorite example because regarding PORNHUB service, who knows PORNHUB service? Trick question, but I appreciate those who raised a hand, either so brave or so... brave, let's leave it at that. Service with movies, right? With sports movies, let's try to simplify it, because streaming all over the world, probably everyone is already 18. 10,000$ for this bug.

Well very funny thing again because on one of addresses, that was slightly external domain NETREACT.EU, this person found exposed part of Subversion repo configuration. And there she found login and password STEFAN, password 123456. And of course she checked if these credentials work. It turned out this their repo was also exposed to internet network... And one could obtain data including PORNHUB ones, because this higher level service would oversee PORNHUB and various other ones. As a result they obtained access to source code and practically to whole code of site. It was also cool that this Stefan person had +W, therefore he could commit something and wait till they put it on their production. Well but legally according to Bug Bounty such things shouldn't be done, they preferred picking up money - 10,000$.

Must remember to update software, and this is such a mantra that I've been talking about for over 10 years and really little has changed in this matter. We have Chrome which tries to update itself and it's some inglorious exception. But managing updates in companies, in software, at production servers is really bad. Today, slightly thanks to those DevOps, it might change because administrators, sorry, DevOps, SysOps have their environments, have Staging, have Testing, have Production... Where they try to implement such changes in a rather reasoned way. But really on a scale of those tests we do and observe it is a drop in the ocean. Here you have an example of server management software to give many commands on many servers, example of such zero-day that was in Jenkins and allowed to take over infrastructure.

Updates again, remember to update workstation also. All these flashing icons in corner: Java, Adobe etc. Who uses Java? Well Java programmers use Java. However, remember that it's not only servers because as I told you to enter a company sometimes - to their infrastructure - we very often need workstation and we use these bugs and this outdated software on stations. Well here example exactly with remote code execution in development environment IDE, to be able to totally take over workstation. Therefore remember - server environments, but also workstations. Remember that sometimes really weird software that seems like not needing update because who uses wget at server also should be updated. Like here a bug where you want to save one file, but actually as a result of wrong interpretation it is being saved down here - for those who can't see - .bash_profile. So at next logging there's a chance this file will launch negatively.

Often this type of commands like wget, curl are used by administrators for automating certain tasks. Sometimes it is like when you take over one server in corporate network of client, then you watch what happens on it and you see that logs from other server from such software come which is not updated. You put your malicous code and next day you have access to next server, next, next and you take over all servers. Well KINOMAN.TV is probably not existing service. This slide also on one hand funny, that there's a warning with many info, but among others there's password to database revealed here. And this is something that on one hand it's said there isn't much of it anymore, on other hand try typing in Google simply site and specific domain and often you can still find it...

Without even touching a site you can find for example warnings from dev framework, from framework on server app side. Sometimes very useful information is there. Sometimes trivial, could seem that okay - login and password to DB, but database is behind firewall, so it's not useful. In the first moment it's like that. But then as it turns out when you take over other server instance, it turns out that from that other server there's possibility of logging because there's no proper separation on backend level, because nobody worries about security there since it's backend. There you can log to this database from this one server. And it happens very often during penetration tests that such minor information, minor leaks are useful later in tests.

One must manage its data, packages, whole server in a wise way. There was this news where a guy probably was writing a master thesis if I remember correctly, decided that he'll create packages in which names will have typos. He created thousands of different packages where he changed letters and put them to public repos. And waited. He took over 17,000 different servers. Because people installed, people made mistakes, something was getting installed with that correct package for it to work and so on. Such an interesting point regarding managing software with your packages, how you install them, whether you enter prod and apt-get install because something doesn't work, so peace of mind... Or do you have a process that sorts it out, verifies every time that it's what you want to install. Remember about it too. Sometimes I hear: yes, but we have Dockers, virtualization, everything in containers so it's safe. Well not at all.

Because here example of exploit - a bug in software exactly for having separated instances, virtual environments. And bug allowed to exit such env to the managing machine. This software also used in Google's cloud. Of course fix was released, researchers published info. They didn't sell it on black market. But you must remember that you can buy such bugs, and the fact that you have virtualization doesn't mean at all you are safe. You have next layer you must care about. Care about firewall, separation, access, roles etc. Similarly even in Docker, during several years there were different bugs. Therefore when I talk about security and DevOps say "but we have Docker, it's safe"... Be very careful about such way of thinking, because that's not it at all. Don't know if you also associate such a bug among administrators, that a guy on forum wrote that his automated script launched such a command RM -RF and here were two variables.

And it was supposed to clean some catalog and file, but variables became zero. As a result it deleted whole server. And there was a huge storm among admins regarding this post - how it happened, backup and so on. Turns out it was a bit of scam because this man wanted to advertise his company which manages servers. If you launched it using Ansible - because they used Ansible - well it actually won't launch. Because internal mechanism will block it. Turned out very few people in DevOps environment checked if it'll even work. Simply believed. Therefore very important is that you know not only configuration but how your software works. Using such tools check how it works. Bit from stories from crypt, response from Polish hosting. On previous slides of Jakub there was a question whether free hosting or paid. This is answer of paid Polish hosting, one of bigger ones.

Files with backups disappeared for me - they were 2-3 GB each. I click once a month. My blog etc. and this backup disappeared for me alone once. I said okay, I downloaded it, no problem. Second time disappeared, something's wrong, report to them. This is response from them: there was a total lack of space on server partitions. Some script from analysis system had to delete a few backup copies of users. We add one month for free. Okay. I didn't check if they added. Yes. Reminds me stories about BOFH. Yes, 8 MB... exactly so. Well such an interesting thing anyway, right. Some other examples. Friend checks backups, well last backup April 2015. Friend provided outsourcing services for various companies. Pressure rose, stress and so on. Searching, watching. Oh it's old backup server, we didn't turn it off and it just stands there. Fuck me.

Therefore order, the order we hate as IT people: documentation and related stuff, basically knowing what servers we have, how many, what's on them, what versions and who is responsible for it. Well it is unfortunately necessary because without it it'll be hard to make a decision what's important for you, what to secure, what to test, what can be turned off. Next story from crypt. Once friend read somewhere hackers will make a contest who defaces most sites. Well therefore he logged in remotely and set IPtables input drop. In big simplification he totally blocked access from internet, also for himself. Of course it was a server he didn't have physical access to, had to do restarts etc. Funny stories but showing that process of managing how you introduce changes should be bit smarter than here.

In advanced solutions we heard about instances where company handles configuration changes on switches like this: admin introduces virtually in appropriate software. Configuration is sent to second admin, he reviews it, and only then it's sent to device. But that's in one company during past 15 years. Just as such an interesting thing. Well here also friend - as I asked about such cases - turned out guy copied SQL from net without check. And it turned out it blew up the base. Another did update but forgot to put WHERE... so instead of one field - whole base gone. People say programmers wouldn't live today without stackoverflow, just copy-paste and you can be a senior suddenly. Not necessarily even knowing what launches at production servers. Well here was funny backups in move, a few hours lost but no tragedy. Remember when... that's the biggest bug again from a dozen years as we test security: Redundant resources.

You expose a file for a moment to download it. Or you leave database in TMP for a moment because dump must be loaded from backup. And for a moment sometimes means three, four years as we find on penetration tests. We obtain access from unauthorized account, for example some WWW user. Well it's enough to dig properly on server and it turns out logins and passwords to further servers, apps, keys, keys to VPNs with private passwords. Everything. Really. And remember that if you must expose something... make address complicated, some hash, something else than DATA.TARGZ. There are tools you launch and such a tool has for example built-in 300,000 different popular files and catalogs. You launch it, leave for 2 days and after 2 days it turns out there's plenty of things left.

These are bugs about which we talk for years really and it still happens. Something fooled me, here example of configuration bug in server IIS, WWW Microsoft server. There's a bug that if you know how to approach it allows reading first several characters of all files and catalogs. Well if the name is very complicated and long, it might not be useful. But often it turns out it's some VALID F... what can this F be? File for example. Well again, might find some redundant resources for further attacks. Therefore you must know how to configure safe environment. Most popular bug again, exposing for a moment some COPY.ZIP, CSV file with client data. Sometimes it's enough to review such a copy - it was a file from mail app DB from Outlook.

There was 200 MB, we downloaded it, checked this mail for most popular words: password, login etc. Well turned out to other system for example there was a message: login, password. Again, further escalation. And here example about redundant resources. Type in site: error site:intitle: man in the middle, master points this attack is called. Search for some weird extensions, weird files. Worth doing it for own servers from time to time even automatically. To see if we left something for a moment and it was indexed by Google. Example from a few days ago, one client had 2 servers, one was blocked, second exposed. If you manipulated Google well, you could read whole site, and it was a very secret project.

So regarding redundant resources, it's cool to make one day of order. Once a month sit for 2 hours and clean everything you left for a moment 5 years ago. Check your HOME, your TMP and where these TARGZs ZIPS were thrown. Teach your employees this too. It works great. We do it like that, it is burdensome, but we minimize risk. Redundant services, here Nagios was launched for a moment to see how it works. No services connected. They saw what it looks like, decided nothing interesting. But in this Nagios left launched, there was a security bug that allowed us for taking over next server. Care about services, don't leave those you don't use. Update and secure those you use. Sometimes firewall on network traffic side. They poured me mineral water here, interesting if it's water. Sprite. Hardcore. Thanks.

Let's do a survey immediately. Who paid at least 5 PLN for BSides? Thanks. That's what it was about. Because it's from you for you, that's the coolest. I thank organizers that they want to, it is cosmic work. Bravo! I hope next edition - because it's too hot - at stadium. I think we need bigger scale. Then we'll be bringing these bottles on some trolleys, everyone gets a bottle. Maybe it'll work out. Real situation, company secured themselves great - firewalled and so on, production servers in internet. But nobody noticed their servers by default have IPv6 turned on. And it's configured and works. One can connect via IPv6. Careful about it. Firewall nowadays not only IPV4. Do you remember a year ago I was at lecture... told you about APT x 3? Who remembers zero day bug in Quake 1? Good, some remember.

The one cheering found the bug. Mateusz found bug in Quake 1 on penetration tests where client comes and says "show us how to rob us". And we agree: these IPs are yours, this office is yours, these LinkedIn people are yours. Turned out these employees were devoted players of Quake 1. Mateusz who really likes to break what he likes also found quite critical bug in software he uses. First could obtain Quake admin rights, then found second bug for running any commands on server. Server was virtualized only for Quake. But as a result of configuration bugs we went to other servers, and next ones, took over about 30 servers. Finally obtained access from their JIRA to their private internal corporate network because someone in JIRA hid keys with passwords to VPNs.

Finally got to the company itself, and it started with exposed Quake 1 for five players in company. Redundant resources - terrible. Back in March 2016 SQL dump. Everyone has read access. Plus R - most terrible thing. Possibility of read by any user you obtain on server. Second example: wrong permissions. Scripts of Cron that launch from time to time. Script database_update.sh. What's wrong? Everyone can write. You have regular user account, you open file, save any command and it'll launch with admin rights once a day. Password reuse - obvious thing. Took over one server, we get few passwords, sometimes they must break for 3 days on Amazon... And we check other services internally, externally whether people use same passwords. I don't know if there was a company where everyone used different passwords. Probably not yet. Password reuse always used for something.

Regarding defense and what you should do better: password policy, complicated passwords, managers, teach users... protect important resources with tokens. Update systems and apps - already boring, but we'll talk about it for next 20-30 years. Educate users because really if they don't know why they must do something, like having tough password... if you tell them to have tough password because policy says so or because we ordered so as security department, they don't care at all. If you tell them tough password should be so we aren't robbed, to have money for your bonuses... it'll reach 10% but they'll start thinking. Segment your network. It strongly complicates our life. Spares attackers' time. Firewall helps well. Encrypt data and transmission, use SSL. Do security tests, once a month, once a quarter. 3 hours for most important thing on earth. Business won't understand anyway, but close door, turn off phone and do such a review with knowledge from such conferences.

We as a company which provides security services as well, we also do this for ourselves. It's easier to invoice indeed - golden barter. Sit down and test and check. From time to time it might turn out something was left in TMP a week ago. New OSes, order in networks, limit admin accounts - programmers won't agree of course, they say I need access to everything. As real admins you can use benchmarks, checklists which are ready in internet. Check your systems and it's cool from DevOps perspective. In 7-8 years partially people will start thinking and implementing into their automatons. Today they use automated solutions for availability, but this type of approach can be added to daily work. Summing up: redundant resources - do not leave data for a moment.

You will leave them anyway because you won't listen to me. Then weird files and order. Test your available ports. Not a problem leaving a script, even in Nmap once in a while. 2 lines of script which sends differences by mail. Oh for a moment someone set up a service to test something. One person from PKI industry was really excited that Windows are made by other company than doors. Travelled with wife, asked me since I have camera taped: is it true? Yes, everything is true. In Lenovo I heard they gave such bad software... everyone gives same software. Wife: recently on security training they said that 6-character password can be broken online in 2 weeks. Fine. Discussion about whether it's brute force or not... 10-character password the best. If you don't have PESEL or KASIA 123... order, order, order.

Set standard umask, 700. If you already put something, just Chmod 700. We will find it in a few months and think how to take it. Extra materials. Promo codes for trainings. I don't look like Małgorzata Wasia who was in agenda. She fell sick, sorry that it's me. Testing security, we teach how to find such bugs in WWW and mobile apps. If you want a discount code, because these trainings are hellishly expensive... better car costs more, right. Catch me or Mateusz or Adam or organizers for discount codes. Thank you all for your attention. If you have any questions... diploma doesn't matter too much, certificate! AXE? Got a hacker axe? AX-hacking? Correct, sorry I didn't know. Questions? Password reuse vs L-DAP.

Well that depends. Can do L-DAP authentication from one place well, but can also screw it up so much that exactly in Quake case we used it, because for some users L-DAP was configured so it was possible to obtain all hashes in 10 minutes. Always must think about it, what risk, what data, what we want to do, facilitating users. I was recently in office, lady typed thing for 20 minutes, every moment "it logged me out" and she logs in for 2 minutes again. Smartcard, typing password... is it secure? Don't know if idleness was 30 seconds as she wrote something on a piece of paper. Better with specific application. Always recommend reflecting over what risk and what's important for you. How often do you see 2-factor authentication between CORP-net and production?

Piątkoś, still didn't get the joke. I see it more and more often. Especially such institutions that have money for tokens. But still very rarely. Financial institutions, also software houses. Commercial Gmail with free SMSes. Very, very rarely. 2-FA works. Any last question? Thank you very much.