Encrypted Client Hello: What Does This Mean for TLS Connections?

all right I feel like a microphone right it's the first time I use one of these um right so um I've let everyone's looking for where shark and TLS and stuff like that this afternoon and not not expecting to be in the other room uh so who am I why am I talking about this so I've spent kind of the last 20 years working at an ISP uh specializing in education and we were having to do lots of filtering and so spending lots of time looking at TLS and trying to understand these changes for TLS and what they're going to mean for us and yeah currently work as a cyber security engineer for erm um so why

um why is this this change happened that people are looking at improving TLS security so people remember this photo taken from when me Edwards Snowden Revelations came out they caused people to start thinking about what metadata was was around and how they could encrypt it better um so brief history of SSL and TLS so Sni was introduced with TLS 1.0 um and kind of became a thing in 2006 when um ie7 and Vista came out so people have got used to the idea that this Sni information's there and and it's and it's in clear text as we'll come on to you can see the other versions of TLS have come out afterwards but they've left Sni alone TLS 1.3 made some changes

but it still left us an island so what is Sni so server name indication so it's a plain text header but when you browse to https site it's there in the clear for anyone to look who's looking at their message using a um you know TCP dump or whatever and it's in the client hello negotiation and it's been used by lots of corporate firewalls and nation states as well so it was built for one purpose which was hosting lots of websites on one IP address but it's had alternative uses by the security Community to find out what's going on on the network and I've kind of put AB around me abused because the writers of the Sni standard

the people on the TLs committee don't think that those are legitimate uses of the Sni information it's obviously a different opinion between different people um and yeah the key is once you've encrypted DNS with DNS server https then the Sno information is the last playing text piece of information out there um although the IP address you're going to is kind of in the clear as well but again cloudflare and other people have solutions to it so um how's it been used by nation states so as an example South Korea moved there filtering from DNS to Sno filtering um and there's a one of a large Indian isps also started to do filtering based on Sni so there's there's evidence Sni

is being used at a nation state level to do filtering and this is again encouraging the people making the standards to try and fix this problem so um who is making these TLS standards so assuming most people and probably have heard of internet engineering task force and rfcs but there's a group of people from different companies so it might be from Google from Apple from cloudflare working together to come up with these standards and get them ratified and then adopted by multiple companies so they're often doing it in a sort of outside sort of outside of their day job but they are also still representing the their employers um and you get lots of rsas some of them

have been have been created as ifc's real numbers but never actually implemented because they're kind of jokes or um or whatever and there's others like um born at the top which I'm sure nobody had a backup we had to see but the rpz1 that's used for DNS filtering which never made it Beyond being a um a a proposal because you know the committee didn't want to agree to things that promoted censorship so stuff just because it doesn't make it as an IFC it doesn't mean it's being used and also just because it's in an RSC it doesn't mean it will be used so you have to view it from that lens and one of the

ones I spent many um afternoons trying to think of a network authentication required one that was in one of the rfcs and nope no captive portal actually does it so it's in the RFC but nobody's ever implemented it um so what happened with TLS 1.3 so in TLS 1.3 lots of people probably will remember when it came out there's a lot of Ferrari about it stopping people and understanding what was going on on the network and what they did was they encrypted the certificate that comes back from the server so the Observer now can't see the certificate that's coming back but yes and I had ever went in the connection message is still in the clear

um so it reduced for confidence as to what somebody was going to but it still left a pretty big clue as to what somebody was going to um and kind of removing that Sni header is an unfinished business so the TLs 1.3 people um and so yeah they encrypted the handshake um and one of the things they had to do um is put this middle box compatibility mode in so this was because when we tried to release DLS 1.3 they had all sorts of problems with people with various firewalls not understanding TLS 1.3 and crashing and so they they put in a way of making it look like 1.2 2vs what they describe as middle boxes so

that they don't crash oh and still process message and it added to LS 1.3 adoption so again while they've been working on encrypting Sni they've also been working on the same principle they don't want to break existing networks and we want to make it easy for people to roll this out so whatever people's aims with this and it started off as encrypted Sni and it's now become encrypted client hello so you'll see in the rfcs it it mentions both names kind of yeah but we um but yeah for new name is encrypted client hello um so yeah so remove this last piece of metadata don't stick out and make it hard for somebody in the middle to

downgrade the connection so if a client and the server both agree that they're going to do ech then nothing in the middle should be able to force them not to do ech your corporate firewall shouldn't be able to prevent it if both sides have agreed but it's a good idea [Music] um and the aim is really to be using this with large-scale providers like cloudflare so it makes it difficult to just say I'm going to block that provider because you're blocking thousands of sites and so um Chinese government might do that but lots of other governments would would bulk up a prospect of blocking cloudflare um so why is this hard why hasn't it been

um uh implemented already so the challenge is you've got to send this Sni information before you've negotiated any keys so you need a mechanism of doing that and you need to be able to support cdns that's a thousands of websites before Sni came along you know one IP address to one website was fine that wouldn't be fine anymore and they want to try and make it not stick out um and they want to ensure that they don't break existing Network appliances um so added it work in TLS 1.3 already so you can see here the the plain text headers come through and then you've got your encrypted certificate going back so the um and here's an example of a header

then going through the um server name in the um in the Sni so it's very much in the clear set of Wireshark you can see exactly what's happening um so the first attempt to do this was esni this got as far as being implemented by cloudflare and Firefox and it worked by having a text record um with the public key for the exchange and it just encrypted the the Sni flag but it run into a number of problems for large-scale cdns and DNS caching issues and it stuck out too much and the Chinese government pretty quickly worked out but they could just block anything with that header and everyone went back to drawing board so they came along with the idea of

encrypted client hello so this hides the whole of this Hello message um however in inner hello with a real information so you've got an outer hello wrapping it and then being a hello tucked in inside and we'll talk about how that works in a minute um the key is it's not just hiding yes or no it's also hiding things like alpn which tells you what protocol you're using HTTP to http 1.1 and other things um and the outer hello helps with decryption failure so again trying to make it more um more resilient to network appliances and things um so the way that the authors of the standard are intending it to be deployed is not

on a kind of one-to-one basis sort of this Top Model where you've got one server and and you've got one key um the intention is much more that you create this anonymity set where you've got a client-facing server that has thousands of thousands of websites on and is then proxying it down to the backend server that might be somewhere completely different so very much cloudflare's normal hosting model um and the aim is that you should use the same ech config for all of those thousand sites for cloudflare hosting or whoever is doing it so that all you know is it's one of those sites that somebody's going to you can't work out what the site is and inside that bucket

there might be the phishing site that's targeting your network your corporate site and loads of other sites inside the same bucket so how are we going to exchange this key before we've started the connection so there's a new DNS record that's going to be created or is being used called the https record or type 65 and that stores a load of extra information in it so if we look up and so this text will be too small for people at the back but if you do a something with kdig and you ask it for type 65 you get a binary blob of data back if you look at cloudflare's record where they've got this setup

and it's not very easy to understand but there are some of the tools that have now been upgraded to be able to dig into this so it really is an example it's written by NL Labs so people who do Unbound it's a different tool set written by them but drill will open it up and now you can see that it's got a load of text information but then it's still got an encrypted blob of data that's in Works in base64 but um we don't know what's inside at the moment um and then some ipv4 hints and alpn information so this is this https record is doing much more than just exchanging the keys it's giving the client an

initial idea about how to connect to the backend server without having to negotiate any further and you can see why shark now understands it and if you look at these when you've captured them in Wireshark Wireshark will also decode them for you so what's inside that ech bit we saw so if you look at and this is a different site where they didn't have the other blpn and the ipv4-ins they just had the ech bit but if you look at here and you run this through um this ech CLI which comes out of a fork of openssl and I've put a link at the end if people want to download it but it neatly decoded the information

inside it so you've got some information in here some sort of magic numbers that we'll come on to later a fake Sni information that it suggests you use and then an encryption key and in this case there's two of them for this site so it allows them to rotate their keys um another example from cloudflare where they've got the other information in um you can see in the ech config again you can see a um bare fake um fake information to send for a vsni and then a single encryption key for this one um and that that key is based on a new standard that's also been ratified as an RFC and the same people are involved in

creating this new um HP ke so you can see if there's a cloudflare book a load of these Graphics office presentation got stolen from cloudflare because they've got the best graphics of this um and and yeah this is not just used for exchanging um ech it's intended as an all-purpose an all-purpose tool but this is one of the first applications of it um and there's yeah there's some other goals that they're intending to do with this new flag and one of them is to be able to do the um ACH bits but also enable people to go directly to http free if backend supports it without having to guess without having to probe it and then see we know we're back in

sports that they've put the message out of it so that HTTP free is available or HTTP 2 and also to allow you to specify that you're using non non-default ports and again like the hsts thing allow you to say I want to go straight to https um so this standard is not quite um there yet they've it's still in draft but it has been widely implemented and the challenge and this this came from a slide from one of the ITF committee meetings from uh two months ago the um they've had to bounce the thing because the two standards are kind of intertwined and they're waiting to get one ratified to get the other and they've now split part of it out into a

separate standard but we will see this what the https DNS record get ratified first without bech information and then they'll come on to do that afterwards um so how many of those DNS records are out there and what they got in them so I looked at the Open DNS 1 million list which is sorry disconcerting train whistles um so you can see that there's sort of 84 000 of that one million who've published these https records already despite the fact it's not on RFC and it's not yet a thing um so it's pretty good adoption for something that's um that's yeah not yet an RFC um and most of them have got ipv four or

six inch and there's a load of APL information and just for Life had a look on what's inside that as well so you can see that quite a lot of them are specifying that they can do HTTP free which I was slightly surprised by um yeah the percentage for were suggesting that um and um and yeah it's just kind of slightly useful information um there's also some assembly explicitly support the pre-release the original um quick version though um 327 as well um so this is obviously being sent as DNS DNS is plain text so sending Keys is DNS is that a good idea so traditional DNS you know your browser talks to a resolver and everything's over playing

text um but when the browsers are doing ech they're going to INSP insist that you're also doing Doh which means that a workstation or the browser in the workstation is talking directly to your cloud provider so it won't it will if you're using do 53 it won't even try to do ech it's my current understanding for the route for the implementations obviously that might change when we get from preview to to release but at the moment very insisting you also Doh and do Doh in the browser not Doh further Downstream because the browser needs to know that it's done Doh um and yeah and for anyone yeah who hasn't seen it you know Doh was a more recent

IFC um and yeah and it basically sends for DNS requests as a binary DNS Quest over again post over https and it's also kind of hard to block which is one of the other challenges with it um and yeah current implementation and Firefox and Edge it only uses ech if Doh is also used um so kind of preparing for this kind of had a look at what example code is around for ACH at the moment so if you want to set up the server side openssl there's a fork that's been written by a team and and published on GitHub that you can download and build and works great and that's how I did those ech dumps using the open SSL Fork

there's an nginx version that also depends on that open SSL fork and you can build it and then there's also completely separately a a go client server written by one of the cloudflare people but relies on the cloudflare fork of go I think because of encryption which um yeah you can get set up and works um so browser support for it so Firefox it's already in the regular build I bet you need to turn it on and you need to have enabled Doh Edge a bit more of a faff to turn on so you need to turn on using that new um a DNS record flag which has got a slightly confusing name and then enable

features when you restart Edge and also enable Doh I found Firefox to be really reliable you just turn it on you turn it off and it just works for ACH Edge I found to be flaky to say the least Chrome you could turn it on nice and easily but it only worked to one of the two test sites um and so there was definitely something slightly flaky about what Chrome had done and so cloudflare's test site shows you that you're using Sni doesn't really go into how it's working under the hood but it gives you a nice overview so on this site I found to be much easier so this site if SNR is off it

tells you exactly what it was sent it didn't see anything exciting um and then when you turn ECA in fact and you can see also in package race you can see the original SLI that matches the what you see in the top as you expect and then when you turn on SNL in the browser and ech and browser and reload you see the success so now you see this outer Sni which is what the Observer on the network sees as the cover where is it gone yeah cover um now they've chosen to use two names that are very similar but obviously the intention is in the wild that outer name would be something that Bears Zero

resemblance to your site and it's just a generic name and so if you've captured it you haven't captured anything at all and being a one is the interesting one but for the purposes of very examples both cloudflare and um and this other site have chosen to by her name that relates to their real sight in there um and yeah um and again in slightly more detail um so without ech the other thing I um they've also as part of this suggested that the browsers are going to send this extra grease flag so when you're not doing ech on a grease on a ech clay capable browser it will still send extra data in and Pad it out using

this grease field at the bottom which Bioshock understands is grease and shows it as a reserved extension and then with ech you now see and this is because Wireshark doesn't understand it yet you see this unknown type um and that 65037 Maps down to that feod that we saw in that DNS record um and you've got an encrypted blob of data there encrypted using the public key that was in the DNS um obviously without the private key we can't prove for those match but um and yeah those numbers so um it's in the standard but it's going to be that fe0d at the moment I'm kind of expecting them to change that before it becomes a ratified frfc because it's

in the middle of the reserve for experimental use field but maybe they'll keep it as it is I don't know um um so why why does this matter and how do people filter https today so um like you've got stuff that works down at a workstation totally not impacted by ech so like Windows Defender web content filtering you know works as an API Works before the DNS request is made life is straightforward you've got proxy based stuff you'll still see the proxy connect message so far as I'm I'm aware and that should not be impacted by ech as well if you put it as an explicit proxy or a pack file because for proxy connect

message is telling um where you're going to go you don't have to rely on yes and I information for one but the challenge is where you haven't configured a proxy explicitly and you're trying to do it in a firewall or squid or z-scaler and and you've not configured anything and you're doing it transparently so and often the same things will do proxy and transparent so the same brands in both but most of these tools have have both ways of working and transparent is great if for instance your school network and you want six formers to bring laptops in and not have to configure proxy settings but still get some filtering so there's lots of use of that transparent filtering and

that's where ech will really scuffer it because it just will not be able to see inside it's going to use the Sni and it'll only have the auto Sni which will be useless um um and yeah so I've been this this is the one that's going to be broken there might be ways around it so if you can stop the DNS message going down um and you can prevent people talking Doh then you can probably prevent it and keep the transparent filtering going but it will rely on now being able to block the Doh message block for DNS message to make that work um and said earlier that's not always easy because there are so many different

Doh providers um so how can this be used for um kind of Nefarious purposes well it's kind of too early to see exactly how ech will be used but what we do know is for Doh stuff has definitely been used for nefarious purposes people have found out ways of Hosting c2s on um and using Doh as part of their C2 or building a complete exfiltration kit using Doh so my assumption is that ech will also you know be used for good and for bad and people will work out ways of Hosting their C2 using um using ech to hide in plain sight and pretend to be Google and and slide out if if you've got a network that makes

that possible so um is this good or bad well it's happening whether we like it or not um and different people's threat models so if you are caring about privacy on your local network and you're not trying to filter then this is great people Upstream cannot tell what you're doing that is probably a good thing um but if your corporate Network and you're relying on these tools to work out what your users are doing then yeah that might not be so good um and so you know looking at it from a sort of CIA confidential confidential confidentiality point of view so you know if the question is who are you really being private from and who does

ech help you with and yes it probably does um make it more difficult for those top four to see what's going on but you know you're still going to that large American advertising broker they can still see what you're doing it makes no odds at the end um and integrity well it matters much less than that because that was already pretty good with um Regular TLS availability is some more interesting one we know that DNS sec has had all sorts of issues where people have screwed it up and broken their websites slack not so long ago um I think ech has a good opportunity for more for guns ways to people to shoot themselves in the foot

um break Fair key rotation and also it's going to encourage more consolidation onto those large players like cloudflare who have the ability to operate this have the ability to do that key rotation and people choosing to use those those players to those the websites if they want to take advantage of this um and I think there will be a people that are impacted by this in terms of losing their visibility of the people in the middle of this network management Spectrum so you've got kind of unmanaged at one end and then you've got your tightly managed Enterprise at the other end and it's those it's those people in the middle who have some lightweight filtering at

the moment works in the network using a firewall using a foxy a squid foxy server or whatever who will really struggle with this because they'll suddenly lose that visibility and they'll be forced to either go you know yeah okay sixth formers you can do what you want or actually you know six formers you're not allowed to bring laptops into the common room I have to go one of the two ways because you you won't be able to live in the middle um and yeah so considerations uh Enterprise Networks what things are you relying on Madison I information for today if it starts lying to you will you know where are you where are you getting

that data um and yeah are you using it for things like BYOD to do filtering um is it possible to get to an explicit proxy rather than transparent um and yeah maybe moving more from the network to endpoint security because again things like the Windows Defender where it works on though on the device you're still guaranteed that visibility so EDR Solutions are great but I'll still have a visibility it's the Network Solutions that are less good um and then yeah at the end lots of links that I use to grab the information for this because obviously most of it's not my own work a lot of the pictures are stolen Loosely from cloudflare site um this link at the bottom is

particularly interesting though there's been a group of Industry people sat on calls on a Monday afternoon and those recalls are recorded and put on YouTube um and so if you're interested in this field and interested in DNS or interested in me changing TLS well worth looking at for those calls and listening to them not just on this subject but on a load of others um it's a it's a really interesting venue but you've got people from Google people from Apple people from Microsoft talking and in the open I'm recorded into YouTube so um anyone can go and listen um and yeah and then there's some more of the resources used um yeah questions

so yeah that's a that's a good question um I.E I mean they can go down to China route with esni and just say yeah we're going to block that and probably end up having to block the whole of cloudflare um I think if you can stop a DNS message going down you can prevent the negotiation so if you can stop all of a Doh entry and you on the DNA regular DNS entry you filter out the type 65 messages you could probably stop it negotiating with some degree of confidence but getting that block on Doh is really hard because they you know people will pop up ones on different IP addresses and you'll never find them

any other questions foreign

and I think that is definitely the thing that we're trying to do with the anonymity sets so make those sets as wide as possible to make that as hard as possible and so all you'd know is somebody's going to a cloudflow hosted site or whichever cdns choose to choose to implement a standard but yeah where somebody is hosting their own C2 and has chosen to build their own ech config to hide their own C2 on a VPS somewhere yeah that will stick out like a sort of thumb if people are doing that analysis is there anything else that's missing from my TLS so this is like fixing the updates I think the other sets of changes

they're looking at of like the HTTP 3 the quick getting to UDP and um and there's also um again changes being talked about to do uh doh over um over here over over quick as well drq um and yeah things like the Apple private relays solution and standardizing that as as kind of solutions where they're trying to enable that kind of proxy solution and make it easy for people just do one of the things that are just quick which is which is attracted a lot by by their own purposes

big companies themselves are starting to lose control Google is telling the browser or the system at the end yeah through the quick messages yeah absolutely

yeah I think and again you yeah you see it we've got I've also seen it with H2 as with regular H2 as well so if Chrome talks to Google and it negotiates itself as HD connection you see if yes and I has been google.com you then in that browser go to youtube.com you don't see another connect message it just reuses your existing connection and downloads YouTube over it and and obviously if it only works for things that are posted by the same company but Google and YouTube hosted by the same company but as a as an Enterprise you might have a different policy around YouTube in the office to Google in the office particularly schools do

um and so yeah H2 already causes a problem on transparent networks and yeah H3 will as well thank you

thank you [Applause]