XZ made EZ by Joey DeVilla

[Music]

oh hey thank you thank you all right okay thanks very much well let's let's do this properly and I'll start with a really really quick accordion number and then we'll get right into we'll get right into the XZ

hack darling you got to let me know should I sleep or should I code if I sleep I'll miss the Milestone and if I code I'll break my headbone so come on and let me know should I sleep or should I [Music] code thanks so much so um in the spirit of making it accessible and also because I do not actually do security work as my main gig I'm going to keep this as as intro and beginner level friendly as possible so you do not have to be a member of the neon Temple to understand what I'm what I'm explaining and in fact I'm going to explain a few other things that might actually make you give you a little bit

of extra computer science knowledge so um this is going to be fun this is going to be this is going to be beginner friendly and uh yeah none of the none of this is going to be deep Arcane stuff but you will understand a little bit more about security after this so anyways to begin with yeah I'm Joey Dilla I've uh done a whole bunch of things so uh I am as of actually officially as of Monday I'm the developer uh relations guide a company called unified unified is basically a unified a uh API company instead of having to call 12 different apis to build an application especially a SAS application for business you only have to call one

and you will never ever have to directly connect to Salesforce again that alone makes it worth the money all right okay the other thing is along with an Nitra who is in the lime green leopard skin jacket there uh I run a couple of uh she's my wife and we run a couple of meetups together one of course is the Tampa Bay AI Meetup uh the other one is coders creatives and craft beer that's all conversations no presentations it's just a fun Meetup we're going to um we are organizing ing one has any uh there is a new vampire themed bar downtown Dracula Legacy that's where we're holding it because that's going to be a

hilarious place there is a giant bat Throne there are pictures of Count Dracula everywhere it's going to be a fun bar to hang uh uh to have a me uh to have a meet up in and the slides are available at Global nerd.com besides dxz if you want to follow along or look at it later and I'm actually going to update those slides because the XZ backdoor is an ongoing thing we only found out we only generally found out about it last week and really quickly uh the other thing I do is on my personal Tech blog Global nerdy this is one I do on my own not for any employer I publish a weekly

list of Tampa Bay Tech events go check it out and basically it was a job search tactic that went wrong uh what happened was when I was looking for a job I thought I'll I'll present myself as a community guy and I'll put together a weekly Community list my plan was to not was was to stop publ publishing it after I got the job but the minute I did everybody was going where's the list I want it back so I just decided Well you know let's do a community service let's keep doing it now keep that in mind because that's actually going to be a factor in the hack I'm going to discuss this whole Community

thing all right and once again yeah quick plug because I am newly employed by this company and uh I need to need to let me know because I have to send this to my boss and I have to say look I I'm worth the money you're paying me thank you for hiring me during the Great layoff season and yeah it's at

uh who here grew up in the 80s actually is the other thing so some of you grew up in the 80s there are a lot of people who actually thought that Dungeons and Dragons was a game they made up for the show and not a real life game but DN D has been around since the 1970s a bunch of nerds in Wisconsin who REI who wanted to play Let's Pretend with The Hobbit with dice decided let's make a game and they call yeah and they called it Dungeons and Dragons and vcna is actually one of the old oldest characters from The Dungeons and Dragons game and VNA uh oh and the the important thing about Dungeons and Dragons is it

is a formative part of being in technology if you are in Tech and you have not yet played Dungeons and Dragons go play Dungeons and Dragons or at the very least if you play video games go get the balers gate uh game cuz that's D because that's D and D in video form but yeah by all means uh Reed Hoffman the founder of LinkedIn actually creates his success at Tech to being a d andd player and yes VNA is a lich and a lich basically is just somebody who wants to live forever and is a wizard or a cleric or some kind of magic user who decided through a combination of magic and sheer force of will I want to live forever and

be evil forever and vcna is one of these and in fact there are a couple of artifacts this is from the old school let me see if I can't find it here 1980s actually 1979 edition of the dungeon Masters Guide from advanced Dungeons and Dragons the rules are quite different now this these are the old school rules this is the text from the old school rules there are a couple of treasures that you can get but they carry terrible prices one is the eye of VNA it is a magic eye that used to belong to vcna and as a piece of treasure you actually have to get a spoon and scoop out your own eye to put

in the eye of VNA it gives you great Powers but at a great price there is also the hand of vcna which is vcna severed hand you have to chop off your own hand and then graft on just hold uh actually hold the hand of vcna to the stump where your hand used to be it will magically attach it will give you superpowers but at a price and this is actually also related to the topic that I'm going to talk about and the topic I'm going to talk about actually has a formal name it is CVC uh cve 2024 3094 it is otherwise known as the XZ backdoor and had it been successful had it not been caught by one person who was

just running some tests and noticed something weird and actually decided instead of say ah so weird things happen all the time you know we'll we'll uh I'll just deal with it cuz he's San Francisco weird things happen all the time there anyway but he actually looked into it and that's how he found out what was going on and this is what we're going to discuss now the thing the th the security vulnerability in XZ is a back door and uh yeah a back door is very similar to a portal to the upside down and a back door basically is a way into a piece of software that bypasses the normal authorization or some kind of security protocol now back

doors aren't always all bad if you've ever had to talk to se tech support or um because uh somehow you got locked out of your own account or there are times when of course you know people are going to forget their passwords I once had to help somebody actually at an advertising agency because um they he had this he was maintaining this mail server they had a completely numerical password and somebody decided to prank and when he was logging in to maintain the mail server some guy just started yelling out random numbers and for some reason he completely forgot the password so they had to call me in I had backdoor access to the service and that's what allowed

me to re uh to actually let him into the service so he could reset his password and do his own things a back door allows you to do a back door allows you to do administrative things and it is a necessary thing but it is also a way uh it doesn't it doesn't just let the good guys in it will also let people with bad intent or people who are completely unauthorized and not allowed to do anything in that system to hop in and bad back doors are back doors that are installed often without the knowledge of the people who own the system or who are authorized to run the system and they're generally there to

allow an unauthorized party to perform one of the four s's as I like to call it spy uh spy which basically means look into the system and uh look at information they're not they're not allowed to access steal this is more than just copying information this is possibly uh performing Financial crimes subvert which means basically either take o take over the system to do something you want or prevent the owners of the system from doing something you don't want them to do and of course sabotage just completely mess up the system take out somebody's cap anybody's capability to use that system for its intended purpose and that's what bad back doors are for and that is what got

and that is what got installed into a utility called XZ which is in every Unix system that includes Mac OS and uh I have a little bit of a experience with back doors because I'm friends with the people at the cult of the dead cow they are the OG hacker group they have been around since 1984 that is my friend death veggie there uh that's at Defcon 8 2000 so it's a little while back and um Cult of the dead cow members clean up really well and end up in really really uh really really high places that is mudj AKA Peter zco uh the photo is him on the left in 199 uh 1995 when he was a guitar student at the

Berkeley School of Music and uh that 2011 photo is him at DARPA in charge uh in charge of President Obama's cyber security community so you can go places even if you uh did only three years playing guitar at the Berkeley School of Music okay uh this other guy here uh we used to know a psyched elic War warlord and later on when I found out he was running for congress in Texas I was going wait a minute psychedelic warlord oh he cleans up real well I was just going I never would have guessed that's the same guy but that's bet aor so he is also a cult of the dead cow member and if you want to learn more

about the CDC Cult of the dead cow there's a fantastic book by Joseph men uh he talked to he talked to all the original members of The Cult of the dead cow including uh guys with handles like uh Kevin Grandmaster rat there is Sir distic and then there's my favorite handle uh dild doog and uh the cult of the dead cow had did make a famous back door called back orifice in fact they made two versions they made back orifice 98 for Windows 98 and then they made back orifice 2000 which would run on Windows 2 um which would run not just on Windows 9 8 but Windows NT and Windows 2000 uh and this is around the same time

that a young a young guy was trying to convince PayPal to switch from uh totally insecure uh insecure Linux to way more secure windows and T and he got kicked out of that company uh that guy was Elon Musk the company was PayPal yeah had they done that the cult of the Dead would have totally owned PayPal so good call on just letting him go and sticking with Linux so XZ is totally wizard graybeard Unix stuff um you are you are not likely to use it directly in fact you are probably more likely if you use it at all you're probably going to use it as part of an argument uh as part of the capital J

argument in the tar command and I'll talk about tar a little bit later on but basically it is for doing file compression and it is primarily in what they call posix systems which is a fancy term for uh Unix things so anyways that means Linux distributions and yes Mac OS if you have your Mac right now you can open Terminal and type in XZ space-- version and you can see if you have XZ and if you do what version you're running and that'll be a little bit more important later but all right I mentioned and of course I have to talk about tar tar for those of you who are not who don't know that it's short for something

it is tar is short for tape archive and basically it has just one purpose to take a bunch of files and and just gather them all into a single file that you can refer to by a single name that is it it does not do compression you run a separate program to take the t uh to take a tar file and then compress it but yes tape archive it's called tar for a couple of reasons one you have to remember that this is a Unix was made in the era when you were talking to a remote computer over a 300b modem which is really really slow that means 300 bits a second and you have to remember

at the time and uh you have to remember at the time a character took up eight bits so there was a there is and characters take up 64 bits today with Unicode so anyways it was slow so the idea was keep unix's command short and of course tar was for tape archive backup or backup to Magnetic Tape which at this point somebody's going to go OMG WTF is magnetic tape and I will tell you gen Z it is a way we used to store data actually it still is a way to store data because it is it is still space efficient you can store piles of data on it it will last longer than any hard

drive and yeah definit um and it will if you put it in the right kind of case it will survive a fire and of course you don't when you put it in storage you don't have to keep it plugged in and you don't have to keep it attached to power and uh it's got a life it's got a lifetime they say 30 years if you take really good care of it maybe 60 years uh the DVD ROMs you buy these days this backup May Rock within two years so tape is still a useful thing and in fact there is such a thing if you are new to Unix if you've been living in the Windows or Mac world for a while uh

you may have heard but not know what a tarball is and the tarball basically simply a tar file that has been compressed that is it but tarballs are a ve are a very common thing that we use especially in the Unix SE world for moving around what is called object code and I will tell you what that is don't worry I keep saying I will tell you what that is later but I promise you I will tell you and you will find out but to make a tar ball you have to do compression and uh a lot of people are a lot of people actually don't know how compression is so what I'm going to

do is I'm going to show you the world's dumbest compression algorithm but it will give you a general idea of what's happening with compression so the first rule about computers is that everything in a computer is a number every character you type in is represented by a unic uh a a blob of Unicode and that is 32 to 64 bits but it is a number every pixel on the screen is a number so what I'm going to do right here is let me show you this this is this is a very simple picture it's just 16 pixels 4x4 and they're all orange and let's say I've just decided that the that the number five is going

to represent the color orange orange so I can represent these 16 pixels with this array right here 16 fives that's it so this picture takes 16 numbers to represent now all compression algorithms try and take advantage of repetition so my dumb compression algorithm says I'm going to represent pixels uh lines of pixels by groups of numbers order an ordered pair where the first number is the number of the color and the second number is the number of pixels so basically what I did was I just took 16 pixels worth of information 16 numbers and I squashed them down to two and that's an 87.5% size reduction I'm going oh this is great now a picture that is all

orange is kind of boring so I'm just going all right well what happens if I do this I'm going to make uh 10 of the pixels green after the first five pixels I can still represent that with four numbers uh two ordered pairs the first one representing the first six orange pixels and then the second number representing the 10 remaining green pixels I'm only using four numbers instead of 16 so I'm still say I've still compress my file by 75% now the compression algorithm gets less good the fancier the image does this time what I'm doing is I'm just going okay I'm now representing three runs of colors some orange some green and some uh some pink and uh I now need

three ordered pairs six digits but I'm still compressing The Thing by almost 2/3 so so far so good and in fact let's say I introduce one stray white pixel at the end I now need four ordered pairs eight numbers instead of 16 but I'm I've still compressed the file to half its size so I've going okay the compression algorithm gets a little worse the fancier the image gets and in fact if I go with a picture like this this is where my compression algorithm falls apart and this is where many compression algorithms fall apart each of these pixels is different for each one right it keeps changing there is no continuous repetition of each I

now um I'm using two two numbers to represent a single Pixel and I have to do it all 16 times I have actually expanded I have doubled the size of the file and this is what happens when you try to compress a JPEG JPEG files are already compressed and when you try to compress it you are actually adding more additional information instead of taking away information and that's why jpegs get larger larger when you try to compress them now what I am doing right now is called lossless compression I making sure that I can reconstru uh that I can reconstruct the F the the image here perfectly now let's go back here let's say I say uh let's say I say

you know what this is just one white pixel nobody's going to notice I'm going to compress it like this all right I've lost information uh I've lost information and when I reconstitute the information using these three numbers instead of the actual four numbers yeah I don't get an exact copy it's close and for some purposes that's good enough for pictures most of us we don't kind we don't care about the occasional straight pixel but that is lossy compression so that's the difference now the reason I'm going into all this of course is that XZ is a compression algorithm and we use it to actually transmit large files usually for system updates this is how XZ works

it's a little bit fancier than the world's dumbest compression algorithm that I showed you but it does pretty much the same thing it takes advantage of repetition in data and then goes okay I don't have to repeat that data so I will use uh I will use this repetition so I don't have to copy as much information that is the spirit of compression and it is based now XZ used to be known as lzma and L lzma is short for lle ziv marov algorithm uh marov is basically the guy who came up with marov chains which uh talks about randomly moving from one state to another and lle and ziv were two math guys who just

figured out how to take things like Huffman and coding and turn it into compression there is a dark science behind compression it requires it requires a lot of math and despite the fact that I was an electrical engineering major it is math I don't want to bother with so I'm just happy to go and use something called the lzma library and go you know whenever I need compression I'm going to use I'm going to use the built-in functions that are in that library that are available in just about every Unix system because somebody has done the math for me I'm trying to write an application I'm not trying to write I'm not trying to solve

math equations and this is how you actually use tar with XZ typically what happens is you you almost never use the XZ command itself you use the tar command followed by uh remember this is UNIX you usually put the options after a dash immediately after the command and it's the capital J that says hey by the way while you are creating the tar file compress it you using XZ and then you say I want and when you've compressed it I want it to go into this file and I've given the the resulting file by name tarball tar.xz and the files that I want to compress I just list afterwards that is this is a very common thing lots of

symen do this every day on Unix systems now the problem with XZ is it's compression it's not sexy it's not fun it's really hard to get a developer to do it for fun normally this is why we actually pay developers a lot of money it's because a lot of the stuff we do is just downright boring and we would never do it otherwise and yes there's always a fad right now and there's always the hotk new thing and every developer wants to be in AI right now yada yada yada the problem is XZ is boring as all hell and have any of you seen this xcd cartoon yes okay this is very important right here this random this random

person in Nebraska who has been thanklessly maintaining it since 2003 I thought this was well actually a lot of us thought this was an exaggeration but replace Nebraska with Finland and replace 2003 with 2005 and you get this poor guy this is Lassa cin he is finish he go uh and he is X Z's loan maintainer or he was XZ loan maintainer he was experiencing a lot of burnout this is his GitHub commit list there is no shame in this all right uh I have an uh I have uh for my old company I actually have a very big green GitHub commit commit list but this was as an employee of OCTA on my personal list there's not

that much cuz I was just going well you know what I I want to do other things aside from code but lass Lassa was experiencing a lot of burnout and of course he had been maintaining XZ alone and all the XZ versions which includes XZ written for C XZ the XZ port for Java the XZ port for various system architectures and yes he was experiencing burnout so enter this person who's going and I have to put their name in quotes because we don't know if this is their real name Jan creates a GitHub GitHub account in 2021 uh has an email address that is uh has one of these email address that is name three digits at uh vague email

hosting company creates an account this account is still active but suspended jitan contributes a lot of stuff and um November 2021 makes a pull request for lib archive now for those of you who are not familiar with GitHub a pull request basically is hey I have a correction or I have an enhancement to this code take a look and if you like it you can add it to the uh you can add it to the main code and the code that he wanted to make a contribution to is something called lib archive which is a C library for archiving and compressing files it's a very nice friendly positive open- Source thing this is what Richard

stalman and Eric Raymond talk about uh see did oh I'm missing a okay actually I'm missing uh actually no I'm missing a slide here uh no no I'm not but okay positive contribution let's go with that uh let's go with that but he makes a commitment to XZ a little bit later on this is C code and basically what he's doing here is he's it's a simple thing he he he he just adds an if statement anybody here Cod and C or JavaScript okay all right Perfect all right it's an if statement just to check if the uh the parameter for these two functions uh lzma lzma props and lzma LZ lzma2 props okay one of the parameters

to that function is a void pointer for those of you who don't program in see a void pointer is a pointer to any address anywhere in the system this just he just added codee just to return an error code if somebody gives that function fails to give that function a an address for that pointer because if you point uh if you if you try to access something with a pointer that has not been initialized you get

this and anyways he submits this and the problem with some poll requests is that a lot of people just kind of review them really quickly especially if it's volunteer thing and they just T and they just go lgtm looks good to me in fact you'll see do a search on GitHub sometime for the uh for the letters lgtm in like that you will see lots of commits that get pull pull reviews that get accepted with just lgtm very quick review and one thing he did was Gaton later on filed a poll request that he said did one thing but did something else entirely and the one that he said I'm fixing some weird error messages what he

actually did was he replaced a print function a good safe print function actually it has safe in the name safe F print f for safe formatted printing it's very good at handling control characters and invisible characters versus the old thing FR print F so basically he downgraded the safety on one package which is like and basically it's like kind of trying to serve off brand potato chips which is why I have prongles here all right now hang on just a sec let me step I have hit the wrong button here okay so what I'm what I'm showing you now are emails being sent to the XZ maintainer now here's the thing you know this is

around 2022 Lassa Colin the XZ maintainer is feeling burned out and people are going hey is ex U people are now harassing him why haven't you maintained this why aren't you updating this so there's this guy who goes yeah hasn't been updated in over a year and Lassa goes back and goes you know what I'm uh uh you know I'm really busy I can't I I I can't take uh I can't take care of this all the time but development is ongoing I promise you and this really helpful guy gatan has been contributing to the code you know so I'm hoping I can get some help from them whoever they are you know they see they

seem all right then this comes on and basically goes yeah you know what you're not you don't care anymore you're not doing this all right jigarkumar actually Kumar jigar Kumar uh also has an email address that is jaykumar three digits at generic hosting company doesn't really have much of a net presence prior to to uh this email exchange keep that in mind lassi responds back and um says and uh uh does one of the uh does something that is a bad thing to do from a security point of view he told the truth he actually said I have mental you know I've got some ongoing mental health issues I'm trying to take care of this

please remember this is a passion project are going to and this is what this is how jigar responds so now there's pressure you

know and then the more reasonable guy makes what looks like a reasonable looking suggestion I'm sorry about you know I'm sorry about your mental health issues but in the kind of a passive aggressive way maybe you can pass me uh maintainership of the project to some somebody else little bit of pressure there and lass basically says you know I I I got to be careful about finding a co-maintainer you know while I am busy this is an important project there are systems that do rely on this compression algorithm and he explains later on yes as I may have hinted you know what this Gaton guy he may he's made some contributions he's been solidly helping

maybe I will hand it over to him so on November 30th 2022 two things happen this one takes all the air out of the room this one this one is what everybody's talking about but the other thing that happens is that Lassa Colin updates the bug reports email uh for the XZ project and makes it an alias that sends mail to both him and Gat and at the beginning of 2023 AA releases his last version after which the new maintainer joton releases the next version of XZ from this point on Gaton is running the project The Helpful guy who swooped in at the right moment when people were coming down hard on the original maintainer and then he makes Gia makes a

set of small innocuous looking changes and he opens a page request in OSS fuzz now fuzzing basically is the uh uh fuzzing is the process of basically feeding all kinds of garbage input into a system to see if it crashes or if it responds incorrectly OSS fuzz is open source is a open source fuzzing utility and in fact here is an example of fuzzing right here a tester walks into a bar and or orders minus one beers qxy you know uh orders in Klingon actually yeah that's Klingon there right beside the Hebrew all right so that's what fuzzing is but he fussed he OSS fussed with OSS fuzz he actually uh disabled the use of

a function that would allow OSS fuzz to detect what he was about to do to XZ and he submitted it he submitted it as a page request but he said I'm fixing some weird fixing some weird error messages but what he actually did was he disabled a particular part of the safety system and then he adds an ignore file and he adds a file sorry to the G ignore file and for those of you who are not familiar with Git G ignore yes uh let's see February and uh actually uh oh I forgot to date that one that's about January that's about a month before so a month before he F he he he he futs he futs us OSS fuzz and

then yeah a month later he adds a specific file to G ignore G ignore basically is where you list all the files you don't want checked into Source control and one file he he adds to get ignor is called buil a host. M4 the M4 file is a macro file and you normally ignore it basically it's a build file and uh he he while he removed it from The open- Source repository he included it in the distribution packages for XZ so there would be no way for anybody to actually look at what the that particular M4 file does and nobody would care it's a configuration file it's not the main code nobody pays attention to

that so February 22 23224 we know this only by a lot of security people looking at his GitHub commit history he adds a couple of test F he adds a couple of files to the tests directory and these are test files for compression who's going to look at that and nobody even looks at test files anyways they're just for testing the main program they're not what the program is about so clever test files these are large binary objects they look like this here's the thing all right normally when you compile a program you write your code you write code you run it through the pre- processor in the compiler that produces Assembly Language Assembly Language is closer to the machine but

the machine can't understand it yet you have to run it through the assembler to get code that the machine understands but you may need to use other libraries basically code other people wrote what you do um and in order to use them you need to take your assembly code your object code and Link it with those libraries and that's handled by the Linker and that's how you get the application so that binary BL uh that binary test file actually contains object code but it was hidden it was actually not just they they they did a little bit of math so it didn't look like machine code but then what they did was inside this test

file they took that object code and split it up into pieces so uh they they broke it up so that they'd read 1,24 bytes skip the next two 48 bytes read the next 1024 bytes but basically this code was just broken into pieces just the way uh some people sometimes disassemble a gun into tiny pieces and hide the parts all over their house so what happened was that M4 file which was hidden from Source control but part of the distribution when you compiled the new version of XZ took the evil object code created this little Library here this uh which uh the XZ compression uses and then created a new poisoned after round version of XZ

with now XZ installs this new poisoned uh lib lzma Library this is the thing that actually knows how to do the compression the lib lzma library because it is part of uh because it's a compression utility and it's often used by the operating system got hooked into by System D the system demon Unix users will know this as process ID number one it is it is the main thing that runs the entire show it is the ring leader of a Unix process every time you start up a system and system D of course has access to SSH which is secure shell which is a way to securely uh sec Al contact your uh contact a computer so

really quickly uh the way you contact a secure shell of course is you need a private key I'm going to skip this bit but basically if you download the files this is the easiest way to explain P private keys and public keys but basically what I'm saying is that if you have the private if you have the private key for an SSH server and you happen to know where that sh server is you can log into that system and log in as roots now how disaster was averted remember Murray from stranger things sort of the secret hero okay a guy from Microsoft of all places working on postresql of all databases handed in this message

basically long story short he was going log sshing into a system for postgress is now taking a ridiculously long amount of time but we're not talking that really long we're talking on the order of 30 to 40 milliseconds you would never notice this and he's going but I remember somebody was having the exact same problem with Val grind which is another unix's utility and they're uh he's going I need to look into this and it turns and that's when he started finding out wait a minute what's going on here something is allowing uh something is allowing somebody to ssh in with some private key that we don't even know to log in uh to log in and basic

log into any arbitrary system that has this version of XE as root this would have compromised millions of servers most of them public or Enterprise servers or things that run utilities including water electricity fuel supply Etc and of course this gatan guy who slowly took over the uh slowly took over the XE Library had to do it over two years they were doing the long they're playing the long game doing the long con which makes us wonder is there a nation state and a lot of people the three the three main suspects are China naughty Korea not nice Korea there's nauy Korea and nice Korea glorious leader is Lee

taxor or maybe the not so former Soviet [Laughter] Union and that's what they've been throwing around I think right now there are some hackers who are saying a 20 is it 28 one of the Bears there's fancy bear there's careful bear there's a whole but of course this is all speculation at this point but the first thing we need to know now is that this is no longer the 1990s we uh we're going to if we need if we're going to get people to contribute to open source projects we need to get better at vetting them we uh this is this is not the wild west anymore uh this is this is a different world from when open source first uh

made its big rise to prominence and we have to be careful about that kind of thing the other thing is that open source is no guarantee open source is good I don't want to poop all over open source but it is no guarantee that it is secure uh the line with many eyes all bugs are shallow only works when many eyes are paying attention and not when it's just one Lan person in Finland working by themselves in their spare time writing a utility that major systems use all over the world so like I said not the 90s anymore and remember this this has become the world's most relevant cartoon as of today and that's it thank you very much

are there any questions I'll answer them as best I can yes so open source by definition is completely distrib so how do you enforce the suggestion that you made to have open source developers partip devel unfortunately the way to do that is also to take the open source approach and distribute that uh distribute that training but luckily with open source what happens is there tend to be some leaders that people look look towards and then they emulate those leaders basically what we're going to need to do is we're going to need to find the open source leaders that are out there today to come up with a way to propagate this uh propagate better processes just just as um you know what

there was no sense of poll requests until GitHub made it popular can you shut up for a second please yeah I'm sorry so GitHub uh GitHub actually created some de facto processes that we now kind of accept as uh as standard like pull requests that's spec you know that that didn't exist before systems like GitHub so we're going to need to create these de facto systems as well I think I could take a couple more can you repeat the question whenever is asked oh yes sure definitely record okay perfect do you have a list of Nebraska do I have another do I have a list of other Nebraska like util I don't but I believe somebody does

because I'm sure that cartoon scared people into compiling a list like that I should try and find it but I will update the slides for that yes maybe it's just me but I'm always just curious about everything so the fact that without have maybe I myself a lot more time a lot more curiosity but that just that part feels

oh yeah always be the question is yeah how could this have happened uh with a PR and somebody not looking uh and my answer basically is um security falls apart when somebody fails to do the necessary leg work and one of the tricks that they uh that people use to get people to skip that leg leg work is to use time pressure so that remember that email change that was a lot uh that was probably one party if it may have been several it may have looked like several people but it was probably one paying party uh being paid to add pressure to the maintainer to make uh so that they would make less informed decisions or

skip things they normally would I think I for he put that in after got access think um he got uh yes uh but basically he but he what he did was he established trust and that is the important thing is that you establish trust so you can betray them later in the back though set up for the next one yeah got to set up for the next next one uh I'll uh I'll be out in the hallway to answer more questions but thank you very much and thank you bsides [Music]

[Music]