BSides Oslo 2023

um so this is just a screenshot showing that these three online uh tools that index all packages show different dependencies for the same package so this is still open there's more more details in the blog post the depths.avs see URL lab3 while sneak and the socket.dev see charge set normalizer okay so I talked about um manifest confusion and distribution confusion and this is resolution confusion is a attempt at the umbrella term again if you you've seen this before let me know but it's all the ways to resolve a package into something else than you expected so of course we have dependency confusion where the package installed from the wrong Repository manifest confusion where there's multiple ways to resolve wish dependency a package has and then distribution confusion where you might get the wrong distribution so briefly on how I approach this research so I was actually starting to implement a transparency log for Pi Pi and then I had some assumptions that led to do some bugs and I started reading the powerpi source code and it turned out that the allowed inputs were created and what was needed to do a regular um upload so I I wonder if this was possible to exploit and then of course the you can only have one source distribution I got in the way and I started to look for bypasses for that so the four blog posts are released today is here uh I've touched on most of these but the the second one only reproducibility in Pi Pi that has a lot of other ways where you might get unexpected results with um with python packages and with that I can take questions [Applause] anybody questions up top on the balcony downstairs yes mate thank you um now that you've seen that this possibility exists um have you had the opportunity to scan through Pi Pi for example to see if there are packages that could be vulnerable to this do have the these confusions in place or or other um in the wild occurrences of this yeah so in general because the the file name rules have been so lacks historically there's a lot of garbage in the system and for the most part it looks like it might be accidental but I haven't looked closely enough to to know if it's accidental or not those issues that are present and so that might be a future research but luckily like this is easy to scan for right it's easy to detect much easier to detect than if a package is malicious anybody else questions for steon manifest confusion distribution confusion I like the Rhymes where is the hand keep it up hello so the prerequisite that was listed uh what does that mean and how hard is it to get it so that basically means you you need to compromise a package to be able to uh to publish as the maintainer so it could also be of course not the maintainer turns evil but you need to have permissions to to actually publish packages which should be our high bar and basically when when you are pulling in a project you are trusting that project to not do malicious stuff right as this is just one malicious thing you can do if you have those permissions anybody else questions for stellen about these python package distribution attack vectors hi um I was wondering you mentioned detection do you have any examples of good tools or ways or mechanisms to implement to detect these scenarios yeah as far as I know nobody except the test code I have is is doing that detection all right and uh you'll be here today still if anybody has anything else they want to ask you you can also ask him about Quantum cryptography if you like so thank you very much stay on thank you [Applause] and all right so that was our last talk before lunch so luckily the talk with the sandwich in it was well placed Let's uh find our way down and grab some food the food might not be uh out until 12 when the scheduled lunch was happening but it's coming and we'll be back after lunch for the rest of the program thanks everybody welcome back everybody I hope you had a chance to grab some food and and chat with some some new and old friends colleagues acquaintances um a couple of comments before we get into the rest of the program if you pre-ordered a t-shirt um please pick it up um there are some T-shirts that have not been claimed if you did not get a t-shirt in your size due to our uh delayed shipment from our supplier please go downstairs and talk to penita at the reg sponsor area and work out a solution with her we can do refunds or we can get you your shirt when it arrives we also have some of these shirts from last year's event and uh you can pay what you want if you would like one of those shirts as a donation uh up to you any shirts for this year's event that are not picked up by five we will put them up for sale as well because many people have asked and we're waiting for everybody who who pre-ordered to to get those so just shirt stuff okay uh it was inevitable that we would have a talk about AI so preben is a PhD student at entenu Charles and he's working on ways to break Ai and hopefully some solutions for how to fix it fingers crossed take it away Brevin thank you uh excellent yes uh my name is uh I'm at Engineering also at similar research lab here in Oslo um and I'm going to be talking about ways that you can break AI systems using something called adversarial attacks um and some ideas for for some solutions that you might want to to implement which is what I'm researching in in my PhD uh so first just a bit of housekeeping I'm going to start off with a brief introduction about how neural networks learn how they're trained I'm going to be talking about deep neural Nets which for if anyone doesn't know that's the sort of the main AI architecture that you'll find in chat gbt and the algorithms on on Tick Tock as well um so after that brief summary I'll go through what adversarial attacks are how they work um and why they're a problem hopefully convince you of that um and then in order to introduce a solution we first need to talk a bit about how I've said how neural Nets represent information you could say how neural Nets think if you want a anthropomorphize a bit um and then I'll introduce what we call the causal neural network model which is sort of my area of research and end with some thoughts on the the state of the art all right so first things first uh for the purposes of this talk I would like you to think about deep neural Nets as very powerful pattern recognition machines they're very good at finding statistical patterns and data um they could be very complicated patterns such as for example the patterns relating the pixel values in an image file to the label that you would like to apply to that image if you're building an image classifier um and I'm going to be talking about images in this talk um because they're quite nice and visual but everything I say is sort of equally applicable to AIS that deal with text or speech or you know stock prices um okay so we're trying to identify patterns between some some input X and some output y we have a very powerful machine but because this machine is so powerful it can spot patterns that are false patterns that don't generalize so I have an illustrative example here um the um if you imagine training an image classifier on the cows in the center there and you train your AI to say these are pictures of cows then the AI will likely pick up on the fact that okay all of these they have like some green grassy stuff in the background that's probably related to what it means to be a cow um you know you train this you get good accuracy you deploy it and in production you encounter the image on the right and it's very likely that your uh your AI is going to fail um because that pattern no longer holds uh it was a false false pattern all right that's all well and good it's a cute example um but it allows you to do some more malicious stuff um it allows you to do what we call adversarial attacks by by sort of exploiting these false patterns um and an adversarial attack uh for images consists of uh you start with the image there on the on the top right which is of a panda you have an AI system that correctly classifies that image as being of a panda and then you add some some noise to it some carefully crafted noise just a tiny bit to produce the image uh there on the bottom right um probably from the distance you're sitting you can't even sort of tell that something has happened um it certainly looks quite similar it's just just a tiny bit of noise but the AI system that you train now fails completely to classify this as a panda I think it's a it's a cat in in this example with very high confidence and we think that the reason this happens is because much like the sort of grass cow example there is a more complicated patterns in your data as well um that aren't as sort of obvious and explainable they can be to do with sort of fine structures in your pixels um you know all very complicated stuff and it means that you can add noise in just the right way to to trip up the AI um so that's what an adversarial attack is um and it's obviously not that big of a deal if you're just classifying pandas but you know if this was your self-driving car or if this was supposed to diagnose like cancer tumors or something um then then it's a bit of a problem um it's very interesting to note that although AIS are very susceptible to this type of attack humans are basically immune like you hopefully no one in this room looks at the bottom picture and thinks it's it's a cat right all humans sort of naturally are able to defend against these attacks um and the hypothesis is that the reason humans are very good at this um is because we are naturally good at extracting uh what we call the causal information in in this case an image and that means we're good at extracting basically the important information the important the information that affects um in this case the classification we're trying to make and we're not fooled by sort of random noise in the background um the way that these statistical pattern machines are um and there is a theoretical solution to this vulnerability and it is if you're trying to recognize pandas you just need to collect all images possible of a panda you need to go out with your camera and take pictures with all types of backgrounds all types of lighting conditions and camera angles and lenses and and all that kind of stuff um that's obviously kind of Impractical so we would like a solution that we can do without having to leave the office um so to introduce sort of why or how we can go about solving this we first need to dig a little bit deeper into how how neural Nets uh think how they represent the information that you give to them uh so I've drawn a very simplified schematic here of a neural network um in this case processing images so this could be the the panda classifier um and we see that a neural net consists of a bunch of layers um and they process the information you give it sequentially starting with the first layer and then moving up to the second and so forth so so the first first layer here um looks literally at the RGB Channel values of every pixel so that's very that's a lot of information you know you can have 200 by 200 pixels in an image three color channels that's 120 000 I think um so but that's that's very low level information there's a raw pixel data um then there happens a bunch of processing and the information is fed to the next layer which is smaller um so this this next layer has a a smaller capacity for sort of storing information and it needs to sort of disregard the low level details and then try and sort of extract some some more high level stuff uh so the next layer might sort of learn to identify basic lines and Corners maybe in your image do some like very basic processing and then you sort of move down the chain here to to deeper and deeper layers and at every stage the information is compressed down to a more uh sort of abstract high level representation um so at the very end you might have quite a small layer that encodes information about I don't know like faces and sort of fur and snouts or whatever for um for your animal classification and then at the very end a prediction is made based on that sort of compressed information as you can basically think about this as a as a lossy compression algorithm aimed at sort of extracting um a particular type of information okay so so this allows us to sort of reduce down from all the possible combinations of pixels to a smaller combination of sort of high level um codes or are compressed uh representations of the same information right we now have all the pieces we need to think about what we call the the causal neural network architecture um and instead of having a single processing pipeline like we saw in the previous example the causal neural net takes in the image and then produces two independent sort of information streams level one C here for Content the content of the image such as uh subject and shape um and there's one called s for all the other information the lighting the camera angle call it the style information um so this is um an architecture which aims to separate out all that important information to to put it in in the C stream and leave all the unimportant stuff all the style information um although like grass information uh in the S stream um and if you're able to do this uh sort of correctly or or sufficiently accurately then you can do something quite clever which is that you can introduce those little box here that says perturb signal what that means is you're taking your style signal and you jiggle it around you add some noise to it uh you you flip a few bits here and there just to try and try and corrupt it a bit make some variations on the original style signal uh so you might you know produce sort of say 10 different variations on the actual style signal that was in the image um and then you sort of one by one recombine these with the uh your content stream uh and you have a have your neural net make a prediction at the end and very crucially you tell your neural net that regardless of what I do to the style signal you should always predict the same label um so so this sort of allows you to approximate that um Gathering of all possible images in a much more manageable formats and under certain quite lacks mathematical conditions that I'm not going to go into this sort of converges to a uh to to A system that um is able to to do what humans do to sort of extract the the the causal important information um so this this causal neural net architecture is um is quite quite a new thing I think yeah the first papers I saw discussing out from like five-ish years ago maybe um and uh they've gained a line popularity over the years since um because they are as we've talked about not fooled as easily by adversarial attacks um they're not fooled by sort of adding noise to images or if you're doing text you know it's easily fooled by swapping words and and sentences um but we still have uh a long way to go um even though they have all these sort of uh desirable properties um so that's what I'm doing in my PhD um I got three years left so I'm sort of trying to make some some progress on these things um there's a lot of interesting stuff that goes into that green box that's a separation mechanism that tries to separate these information streams um that's highly non-trivial to to design um it's also a question of how you know that the uh the the C and the S signal streams that they contain the information that you expect them to contain um and you know how do you we've talked in in very qualitative terms here how do you make this sort of mathematically rigorous um so so a bunch of open questions but they have shown a great deal of Promise um they're good at adversarial attacks they're good at generalizing doing stuff like training on one data set testing on another data set uh those types of tasks um and I'm quite a quite a fan of them as you might have guessed otherwise I wouldn't have spent four years of my life sort of researching them um but so hopefully that sort of piqued your your interest in in how these can be used to to make more secure um AI systems thank you [Applause] all right this was a short talk but we we finished early so anybody have any questions all right let's go hi um I remember reading a tweet from some cool guy and neural network some time ago I think it was the only but I'm not sure yeah he tweets yeah probably so basically what he said is that any attempt to make your neural network smarter but trying to explain it how to thing is failed uh you know is bound to fail eventually the bigger neural network with more data will win so it's a failing strategy so what do you think about that and also another question is uh when you showed the first adversarial attack with a some crafted noise um my first what I thought it was maybe it's about how we access this data humans that you know for us it's a little bit blurry we don't get access to the individual values of those pixels so why not just make the image a little bit blurrier with some random noise in other words just put your perturbation step as the first step and that's it why doesn't that work thanks yeah yeah uh okay so to answer the first question first which is um can't you just solve this with sort of more data um the answer is yes eventually if you have sort of infinite images you're guaranteed to get a uh robust or a secure uh neural net but I think the question is more about how quickly you can get to that step because if it requires sort of it's very possible that it requires more images than Humanity will ever produce in its lifetime and then it's sort of unattainable um so the I think the causal neural network texture and first is sort of some guidelines on you know how you'd like the information to be processed to use that data more efficiently uh because you know with perturbing your style you can basically turn one image into 10 or 100 so it allows you to sort of extend the data that you have um I also want to say that young lacun has published a full Manifesto about how you should make AIS think so you know who is he to talk um um to answer your second question um about why don't you put the perturbation step as your first step the that's a good idea it's a it's an established strategy um which we call adversarial training and it basically means you do this to your training set and then you train um the issue with that is that it sort of makes you good at defending against a specific type of attack it makes you good at defending against the attack you trained against um but there are sort of many different attack algorithms and you need to patch each one and if someone uses a different one that you haven't trained on then you know you're a bit screwed um so so that so that's a practical solution for sort of patching problems but it's not it doesn't solve the underlying issue uh do you apply it to the uh the image and the image is the input to the as you well you're applying it to the pixels or to the label yeah yeah oh yeah so um uh you mean like before you train you add you blur your image and then you train so that it's not sensitive to being blurred right is that okay okay we can discuss the ins and outs of adversarial training um yeah thank you very much preben and he's here hopefully for the rest of the day yeah yeah so uh feel free to take up this and other discussions all right thanks [Applause] all right up next we have Swan Bouchard and Gautier Ben m to talk to us about something completely different this is the the fast-paced part of the day where we go back to back shorter talks all right guys take it away how do I get in there we'll make it work where's the pointer yeah okay pretty good all right um hello everyone so today we will talk about uh graphql security and we will dive in the report we published about the topic we also have a few surprise at the end so stay t