Creative Hacking With Blender

good excellent thanks for coming my name is Brett Patterson I am not a computer scientist in any way and one of the things that made me a little bit um nervous about showing you this presentation is I know a lot of tech people will wince of what I'm about to do uh and uh I'm doing uh this this basic using a graphical debugger to inject a process into Chrome and so it's not the most secure thing to do and and Google doesn't like it but I made a music video a few years ago and uh so I I do weird experiments all the time if you've never seen my website um I I do uh like volumetric tests I do lots of stuff I've been using blender for about 16 17 years and teaching it at the University level as well and so I uh currently I'm doing lots of projects where I uh use procedural volumetrics to create basically a nebulas and things like that for planetariums but I also just do weird stuff because I'm I'm an artist um but this kind of stuff is my some of my favorite kind of stuff to do but a number of years ago I was doing experiments with um photogrammetry and not iPhone photogrammetry and lidars and things like that but rather using some software called mesh room it's from a uh a software Consortium in Germany called Alice vision and so all of the links and everything are laid out here in the handout for all the software I'm using but Alice Vision they have a bunch of free and open source tools for photogrammetry and so one of the softwares they have is called mesh room and what mesh room does is it just takes any number of photos and tries to stitch them together into geometry and the process takes a little while so I ran the process last night entrance which basically show how yeah yeah I just accidentally opened up another version of it but at the core of what mesh Moon does is you just dump a photo set into it so I've got a bunch of photos here of the garden from behind a pub specifically this was a prototype that was sent to me um by a a film studio in London they reached out to me two years ago to work on the music video based on the test and everything that I've done they at first just asked me how I did it he then asked if I wanted to be a consultant then asked if I would produce the video so it sort of escalated and and because it was in the middle of the pandemic I I I I I was working from home anyway it was uh it was in between semesters I said let's go for it and so I spent two months working remotely with the film studio in London producing a music video and so what they were doing we did a whole bunch of tests at first and then they sit and started sending me photo set after photo set and what what mushroom does is it just looks at all the photos and then postulates where the camera was in the photos and then creates this point cloud and it looks a little complicated but this is really just the default settings of mushroom you just take a folder of photos like I have set like this photo set right here is just a whole bunch of iPhone photos uh from uh that they sent me from the UK and this works for anything and then uh and then you just chew on it for a little while uh with mushroom uh you just load all the photos in here and you don't have to change any of the settings but you can tweak them for all sorts of different settings each of these little nodes does different things at first just grabs the data then it processes all the photos looking for feature relationships and uh and matching and and then it builds a structure from all of that yeah it doesn't have to have like the cameras like uh structured location or like uh you know spacing and all that or is it only looks at the images it doesn't look at any image so the camera placement doesn't really matter it doesn't look at any of the metadata in the photo it only looks at the images so I it doesn't just have to be photos you can export an image sequence instead of just photogrammetry you can also uh generate camera tracking so if you're doing any type of visual effects which is sort of my specialty it's really good for that it's completely free program and it works really well but it doesn't work perfectly and that's the thing because photogrammetry isn't exactly perfect but what's really great for are stationary objects statues objects and things like that so if you're making a video game I think is fantastic for assets uh so if you wanted to just take a picture of a car just take 50 pictures of a car going all around it and it makes a pretty good mesh of that but if it's a person people move and so it doesn't work very well but I like that because I I'm an artist yeah distortions that also mimic reality yeah so does this software have some sort of like death perception mechanism or do you have to get photos from all angles you have to get photos from all angles and otherwise it'll just generate data from what it sees so if you don't have any of the back the back could be flat just be flat it'll try to extrapolate that it'll import coordinate the background with the and if there is no background then that sometimes can help but [Music] like click on one of these it should texture it and mesh room only works on a machine with a Nvidia card so that's kind of a downfall so it's just if you've got just the integrated GPU or just an AMD card not just but if you just it unfortunately only works with Nvidia and so it doesn't have to be like uh cards or is it not necessarily when I did all these projects a while back I did it all with a 2070 and I did it all with Linux and so it worked perfectly well with Linux and I and I produced the whole music video on Linux uh so here is uh the mesh and then you can see where it sees all the cameras and and then um yeah it's never quite perfect but uh it does it does quite and depending on the photo some things will have quite a lot of detail and other things won't um but the process is super simple and so if you're interested in doing these kind of assets it works and and so basically you can just take this texturing area right here and you can open that folder and inside of that folder uh you'll see here is uh the obj that it generated from that note so each of these nodes can export different types of files so then you can just take the contents of that obj and then you can import it into blender and so if you is anyone in here ever use blender maybe you have to make my students use it I've been working in with blender in at the academic level like I said since 2006 and I've been working with a lot of people over the years who are like scared to death of of Open Source software and uh and I and I'm if my students know I'm a big advocate for it because I think it just it's a I gave a talk at the blender conference a few years ago called Don't Go Pro look it up it's about how software markets itself in a corrosive way it's such that people think that if they know the software then they're an artist or at least a professional and I I do issue with that but Let's uh let's see if we can import that now the machines acting really slow which has got me very nervous I don't know why it's running so slow I might have to do a restart because this is normally a pretty fast machine it has a 3080 on it but that tells me something's wrong and the way it's becoming not responsive which I did a test before and it's working fine but it was just a lot of things this is always how it is yeah it's just a live demo this is good we have to go out of the room and then it'll work fine let me just do a real quick restart because I don't trust what the way this thing's behaving and I apologize I can take this opportunity to answer questions because I wanted to show you the music video and stuff that I made with this uh and uh it's gone on to win some awards and generate a little bit more work but I'm I'm currently writing a book for a publisher who reached out to me because I'm not busy enough about all the tips and tricks I've I I've come to use over the past 15 CC here have been teaching blender so it's just going to be 100 tips that it's an intermediate level book and uh and and I and I'm a good writer but a super slow writer and so I have this sadistic um quality about myself where if I see a deficiency I put myself in a situation where I have to overcome that deficiency to make myself stronger in it and uh one would think after 15 no 20 years as being a teacher I'd be a better speaker in front of people but I wasn't good at speaking in front of people so one of my first jobs was working at a radio station and to overcome that problem and it didn't help but I was I was the worst radio DJ when as soon as the mic came on I couldn't speak and I was uh uh I was only 16. but I would have to write everything down to know what I was going to say um but I just put myself in bad situations um were you going to say something I I have one question yeah um it's working it seems to work really neat with flat files how would it deal with something like say a photosphere a stitch a stitch 3d full I I should have brought up the examples of that because I did use I did use uh 360 recorded video as the tests and it worked um my buddy uh we were going to do parts of the music video in the London Tube and so we took a 360 camera and just ran through the trains yeah and uh and uh and it worked it worked it didn't work with a whole lot of detail because the 360 camera it was only 1080p okay and what most people don't understand is that naap is for the full fear right not just any single frame of the view um I take ditched 3D photos with 16 megapixel camera so I get like you know five five thousand by two thousand right photos here so I just recently ordered a a new 360 camera that does 4K video yes and it so each section will still only be like 1080 or so right but I'm hoping that we'll get better results from that okay so the higher and one of the other issues is when you're dealing with 4K uh Stills that if you're on a on a lower quality GPU it won't work it'll just say out of brand video around so it helps to have a beefier GPU to make this work okay and for the music video since I was on a 2070 and I think the 2070 had 16 gigabytes of RAM I had to scale down the photos that were taken with the DSLR and it were like 6000 pixels so I'd have to scale down the sequence down to maybe 3 000 pixels okay and then the processor could work and but again you would have less detail in the final processes that came out of that let's see if that worked notice how I I took up the time of the reboot and it doesn't just keep him quiet let's think of what camera did he get what's that what camera did you go on I can't remember it was the one of the top rated on DNH I had a quick uh ask on an admin I got some money then buy something what do you want this and I was looking at the insta360 yeah did you hear our experience with them not too much you know most of the photogrammetry work I've done I've just done with Android phones and yeah but now that they're almost being coming in yeah like a commercialized device um so anyway Let's uh let's try just check it out in a blender here and see if we got a more responsive machine this time then and this machine's been acting a little bit funny ever since the admins did an update to it that's better that's another rule is never doing update before presentation that is that is uh some hard truth right there uh when I was in completely in for for four years I did this experiment where I only used Linux and in that world updates were always good uh at least for me but since I've switched over to this Windows machine because I just go back and forth um wow that redness is terrible I'm sorry it updates are much more of a Russian Roulette so let's go here to [Music] and I am just getting over a food poisoning episode this week so I'm still a little not myself don't need a Bob Evans in West Virginia that was a mistake foreign so I'm navigating to get in there oh you know what I didn't texture that one or did I that's something that could be different let's go over here too I'm sure we do yeah and I did rehearse this I'm glad they gave you 50 minutes so like I said if you right click come over here because this is only the first part of the talk the other talk was hacking Google Maps so Boulder besides measurement one it's in that it's in that wonderfully named folder right there you should I don't think you can import by default you can import obj's into blender and Export all these different kinds of formats so it's very formatting uh okay let's go back so these cache files that it generates get really big so the more you do with these it will fill up your hard drives most of these cache files can sometimes if you have a thousand photos in there the folder will become 30 gigabytes sometimes 80. so it I normally do this on an external hard drive after you get the obj and the textures you can delete everything else but the process itself takes a really long time and depending on how on the settings that you have in measuring the geometry can be really huge so if you're doing this for building Assets in a video game you'll want to read to apologize the mesh and blender has a lot of tools for doing that and there's some pretty good plugins these days that we automatically do that but the meshes that meshoon generates are not appropriate for like characters to animate because they're filled with triangles instead of quads and if you know anything about video game 3D stuff triangles are the double and so you don't want to do that but so I think it was this one and so you can see this obj is 386 megabytes for this Garden but then we can import it [Music] this is it so this is the mesh if we switch over to shading it'll load all the textures it takes a moment and so here we are so this is the garden Pub and when we did this and this mouse is done because that's what I chose to use today and so when we did this in the music video I could add lights and we added other picnic tables and then we photographed people individually to fill the space and so it worked really well specifically because we were leaning into the gnarled look that this geometry generated we didn't want it to look perfect we wanted it to look um because the music video itself was about memory loss and losing people and and so forth so let's come back over to that music video and I gave my talk at the blender conference last October with my collaborator about that music videos production so I'm just going to show a bit of that music video because it's a it's a long one and it's a it's a guy named David Baff and it's an Irish band called for those I love um and we spent countless hours color correcting and working on the color of the lighting and everything and then the record projects and I couldn't get it a little tender it's not fair it's not the song so we would take all that photography and put together some really large seeds and then I would fly to China and right here please [Music] [Music] [Music] [Music] [Music] Africa [Music] [Music] simple [Music] [Music] but foreign [Music] [Music] [Applause] having people like moving around larger and smaller to make it you know psychologicals there's a lot of details [Music] no no if mushroom has tons of seconds and you get really overwhelming and so you can squeak the settings and how much detail is possibly extracted there's several ways to get different people out of it one is to feed in the full res DSLR put a little bit a lot of detail that way but people move and if they move a new ways they get all this wonderful Distortion uh or you can scale down to photos and more consistent but there are scenes in uh the raid into the surrounded until the next day yeah the raid right here um where some of the uh we have too many characters to put in so we did cheat a little bit and uh we did use a iPhone an iPad lidar to do some characters really quickly and so because when mushroom processes these photos and these scenes depending on the photo set like the alley that we did right here with the alley it was an alleyway it oh it's after the ring that's right this alley had 1 000 photos and the person just walked down an alley and uh they call it a lane in uh in Dublin and uh this actually had a lot of detail more than we want Danielle yeah and then I ended at it manually I just created one light post and duplicated it over and over again and then put a light inside of each one so that the lights would animate coming along over time so then then I would dirty it up by using some remesh and some displacement on the geometry to make it to make it fit to make it look like that it was more novel with everything else I don't know if that makes sense but yeah so um you use mushroom only for the objects right like and if they have them console and then everything else is just the photo I use mushroom for the sets I use mushroom for people and networking for any object that might be in there occasionally we used iPads and iPhones with lidar and generate some of the geometry but generally speaking that System created more Lobby shapes and less garbage shapes and so uh I I don't know that's not real professional on Geometry does that make sense yeah yeah so we used a whole bunch of different tools and and all the rendering was done in blender and then a studio in um in the UK edited all the renders together and then another Studio did the color correcting which took out all the color and and that was that was a basic project uh here's a behind the scenes get a better idea so you can see all the photos that we use for it my collaborator um who's never used Linux or anything before he he used a I think it was just a MacBook Air so that's where you saw some Mac OS extraction you modify any of those like images you I know you kind of purposely were going for that broken up look so that some of the wall was locked and things like that modifying videos yeah after we after it was in blender material right there like a pub here it was way too big so I cut the pub in half and then and that's what you just saw in the video and then I would bring the walls together to make the pub seem smaller so I could change the geometry and because we were just embracing the gnarled look I didn't have to worry about the detail and because we were lighting it because we were lighting it very dark that that helped too I could just add things people only look at what you put the light at and so they'll just assume that everything else is all put together and this is this is true in all filmmaking and all art that you can direct attention so that all the garbled and messed up stuff and people won't even notice they only notice what you're asking them to see and so where the camera is pointing and where the camera is moving will will direct attention and all the other stuff just gets filled in and selected [Music] [Music] but some things wouldn't show up and so that Randomness is really wonderful for people it's harder to you know capturing all the images because people moved if for somebody said you have to tell them to hold still while you were getting the images or oh yeah yeah you have to tell them to hold really still and while we would move the camera around and not just 360 but up and down and all around so we could use it but when when people didn't hold still their limbs would just go missing so the way measurement works is if they can't figure it out it just won't put it in there um and so Dad bought was the company I did that with people uh and so uh the whole point of this is ask yourself whenever you're approaching any software is what was it meant to do but what else can it do and so in embracing the weird artifacts of the software instead of trying to hide them is is the fun part and it actually makes it possible and and that's that's sort of my motto with a lot of software is if we use it wrong what interesting results uh do we get and so I really enjoy that process