BSides LV 2023 - Breaking Ground - Tuesday
Show transcript [en]
[Music] thank you
[Music]
[Music] foreign [Music] foreign [Music] thank you
[Music]
foreign
[Music] foreign [Music] foreign [Music] thank you [Music] thank you [Music]
[Music]
[Music] foreign [Music]
foreign [Music] all right [Music] laughs [Music]
[Music]
[Music] thank you [Music] foreign [Music]
[Music] thank you [Music] foreign [Music]
[Music] [Applause]
[Music] foreign [Music] thank you [Music] [Applause]
[Music]
[Music] thank you [Music]
baby [Music]
[Music] don't leave me alone [Music]
[Music]
giving me Wind and Rain some kind of butterfly baby [Music] [Music] oh [Music] but I don't wanna miss you baby [Music]
[Music] thank you [Music] baby [Music] don't leave me alone [Music]
[Music] foreign
[Music]
[Music] oh [Music] my God [Music]
[Music] all right [Music]
[Music]
[Music] okay [Music]
thank you [Music]
[Music] moving up all over
there
[Music] thank you [Music] I don't know
[Music]
[Music]
[Music]
[Music]
thank you
[Music]
[Music]
[Music] thank you thank you [Music]
[Music] thank you [Music] thank you [Music] foreign [Music]
[Music] thank you [Music]
thank you [Music] thank you [Music] foreign [Music] foreign
[Music] foreign [Music]
[Music]
thank you [Music]
[Music] foreign [Music] foreign
[Music] foreign [Music] thank you [Music] thank you [Music] foreign [Music] thank you [Music]
foreign
[Music]
[Music] foreign [Music] [Music] thank you foreign [Music] thank you [Music]
[Music] foreign
[Music] foreign [Music] thank you foreign
[Music] [Applause] thank you all for your patience as we figure things out again here that can we do every year but uh I have to say it is really really good to see all of you here again it's this uh the the vibe the the energy having all these bodies back in this space is is just um it's a tonic for me uh we've been doing this a long time now and uh it feels like we're getting back to something that we've kind of you know have had missing from uh from my life at least for the last couple years so that's down to you folks thank you for being here um on that note I do want to say a couple
things uh we do have uh masks available out at registration and at info Booth if anybody is interested in those it's not required but again highly encouraged there are people who do depend on this you know for uh uh for their safeties so up to you but uh that's there for folks if they want it now now I say we've been doing this for a while one of the things that we have been doing for a while is uh I am the Cavalry right this is a a movement a ground swell a a Bottoms Up you know answering the call and the need of the society that started here and has kind of grown with us it's grown
Beyond us it takes place many different places many different things during the year it takes place in you know small ways it takes place in the halls of power in Washington DC but it's something I'm particularly proud to have like been able to in a small way enable through our participation here um it's been 10 years since that happened and a lot has been accomplished uh and I think it's nice to look back and reflect on that so our keynote for this first day is going to be Josh Corman one of the founders of that movement uh talking about kind of how we got here uh what has been accomplished during that time and what the future
looks like uh and it's just one of the things that we do here I I I love uh what Jack Daniel has always said all we're doing is changing the world right one conversation at a time one talk at a time one handshake at a time and it's the the engagement and the openness in the community here that that makes that possible uh I love to just just the welcoming vibe that you all provide so please if you can do anything while you're here for the next two days make a new friend haven't conversation with somebody that you've not had a conversation with before this is our our chance to do that again after you know years in isolation
so uh just I I want to applaud you for what you've done so far and for what you're going to do this week thank you
and without further Ado uh I'm going to uh get out of the way because other people have more interesting things to say so please join me in welcoming Josh Corman uh to give a talk about how we cross the river together foreign
get this to work again um hi
so uh warning I'm feeling a lot of feels today so uh I'm not even sure where we're gonna go with this but we'll find out um I can promise you this it'll be authentic um so I probably should explain the title and together we cross the river but I'm probably going to postpone that a little bit um so it was uh August 1st 10 years ago after being rejected from Defcon uh Nick and I offered a talk title vocabulary isn't coming and it was a conversation it happened in this room configured a little differently and uh Banshee and Jack and Damon moved Heaven and Earth to make sure that this important discussion happened and uh
and after some circulation of how the hell did this not get picked a dark tangent and gave us the keynote stage Sunday morning a couple days later at Defcon and I think we're still in one of the top 10 talks of Defcon um not because it was a flashy o day but because it really tapped into purpose in a North star and how important what we do could be to our families our our societies Etc so I'm not gonna they were in the build up to this I had several competing theories one was I wonder if I could give the exact same talk and how it would hit a decade later so I want you to think back to 20 I'm not
going to do that um I want you to think back to 2013 okay I don't know where you were living I don't know where you were working I don't know what car you were driving like I don't know what evokes your memory there but people were pretty pissed off there was a trend towards the increased criminalization of research hacker was a dirty word uh Snowden kind of shattered trust kind of amongst the community amongst Guppies President Obama said I'm not going to scramble some jets for some hacker um there was a lot of concern in existential dread amongst the hacker community that year um I felt moved to do this for some very personal reasons and when I went and
watched the video yesterday um some parts hit me pretty hard and things I remember saying in that room I never actually sat in that room so I might add some color and context today but uh and that will help make sense for why the talk is titled this um but what I did say in the room is that our dependence on connected technology was growing much faster than our ability to secure it in areas affecting Public Safety and human life uh and after doing a whole lot of looking high and low in the government and for the adults there weren't any and it was incredibly demoralizing to see that the Cavalry isn't coming to save us
after researching Anonymous for a couple years with Jericho and being concerned about the rise of personal power curve of the individual in a hyper-connected world the corollary to that is all right if they're powerful so are we and if no one's coming to save you it's also empowering because then you know it falls to you it's you or nobody so you don't feel helpless you make a choice am I going to fight or not so the call to action was what are you willing and able to do can we be that voice of reason that technically literate uh honest broker can we be a helping hand instead of a pointing finger can we can you transcend the rock
star culture and the glory and ego culture and instead try to solve real problems and instead of bringing a pointing finger and anger can we bring empathy and a helping hand instead of taking a tactical view of finding and fixing a single flaw in a single medical device from a single manufacturer in a big contested public debate could we hack the incentives so that all medical devices were safer and we had no idea if any of this would work I I think I was at my feeling my most powerless and shattered when we made the call and a decade later why does the world look different so I don't know your personal touch point for 10 years ago but to bounce
between then and now quite a few times hackers are cherished in public policy circles we have government officials here today we have a defcom policy track we have a black hat policy track we have hackers on the hill we have hackers in the White House hackers helped write the White House National cyber security strategy hackers have passed federal laws hackers have passed UK laws hackers have influenced transparency regimes across the globe we went from a full disclosure Mantra now to every Federal agency has to have a coordinated vulnerability disclosure program now that was not an easy Journey from 10 years ago where hackers were increasingly criminalized to having good faith research carve outs for dmca and
cfaa and being invited to White House meetings and testifying to Congress and I have no idea how we did it I mean we can give you data points you can come to our track today and tomorrow and you can hear some of the success stories and some of the blueprints but Beau woods and I joked that if we ever wrote a book it would be called we have no idea what we're doing but it seems to be working but I think when we reflect on what worked and what didn't um something like this something as transformative this could not have happened outside of the b-sides family hackers are not a single tribe we're a tribe of tribes and sometimes warring
factions but this is the community that thinks of their place in the world that wants to Mentor others that wants to give people their first speaking I mean the very birth of b-sides was that first time speakers could never break into the cabal of the gatekeeping of the black hat conference we're hearing the same things over and over from the same rock stars yes they're accomplished yes they're amazing how do you get new blood and new Talent and it's very hard to break through so one of the spirits of b-sides was to democratize that power instead of hoarding that power to inform Inspire and influence and help and shape and cultivate and only this community could have given
us that Refuge to ask this crazy thing 10 years ago all right I've been rambling
this journey for me did not start with this talk
some of you know this I'm going to give a little bit of it not all of it a little bit of it
so there's a lot of fields in here for a lot of different reasons
I had researched the rise of anonymous and hacktivism I was concerned that this was a significant moment that it was going it was the front line of what happens when large groups the post-national use opt out of social contracts and take direct action online it's an emergent property of the internet I felt that it could erode social contracts because I'm a philosopher hacker systems thinker idiot altruist who spent way too long in the hacker Community but I was worried that it may Inspire things like cyber terrorism and it did uh a team poison member from UK named Janae Hussein a Pakistani UK honor student from Birmingham joined team poison hacked Tony Blair's website got arrested went to jail
and in jail was radicalized and when he got out he moved his Anglo-Saxon punk rock wife and child to rock Assyria where he founded the Cyber caliphate and uh was recruiting an inspiring physical attacks in the U.S and abroad and hiring hackers and I was terrified at the concept of what could someone willing and able to take human life do with showdan script Kitty tools and the answer is a lot so I kept these to myself and I tried to find the adults in the room and I tried to whisper to people in government in the intelligence community and allies and eventually I'm going to skip ahead for what happened with him but they eventually put him on the U.S kill list
and I think he was number four the most dangerous person for our interests and he was eventually killed by drone strike uh in raqqa but I was worried in a world of seven billion people it doesn't matter what most of them would do it matters what one of them could do and it was way too easy to reach out and touch someone so after successfully spawning false flags and predicting movements um the intelligence Community said how are you doing this and we showed them how and I got invited into Fort Meade for two days and I got to pick five hackers and I figured each one of them are really strong maybe we can form the team
of Avengers maybe we can see what we could do and the goal was to help General Alexander figure out his attitude on different legislative proposals for cyber there was no cyber security framework at the time there was no didn't yet in fact this never would have happened had it happened in the wrong order but we brought Kaminsky and uh HD Alex Hutton Gene Kim um David etchu and we answered some really really important and hard questions like if you could add one sentence of legislation to have the most material impact on Public Safety human life and U.S critical infrastructure and the Hemorrhage of intellectual property from the U.S economy to China what would that one sentence be
and it was amazing to see their powers combined um part of the fields is you know somebody who's been gate kept and never felt like I had something to contribute here I am talking truth the power to some of the most important people in the world and trying to put on their agenda that we're worried that we could have mass casualties and loss of life in our food supply in our hospitals and our power grids and I'm watching these people that are strong alone be so much stronger together and that's all great but what what else was going on in my life is my mother had had a stroke and uh we knew she'd have some speech pattern
things to fix but in between day one and two of the most memorable moment of my life what I thought was gonna be the Pinnacle of my impact on the planet if you want to Dent the Universe I go to my car uh grab my cell phone I had 18 voicemails saying I'm so sorry Josh I'm so sorry Josh I'm so sorry Josh and finally when I got to my sisters I'm like finally figured out what they were talking about but it basically it wasn't just a stroke it was pretty aggressive aggressive brain cancer so we knew that we'd be ending her life soon so I sucked it up went and taught a class went back didn't
tell my friends tried to do day two and we came up with breathtaking ideas and we answered all the Challenge questions and at the end of it when we did our readout the answers were we can't do that one there's no statutory Authority for that one people would have to die first for us to try that one you're absolutely right about this one but good luck getting that through Congress and basically at the end of the readout we couldn't do a single one of our transformative ideas not even one so it was both magical and demoralizing and that was at the airport bar when none of us spoke for probably 30 minutes that I broke the silence and I said half
of the answer here I said the Cavalry isn't coming and we all got on our airplanes and we all flew home now meanwhile I didn't have the answer to the other half but we start hospicing my mom 58 years old trying to watch her die with dignity came a point where we had to take her away from her home to my sister's house more closely uh close by and all she wanted to do as a superintendent of a school district and a very active member of her church to say goodbye to her friends one last time and uh it [ __ ] luck it happened to be the Sandy Hook shooting weekend so she didn't even get to say goodbye to her
friends because everybody was shell-shocked all the teachers she responsible for all the students everyone's afraid so for hours in that church we just heard her preachers say why is there evil in the world why is there evil in the world and I just remember being angry and hurt and I'm watching my little girls afraid to go to school and watching my little girls hug their grandmother who's dying and it was you know one of the most gutting at bottom moments of my life and then we fast forward and we're hospiceing her for a little bit longer she dies in January I have to go back to that church I have to walk back into the place where
I last felt angry thank you Jack [Applause] I have to walk back into that place where I felt angry and I don't like to be angry I want to be constructive so I had to metabolize that and somewhere between walking in the front door and getting to the stage to give the eulogy because I was her oldest I uh I realized okay my mom got to be my seventh grade science teacher she was a phenomenal teacher somebody got hurt shouldn't have been allowed they made me had to make an exception and of the many things she taught me Darkness isn't a thing it's an absence of light cold is not a thing it's an absence of
heat so maybe it's not the presence of evil but the absence of good and maybe that's why I was so angry the last time I was there um so I asked her family her friends her parents her siblings her grandkids what is the absence of Marie and I didn't have an answer I just looked at him and I said we don't get to find out because it falls to us to do what she was doing now to get to something hacker related um I finished the sentence in my head and said if the Cavalry isn't coming if something's missing it falls to us to put it there so I didn't know if it would work
I didn't know if anybody would say yes didn't know if we'd have a single accomplishment but I knew it was worth trying Okay so the song I was shattered um I was given the keynote at B-side San Francisco and I enough nothing in the tank um and Jack my brother who just gave me a hug uh he had to scoop me off the emotional floor and take me to Wine Country at the end of this RSA week I couldn't even speak we just played music couldn't speak I didn't know if I'd stay in security I didn't know if I had any energy left and that Puscifer song came on The Humbling River is where this is coming
from and the whole idea of The Humbling River is this guy can conquer everything conquer climb the mountain win the war do all these things but there's one River he can't cross and over and over he tries and he's humbled because he cannot cross the river and as I'm listening to it feeling shattered and Powerless and that I've drawn everything I can and I don't see a path forward there twist the line at the end and says the hands of the many will join as one and together we'll cross the river so I didn't know what to do what to call it but I'm like all right I'm not done we've been doing this as solo artists
let's see what we can do as a team so I'm not doing good on time management I told you I had a lot of feels but that different approach where we weren't looking for permission from rockstars where we weren't looking to point fingers or have combat where we weren't demanding something but we were offering something had transformative results and I'm not going to show everybody's face for everybody's name but I asked Nick initially Nick Rococo you know if he tried this crazy experiment with me a law professor and your mutation have been coming to Las Vegas a lot during the rise of anonymous and Defcon she helped nudge me during thought con the prior year space Rogue whose cold dead heart
had closed off started like saying his heart grew three or four sizes that day so space road kind of became yes that's a huge boot full of beer at thoughtcon no beard you might not recognize them um but you know we're talking about what did and didn't work with Loft and maybe could we try something like that again um didn't even know about Woods Bo missed my talk he was giving a talk on how to dodge U.S tax codes by being a digital Vagabond and traveling the world didn't even see the call to action and has become the first and most dedicated and longest lasting recruit that has helped to change the world that's caused by the way
you know we met people like Craig Smith who wasn't to the Chris and Charlie rock star hacker but had written more tools and democratized more access and helped start the car hacking Village we traveled the world got sworn by camels um we don't we thought that that was our last dinner there on the on the left uh we started entering the halls of think tanks in DC as uh secret Invaders he became Dr Horrible to do the biohacking village we eventually ended up you know briefing in inside the White House on more than one occasion because they've started to realize they needed help so people that were afraid of hackers a decade earlier are now completely embracing us
Jan Ellis was starting her own thing she also didn't see the talk although she knew I was going to do it she was over at black hat trying to like say that hacking its first amendment protective speech and she was deeply concerned that her friends might go to jail and she decided she had to do something about it so she started on her own journey to try to reform CFA and dmca and very quickly we combine forces we started investing in junior staffers this is one of two this is Nick lazerson he's been here this is one of two Congressional staffers that first year with a computer science degree tomorrow you're going to see the other
of the two Jessica Wilkerson but we built trust with Junior staffers that most people would have turned their nose up to that man is now running most of oncd office the national cyber director in the White House more Jen some of these became family Jen was my best man in my wedding last year August 3rd to Audi [Music] um Jack married us um not me and Jen my wife does not like her photo online but Jack has been a brother in a often one picking me up off the floor when I'm emotionally shattered I have a lot of feels lots of hugs we're gonna make the calendar for charity we befriended sitting Congressman we brought two sitting congressmen to
Defcon 25 bipartisan will Hurd of Texas who's now a presidential candidate so I've had shots with the presidential candidate uh and uh Jim langebin who ardently fought to advance cyber security in the Congress he was founded the Cyber caucus in the house he drove the formation of sisa I want to remind you we are 10 sisa is five sisa was in part fashioned after some of the Cavalry mission to do defensive work for critical infrastructure for cyber physical systems and he helped birth CIS cyberspace learning commission and fought to the end of his administration he just retired but to the end to advance hacker rights coordinated disclosure and has been incredible teammate we befriended hackers who grew up going
to Defcon and became Physicians and we started cybermedsummit.org a non-profit to do ER hacking simulations and Christ simulations with doctors we worked with patients like Marie Moe who is both a cryptology PhD hacker and a heart patient who engendered empathy and gravity when we tried to reform public policy we befriended nurses like Malena of dongs International hacker celebrities like Karen who'll be here Billy Rios who hated this idea at first and was the one doing the prolific research and angry with the art with the FDA learned that coming to the table and finding common cause and common purpose took his previously ignored research and caused the first recall in history or medical device for cyber reasons an
unmitigated Pathway to harm the prior standard of care was somebody had to die first there would be proof of harm and enough proof of harm to Merit a corrective regulatory action but we convinced them that in cyber security an unmitigated Pathway to harm was enough and nobody had to die first like Mike left his own company to go to GE to make medical devices safer because he heard the call and he led and he built teams and he mentored people and he trained stuff and he led by example and when he thought he'd done as much as he could and went to look out he heard a talk after our Congressional task force and he's like
No One's Gonna fix this I gotta fix it and he started scope security left his career again to put his neck out in Advance Medical
and was also like Jack one of the people that picked me up whenever I Was Defeated
Dan was there before there was a Cavalry Dan stepped up behind the scenes in front of the scenes whenever we needed him too and he reminded the old school hackers that were sabotaging us and backstabbing us and gatekeeping us it's not about us it's about them it's about the people true hero
and a huge loss Damon has not perished Damon has been uh the supporter I feel happy I'm just gonna rifle to realize we had quasi-gummies like uh art Manion we have our honor roll of government hackers who helped us save the world like Alan Friedman who's here not only did Alan help on s-bomb uh and iot labels but also on coordinated vulnerability disclosure and had we not suffered the sling's arrows and attacks from some of your historically favorite rock stars we would not have had carve outs in dmca and cfaa we had to normalize and demonstrate value for coordinated vulnerability disclosure so Alan continues to take up the unpopular sexy controversial topics and make them
boring for guppies Leonard Bailey from doj specifically wrote the prosecutor guidance that if you feel like Prosecuting a researcher acting in good faith don't Suzanne Schwartz who will be here tomorrow has been an incredible hacker she even made her hair purple at one point she has demonstrated bravery Head and Shoulders among any others and every time she had a victory with us we were able to use that to cause pressure on other Regulators other executive to do what she was doing so she set the pace as a sprinting partner for this and uh our greatest achievements besides decriminalizing research have been in medical and specifically because of her and her teamwork in Partnership on changing the world we'd had some
Media Partners I only grabbed one but Lori Siegel put us in Time Magazine did two CNN documentaries on hero hackers and on the rise and fall of cyber terrorism and just took the time to lean in and make sure that the stories were told well uh Sunil you and I uh he's going to be your keynote tomorrow Sunil is one of the smartest men alive Bryson always reminds me right in front of me that sunil's the smartest cyber security person he knows we got invited this year to the UN General Assembly in January and it was the most surreal thing I can explain I had world leaders introducing introductions of introductions of introductions and every one of them was
some accent to some degree said something like our dependence on connected something something is growing faster than something something in areas affecting Public Safety and National Security something something and I had this oscillation of incredible pride and validation an incredible Crush defeat that it's we wasted a decade and then said but at least they're getting it now but then realize oh no they're going to go for information sharing first and then I and then I just kind of threw my script away and Sunil and I just spoke from the heart and hopefully we've saved them another decade of wasted time but like we have this community is finally a decade later in the International Security mindset
and of course our celebrity member of Dwayne The Rock Johnson that's a joke um he did say that and he spelled it correctly so my biggest regret about the Cavalry is the name um no but seriously um not only is it always spelled Calvary because it's a real word where they killed Jesus of Nazareth completely different tone than cavalry um but also you know we've lost something in the last 10 years um the cap I am the Cavalry was not Josh it was not Bo it was not Jen it was not the thousands of volunteers wasn't the early adopters like Adam Moran it was meant to be something you said like this was your personal Community
this is not a spectator sport and while we do love the praise or the of the thank yous we get sometimes what we really want is your participation because some of the biggest contributors were not Elite haxores they were nurses they were policy lawyers they were Junior staffers and uh in the last 10 years I fear we may have become a crutch so as I asked myself the 10-year Mark what do we do with the cavalry is it mission accomplished did we succeed can we end it do we transform it to solve the new missing pieces and take on a new Mission or two or do we combine it with other initiatives to get to critical mass in
the last decade we've not taken a penny of funding it was a choice often debated but I wanted to be free of any sort of appearance of conflict or any way for someone to add hominin dismiss our efforts how am I doing on time terribly right
I don't know what that means five minutes oh my God okay so we're not going to do what I intended to do okay um so so verbally here's some accomplishments okay and this is not a Josh thing we did this we crossed the river okay we said we focused on wherever bits and bites meet flesh and blood and that meant any cyber physical systems we started with cars we published a five-star Automotive cyber safety framework on our first birthday it said anything all systems fail you should avoid failure by taking uh having safety by Design to avoid failure coordinate disclosure to take help avoiding failure capture study and learn from failure prompt an agile response to failure and
contain an isolate failure a year later we did a Hippocratic Oath for connected medical devices to work with Suzanne in that same year we got the first ever recall and nobody died because of the trust we built we put pressure through Congress on Nitza the national highway Transportation safety administration to try to regulate cars similarly she changed the pre-market guidance to bring medical devices to start requiring cyber security things she later changed the post Market to encourage coordinated vulnerably disclosure gave them an incentive that if you have a coordinated disclosure program and you can mitigate your issue in 30 to 60 days then we won't give you a recall there's a little more to it than that
that work engendered enough trust that when the nation asked for a congressional task force on health care I was the one and only hacker at naming to that 21 person task force we told Congress this is not about your HIPAA privacy I love my privacy I'd like to be alive to enjoy it and we essentially pivoted them from a data privacy regime to a patient safety regime we started that task force with a the first dramatic attack on U.S hospitals it was Hollywood Presbyterian hospital in early 2016 shut down patient care for a week that you had to cancel surgeries divert ambulances nearby facilities in LA traffic it was harrowing people may have died but they didn't
measure it right and we ended the task force with wannacry shutting down 40 percent of UK hospital Health Care delivery so we were trying to add gravity uh to encourage more tight collaboration with us and we're going to skip a bunch of stuff in the middle but the Mirai botnet happened and it showed that even cheap consumer iot stuff could shut down the internet for a day so it was unpatchable Internet connected unpatchable with default passwords and Senator Warner spent hours with Bo and I and crafted this the iot cyber security Improvement Act of 2017 which failed thanks to lots of lobbying but following Congress during the pandemic it was reintroduced in a watered-down way and in December
2020 while I was at my low running ciscova task force it passed in the [ __ ] law Packers passed a [ __ ] law [Applause] also when the globe hit a pandemic some of you don't even know I did this and I have scars the rest of my life from doing it but director Krebs of the newly minted sisa fashion in part in our image when the pandemic was declared he asked and called and said do you want to serve your country for a year um I don't know what you do when you get a call like that but I became the chief strategist for the cisacoa task force and our job was to protect the 7 000 hospitals in
the country during record high Ransom activity from a larger volume and variety and record low supply chain resilience and then we asked Scott asked to protect the vaccine Supply chains so I did that for 18 months to the day and a bunch for free on both ends and I'm just kind of traumatized but I'm going to give you a hurricane tour of a couple things those are the good news okay don't make me forget the patch act before he gives me the hook okay here we go it's the idea of the Cavalry is no one's coming to save us what are you willing and able to do generally speaking whenever I testify I say something like
we are over dependent on undependable things in areas that can cause loss of life over dependent and undependable things some context um many of the Cyber physical systems that are exposed or what I called it cisa and now it's one of the best things we've ever done by the way is hack the Lexicon the number of things coming out of public policy officials that we uttered uh you hacked the Lexicon you hack the world you got to change their mindset and reframe things so one of them was called Target rich side Rapport building on Wendy nathers classic living below the security poverty line this was a phrase they could they could stomach and the idea is forever bad guys
targeted The Fortune 100 and Fortune 500 why that's where the money is and forever the RSA conference floor in the black cat conference floor would Target the same people because that's where the money is ransomware changed everything because the unavailability of anyone can be monetized so adversaries figure out how to monetize the cyberpore Defenders still have not and the result of that is we are seeing disruptions on a regular basis at the bottom of Maslow's hierarchy Food Water Shelter safety so when we started the Cavalry I was worried that things were flammable and I wanted people to see that hacking is not just [ __ ] credit cards it's not just [ __ ] privacy it is Public Safety
human life and during my time at system we had successful hacks of the water you drink of the food you put on your table of the oil and gas pipelines that fuel your economies and your supply chains of the schools kids attend to the municipalities who run towns and cities of federal agencies charged with National Security and of timely access to Patient Care during a pandemic would now proven mortal consequences my team published proof that Ransom attacks strain hospitals sufficient to lead to loss of life the federal government is broken this is 16 silos of designated critical infrastructure written by ppd21 doesn't don't read them there's 16 silos they act like silos each one of them has
an owner in the federal government a sector risk management agency each one says stay out of our lane why are you in our lane this is our lane even though risk is inherently cross-sector there's military grade enforcement and as a hacker in a federal government during the pandemic we broke down every wall and barrier we could and I got the scars to prove it but it's not built for collaboration each one has a public-private partnership which usually means a really dominant private sector tells a tells the government don't regulate us and a really weak sector risk management agency says okay but that's changing then came Sissa Sissa was like hey you guys should not compete with each other
to hire and train and retain cyber security talent and physical security Talent so they became a shared Workforce they also said hey you can't manage risk at the sector level you need to do these 55 National critical functions I'm not going to explain what they are one of them is provide medical care can you get timely access to care when you need it where you need it well in the before times we learned from our empathy that a 4.4 minute longer ambulance ride during a marathon had a statistically significant mortality rate so four point four minute delay is enough to lead to loss of life for heart we know from Strokes that there's a golden hour a golden hour is
it one three or four hours time is brain is the difference if you can walk in or talk again if you breathe in our Congressional task force report we said Health Care is in critical condition published Mother's Day weekend in 2017. the hospital says we can't afford to protect it we don't need money if you gave us another five million dollars we'd hire more nurses and we said you both can't afford to protect it and can't afford not to but they didn't get it so we said until they said until people started dying we're not going to listen so we did what good hackers do and Christian demaff Jeff Tully Beau and I started cybermed Summit we started
killing people in ER hacking simulations not for real and we knew that stuff but during the pandemic everything changed okay so I did not make this graphic my friend Ben did but when we went into the belly of the Beast for and Beau Keem as well and a couple other hackers we discovered the vaccine Supply chains protect hospitals I'm going to skip the ball bearings stuff protect hospitals um generally speaking hospitals and stakeholders want to keep people alive how do you do that you need carrying capacity how do you get carrying capacity it's three s's This Is How They see their worlds we have to meet them where they are space supplies and staff such that if you have a hundred bed
hospital you don't have 100 beds of capacity that's your space if you can only staff 80 of those hundred beds if you only have supplies for 60 of those 80 you have a 60 bed capacity so it is the coefficient of the three S's and that's all they want to spend money on especially under Financial constraint but I tried to enhance that and enrich that because as we tried to keep people alive during the pandemic at the one year mark of the pandemic 150 000 people died from excess deaths from non-coveted conditions and my instinct was I'll bet you these are time sensitive like heart brain and Pulmonary and unlike the 500 000 coveted deaths these were young people the
fastest growing demographic was 25 to 44 year olds young people who would have lived but for timely access to Patient Care disruptions so I enhanced their model and I said it's not just keeping people alive what are the latency sensitive think like hackers what are the latency sensitive things where minutes or hours or difference between life and death and also they didn't realize this but the medical technology is a force multiplier of Staff a neonatal Intensive Care Unit nurse in 1990 could handle a single digit number of babies concurrently safely but armed with a bevy of modern technology they can handle 15 kids at a time if they're a remote monitoring stations so if the technology is a force
multiplier of the staff then the unavailability of that is a force divider and what they couldn't understand is that unavailability dramatically affected patient care for the most time sensitive and Urgent Care which is exactly what happened in the first proof of loss of life on October 1st this is not the baby but October 1st of 2021 front page of the Wall Street Journal revealed a court case that's ongoing where a baby lost their life in Alabama when the hospital was ransomed and the unavailability technology compromised the quality of care and the nurse to Patient ratio is sufficient the baby subsequently perished in this neonatal Intensive Care Unit there are more than a dozen connected technologies that are
vital to the delivery of safe care for those patients took caregiver ratios and when they go away it affects the patient on the very same day with a named victim of a cyber incident we published the first statistical proof of loss of life using data science and I'm not going to do the data science now but basically we saw a strong positive correlation between excess deaths and ICU bed streams so when hospitals got over 75 percent nationally of their ICU strain you saw eighteen thousand dead Americans in two weeks if it got to 100 you saw 80 000 dead Americans so when we say we care about saving lives this is where the rubber meets the road folks
and in lesson until policy makers could understand that a cyber disruption can strain a hospital sufficient to lead to loss of life so we took this measurement it has nothing to do with cyber when we applied it to a state hit hardest by ransomware and in the same state with the same population adjusting for uh Hospital type and size we could see that the affected regions achieve these excess death stress levels sooner and stay there longer than their peers and could quantify minimum maximum and most likely loss of life corroborated by state level data so now we have the first name victim and the first statistical proof of life and we can go to Congress and tell them you got
to do something about this and they have so this is a hot mess don't try to study it but basically what we realized is to provide medical care it's not just HHS and it's just their public-private partnership they depend on other critical functions from other sectors and if you take away water you don't have a hospital anymore you take away power you don't have a hospital anymore you take away supplies and transportation so back to Maslow's hierarchy what we realize is the way the government and not just ours the UK and Australia they're all listening to this new framing is that when everything's critical nothing's critical so we had to stratify so one way I did
it is latency sensitivity if you shut this critical function off for 24 to feed it 48 hours does anybody die and what you end up with is less than 10 of the 55. are latency sensitive enough to lead to mass casualties so these are some of them poorly plotted but provide medical care is probably the most important of all and they depend upon each other so any disruption in a dependency could affect your ability to provide medical care in a region and as people suffer excess deaths it's cutting into the workforce that allows those things to stay resilient so it's a positive feedback loop with negative consequences and because of these uncomfortable truths PPD 21 or
presidential policy director 21 which is the Obama era definition of the 16 critical infrastructure sectors and the shared responsibility models is not getting a refresh it is getting a rewrite informed from hackers and systems thinkers and hopefully we'll start to look at cross-sector risk but the overwhelming majority of those things I just pointed out are Target rich and cyber poor they don't have cisos they don't have security budgets they don't participate in public-private Partnerships they don't have someone to send to an ISAC or the money to pay for one and Emma's going to talk about some of these Target cyberpore in the talk and to start the Cavalry track but with each wave of the
pandemic we were further cutting into the workforce so Hacker's gonna hack I'm skipping a bunch of other stuff here but some of the leave behind so that we can live off the land later uh is I said screw uttering best practices and just do zero trust and just do this we need to talk about the bad practices so we named three bad practices things like the use of unsupported and end-of-life operating systems in service of critical infrastructure and National critical functions is dangerous it materially elevates risks to Public Safety economic National Security and human life this dangerous practice is especially egregious on Internet connected Technologies in other words if you're using an end-of-life operating system on Shodan
it could lead to end of life of humans so we wanted these things to be negligent number two I couldn't say [ __ ] and I couldn't say Shodan so instead of saying get your [ __ ] off showdan we publish get your stuff off search so if you have zero security play at least get your [ __ ] off Showdown see what your adversaries can see because often that's enough to disrupt things the most important vulnerable weak Link in the vaccine Supply chains for the candidates was a single sole sourcement manufacturer on the planet they had one plant three it people zero security people and they were all over Showdown you could have sneezed on them and
killed another couple million people so we might feel good about our public private Partnerships we have neglected the target rich but cyber poor in ways that can affect your life to their credits this finally accelerated and started publishing the Kev list the known exploited vulnerabilities list that takes the out of all the cves I've written three percent ever get exploited and they will down to the ones that are known to have caused harm in critical infrastructure you should be living by this not CVSs stuff they also made the cpgs the Cyber performance goals the White House really liked my bad practices and really liked my crawl walk run kind of get your stuff off search and said what do you do after
that so this is 30 of the 430 controls of the 36 controls of the 400 page in this cyber security framework because 10 years after the voluntariness cyber security framework what's been clear is most owners and operators of critical infrastructure have volunteered to ignore it so if you can't do all of it do the crawl stage of crawl walk run I've been pushing transparency and S bombs saying s-bomb's coming it's here the patch Act is an acronym but in early 2022 Congress in a bipartisan way introduced the law saying we need mandatory minimum cyber security requirements for all FDA approved medical devices the lobbyists lost their freaking Minds but part of why they did it is we are
starting to see losses of life and we need to preserve the trust and safety of the public so I was introduced in a bipartisan way and passed almost unanimously in the house it was almost dead on arrival in the Senate because of millions of dollars spent to kill it in May of last that year I testified to the Senate um I considered playing that five minute for you but the job was to convince one particular holdout senator if he should fight for this or not and even though all the patch Act was stripped out of all the legislative vehicles and it should have been dead in December while I'm on my belated honeymoon with Audi
he fought his ass off and he got it stuck in the Appropriations Omnibus Bill in the patch Act is law of the land hackers passed a second [ __ ] law
this was a team effort to be sure Kevin Foos original work the fda's courage hell staffers bow lots of people raised this Village raise this child that said you cannot bring a medical device to Market anymore if it is not patchable if it does not have a coordinated vulnerability disclosure program to work with good faith hackers and if it does not have a software bill of materials and threat models and a bunch of other stuff so it won't fix the Legacy problem we have today but going forward large small medium rural hospitals will have more safe and defensible things hackers helped write most chunks of the White House National cyber security strategy Senator Warner is doing it again he
wrote a paper that said cyber security is patient safety and he's intending to introduce regulation on the hospitals who are to fragile to care so what we have to do what the hackers have to do is ask how do you take the fact that for the next 15 years no matter how much help and regulation we push hospitals are going to be routinely ransomed they're going to successfully rearance them for the next 15 years so one of my analogies is after 9 11 we had we recognize you can have hijackers get on a plane and turn the plane into a missile and we did a lot of stupid [ __ ] as a country and as an International
Community in response to that but one of the smart things we did is we added steel reinforced cockpit doors so they'll get on the plane they won't get in the cockpit is the idea so I've been asking what are the steel reinforced cockpit doors of hospitals what if what things if you shut them off could lead to loss of life it's electronic medical record system it's heart brain and Pulmonary we also have to ask what's the regional impact of the 7 000 hospitals in this country if a hospital goes down here in Vegas there's another one within driving distance maybe to not have loss of life but if a hospital goes down in the middle of Rural America
you're probably going to see elevated loss of life so which systems are too isolated to fail unfortunately they're failing in droves so I've been asking how can the hackers help here not to hack things in hospitals but the things that are truly connected to loss of life or national security resilience same thing for the food supply hackers are turning to the food supply while you've probably seen a dozen hacks like JBS or Pilgrim's Pride or Dole or Americold uh the newly forming ISAC for Foods because we've never had an ice act for food until now because we didn't care about food um they've tracked over a hundred successful electronic compromises in their database so we want to see what's
the food supply because like the healthcare supply the food supply depends on chemicals on water and wastewater on cold chain electricity and you're going to hear about hungry hungry Hackers from sick codes and uh Casey John Ellis and you're going to hear about it from Paul Roberts you're going to hear water water everywhere after that so I'm very concerned about these water food electricity and emergency care that are Target Rich beside Rapport and geographically isolated that if disrupted could lead to loss of life or loss of food supply so the idea here is uh maybe one of the futures for the Cavalry is focusing on these Target Rich cyberpore basic human needs like food and water and shelter
and safety okay I'm basically getting the hook I'm going to stop that line of thought and tie this up I'd like to tell you I'd like to tell you A Tale of victory I can't
because while I thought the Cavalry could end and we could say we did a good job I thought I could tell you that I thought I could say well maybe the Cavalry should transform maybe we should just focus on food and water and electricity and that's what the track today is about and then I said well but if I spent the next 10 years on hospitals alone I'm still not sure we could succeed because it's one thing to get the medical device safe but this map guys this map or Hospital closures There's 7 000 hospitals in the country 85 percent of them are medium small and Rural 15 are large the 15 percent have a ciso they go
to ice ax the 85 percent don't in this time lapse photography these are hospitals that have closed forever if there isn't a nearby hospital the people that live in those zip codes are going to have a higher death rate for heart brain and Pulmonary they are closing and no one's replacing them now they were closing before the pandemic they're further strained during the pandemic and many of these small rural hospitals have four weeks or less of cash flow on hand four weeks or less so where does cyber security come in in preparation for this keynote about about two months ago Saint Margaret's in Illinois closed forever it's just one hospital it's not the first closure people like ah hospitals
close we'll we'll buy them we'll put them in but a lot of these aren't getting bought they're just going away some of the ones that get bought because they're distressed they get put on life support strip for parts they take the doctors they take the equipment they shut down services so they're basically in a coma so you're seeing hundreds of these 7 000 hospitals where people live going away and if it's more than three hours away you're going to see a lot of dead people from strokes and heart and we're not replacing them so here's why you St Margaret's gutted me it's the first hospital to cite as part of their cause of death their Ransom
distress because if most of these hospitals have four weeks cash flow on hand and a typical Ransom will shut you down for six to eight weeks six to 12 weeks it's a death sentence so while it is not me the thing that made them financially distressed privatized medicine did the pandemic did it's the straw that breaks the camel's back and we're having 700 plus ransoms a year so how many more rural hospitals where you or your families live are you willing to see go away forever so I can't fix the Health Care system outside of cyber I'm not even sure I could fix or we can fix the Health Care System inside of cyber but what I know
is they can't afford to invest in minimum cyber hygiene they can't and they can't afford not to and I don't know what to do about that and if we spent our next 10 years on this I'm not sure we would fix it so I am humbled again while we have crossed the river and we have done exactly what I set out to do I didn't want to fix a single medical device from a single manufacturer I want to hack the system and the rules for all medical devices we did that and on the other side of that River I can now see more and more turbulent Rivers ahead so we are not Public Health officials but we have
failed to integrate into their hazards model that if you don't spend enough on Cyber resilience you might go out of business so we have to have empathy for their situation but also advocacy that if we don't do anything we could see another several hundred closures or predatory Acquisitions and you may not get timely access to care it's one thing if it's a consult and you have to drive overnight to get to it it's another when you're in desperate need of time sensitive care so I'm basically out of time but when I look at this and I zoom out by the way straws are back the camel's back um when I zoom out I've been asking
since January do we end the cavalry do we transform it into something else like the bottom of Maslow's like pure Healthcare or how do you get Scaled because what we're doing at current course and speed it's not enough and we've asked a lot of you if you look at Beau or Jen we're exhausted so if there's new recruits if there's new leaders then maybe we kill the Cavalry and we start the Cavalry Academy what if we make a boot camp and a recipe book for if you want to save the world if you want to make the world a safer place we will Mentor you accelerate you boot camp you so an incubator accelerator for people that
want to change the world this only works if someone in this audience wants to pick up a project and have the audacity to try to pass laws or change incentives or connect the dots that make sure those hospitals are not just evaporating on our watch we did not cause these problems but we have a unique ability to solve them so I'm still trying to answer that question Friday was my last day in the private sector I'm uncertain what the path forward is but I'm committing myself to spend the next up to three months seeing who reveals themselves I didn't even know who Beau Woods was when I made the last call to action and he helped me
change the world some of you in this room can help for the next decade so I don't know if there's any of you or which of you but if you'd like to do something bigger than yourself as the world increasingly depends on connected technology they increasingly depend on you so who wants to change the world finding
and we do have a mic and a little time for questions please be very careful anybody going by the projector here it's got a broken leg and it will topple right over and break
I should have um uh the best place to find me for the next two days is in the Copa Lounge for them the Cavalry track and several things we touched upon will be explored in Greater detail you can find me online most places at Josh Corman j-o-s-h-c-o-r-m-a-n
or I am the Cavalry on Twitter
all right thank you everybody we have uh the first round of talks starting at 10 30.
[Music] thank you [Music] foreign [Music] thank you [Music] thank you [Music] foreign [Music]
[Music]
[Music] thank you [Music]
[Music] foreign [Music] foreign [Music] [Music]
thank you [Music]
[Music] all right
[Music] thank you [Music] thank you foreign [Music]
today's talk is authentication proxy attacks I got to admit um finding out that I was right after Josh Corman the indominal Josh Corman was was it was a bit intimidating um but uh you know we're gonna we're gonna talk a bit about this and and a few of the things Josh said resonated with me um the first is the fact that it's it's up to us it's up to all of us um to to make this difference make the difference in the organizations that you all collectively represent um so in this talk um my focus is on the practical I'm hoping that everybody regardless of where you are right so we talked about Wendy neither's security poverty line
Google it if you haven't heard it but I know that many of you are below some of you are above my goal here is not just to speak to the people who are at or above the well-resourced right the ones who can send you here and you know pay your way and all that kind of stuff I want to make sure that everybody has the opportunity to take something back with them
so a little bit about myself my name is Chris Merkel I'm a senior director of cyber defense at Northwestern Mutual which is an insurance company uh I've been doing security for a long time long enough where I stop telling telling you what it is in years because I just rather not it's been too long um I've been coming to b-sides Las Vegas on and off for over a decade now I love this conference I love the vibe I love the people that come here this is hands down top five uh I like to reverse engineer malware for fun most of my days are spent leading teams of people who do the fun things so I I still try to spend my time doing
that I also have um bad habits and opinions uh yeah I like Nano over Vim um uh I have been convinced that pineapple actually does taste good on pizza so my mind can be changed and I put those two bullet points in to remind myself that my terrible opinions and decisions do not represent those of my employer that's like a that's like a little mental bookmark right there all right and then the last Point here uh I got to meet John McAfee here almost 10 years ago how many of you were here when he came to to b-sides a couple of you that was wild okay um that experience of uh hearing him get grilled in depth by people who
understood uh facts details and timelines was was was crazy um and reflecting on it my first point here about being in security for a long time the longer I've been in security the longer I can start to understand John McAfee's uh overall uh Arc to go from cyber security luminary to bath salts enthusiast to crypto grifter I used to think man how did that happen and now I've been doing this for a while I'm kind of like that doesn't sound half bad um I'll have my contact information on the last slide as well you can find me on the uh the the fediverse and uh on the zuckerverse I'm out there right now um the other thing I want to point out
here is that the the stuff I'll be talking about here is uh not solely my research I work with some of the uh most brilliant people working in cyber security in counter threat in intelligence in threat hunting in incident response in detection engineering um and and I am sharing that Collective knowledge with all of you so let's talk about all of you you got multi-factor authentication look it is the year 2023 and I know some of you are thinking to yourselves well yeah but mostly oh okay that's fine but take the Victory lap okay that's a big deal if you've pushed your organization through if you've had those conversations about uh user experience and the challenges
that come with that particularly if you're working with consumers clients people outside your organization those are tough conversations you did good now some of you may have also moved on from SMS okay SMS is weak but SMS is great okay it can be both I'll talk a little bit about why that is if you're in this position most of the threats that your organization faces um against your your logons your sign-ons your authentication you've mitigated those that's great okay um but as We Know our adversaries they change their tactics and we're going to talk about that so um with the the good news comes the bad news and and the bad news is that even while you might have
multi-factor Authentication protecting your organization and its assets increasingly it is not enough so we're starting to see uh attacks that were really demonstrated to be possible like well over five years ago starting to actually materialize okay um so we're we're seeing these these types of campaigns going on and so what I'm going to be talking about is an evolution in adversarial tradecraft that's taking your typical adversary in the middle and moving it to the next layer um for for targeting organizations for whom they've done those fundamental cyber hygiene Basics like turn on multi-factor authentication okay so that's the bad news now um shout out to sisa so sisa entered this conversation about almost two years ago and they they
released a a paper on this I strongly recommend you look it up but I have summarized it for you um it's it's fantastic and what I love in particular about this and and by the way again another shout out to Josh Corman he talked about changing the dialogue changing the framing right so what did we used to say we used to say that you need to make sure you have strong Authentication and and if you've gone to the weeds with somebody they would say well I've got a long password it's a strong authentication oh no no no no you need multi-factor authentication strong authentication well okay okay I can do that I can do that well well the problem
is and I'm going to assume most of you understand this problem to some extent or another um these other second factors have weaknesses and specifically the weakness of token theft is what I'm going to talk about today and so they embraced the term fishing resistant multi-factor Authentication now here's why I love that term it's a term of art I'm saddened that it was first an industry term before Cesar came up with it but what that means is as you are talking to the decision makers in your organization and they're going to ask you questions like are we resistant to phishing attacks you might say are we okay you might say it's time for your outrageous speaker request my what
when we uh have speakers who apply to speak in the program we have this thing called an outrageous speaker request it's a little field at the very end of the the application okay that gives them a chance to ask for anything else they might want what I asked was it was really late at night in this case uh we were asked to bring back green apple Skittles yes discontinued and replaced uh with lime I believe again lime is terrible got rid of the lime for those who are not up on the the drama of Skittles lime was around they got rid of it they put in green apple now they got rid of green apple brought back the lime and so now
everybody's angry this is It's a classic Coke New Coke thing anyway so there is now on change.org uh a petition to bring back green apple Skittles and I have here Flyers to hand out to everybody in the audience uh I ask you to please consider honoring our speaker's request and helping us to bring back green apple Skittles here at besides change the world one person one thing at a time one Skittle at a time so so according to this handout that he's provided me the change.org petition only has 834 signatures that means if every one of you in this room by my rough count once and petition for this change we could get this over a thousand people we can
do this [Applause]
all right let me transformation back into uh where I was all right so so again we're talking about changing the framing to change the discussion if you are talking with your leaders again if you're talking with your CSO CIO board member and they want to ask you are we resistant to phishing attacks now your answer can be some but not all we don't run fishing resistant authentication in our organization and it's not just your opinion now you can bring up Eagle Shield because that carries a lot of weight we'll talk a little bit more about the technical mechanics of this in a minute I promise token replay attacks are on the rise I don't know if you any of you have read
the long-form wired article on the hack at EA but it is fantastic and I strongly encourage you to to look at what can happen when you start with one stolen slack token okay this data is a little old but it comes from a good source from from Microsoft I do reserve the right to give them grief but they've been making some positive moves so I might pull my punches but we can see that the use of tokens is on the rise so let's get into how this all works now it's a bit of a complicated diagram and I'm going to keep staring at this screen over here because it's it's a little bit bigger than I can see on my
presenter view but we're going to walk through step by step technically how this attack works okay so so first it's going to start with a phishing message you all get this okay your victim is going to enter their creds and they're going to enter in their MFA now that could be a code request from SMS that could be a device approval uh something along those lines now again I'm not talking about 502 web authen this is everything that's not that I think the attacker has a proxy setup so what they're doing here is they are taking your actual logon page and just proxying it they're not making a copy they're not doing like copy and paste
into word and then back into HTML I always laugh when I see word HTML in adversary uh pages and stuff like that it really cracks me up it also makes me sad because it totally still works
um and what's going to happen is when they put their credentials in that's going to get forwarded across the proxy to your identity provider so your identity provider is like oh I I've received credentials because they were asked for they were requested by this proxy and now I'm getting this back this all looks normal to me the attacker along the way is going to steal the credentials because you know you can use those later even if you're not even if your primary target is the token the identity provider they don't know what's up this is just a request from a client for off that's normal so they're going to go yep everything checks out MFA A-Okay here's your
session token now the attacker is like yeah cool I have your session token that session token that gets passed right back to the user so so the other thing you tend to run into with these phishing attacks is the what have you done once you've actually updates thank you jam um what do you do when uh you've you've successfully conned that person you have to take them somewhere and and this is where where adversaries are kind of like I don't know maybe I'll dump you at google.com or something who knows or Mike whatever right um but by forwarding that that session token back your user has a valid session so where do they go next they go to the actual
site that they've just authenticated to so to them from their perspective and experience they just successfully logged in why because they just successfully logged in because that's how this works now that becomes a real problem for your security awareness and education right because at this point nothing looks different you've successfully logged in the documents that you've most recently worked on on Microsoft 365. they're all there
so now what does the attacker do they just replay the session token now um I I I'm not an expert in um you know all of the Microsoft primary tokens refresh tokens sub tokens app tokens it's complicated suffice it to say if you can get your hands on that primary refresh token uh by default in Microsoft 365 you have seven days of access that you can parlay
uh and then of course you know those credits go on to secondary markets maybe get used in password spray attacks uh go find those little uh corners and edge cases in your organization where uh you haven't quite gotten that two-factor authentication in yet so let's talk a little bit about delivery um I could give a whole talk on this instead I'm going to give one slide um delivery methods are getting pretty interesting in my opinion um first and foremost we are seeing these types of advanced MFA proxy attacks coming across bog standard dumb phishing emails okay still works why why change right we're also seeing what I call encrypted message hollowing um and what this is is so if you've if you've
ever used a you know proof Point Microsoft sending to Gmail um mime cast Etc you've gotten this message that says you have a secure message waiting for you you need to log into a portal yada yada yada okay most of the time those types of systems are used in business to Consumer type of relationships and what that means is you don't want to necessarily burden that poor end user with having to set up full MFA or whatever it is just to read that one important email that you want to send them about a a health care issue about a financial transaction about a real estate deal whatever it is right well what attackers are doing is they're
gaining access to one of these encrypted messages how do they do it a traditional account takeover uh you know those kinds of things they hit a link maybe they do a password reset whatever it is and they get into that corporate uh Email encryption solution from one of these big name vendors and if it's not configured properly they go into this message and they hit the reply button but then what they do is they just blank out everything in there um or I'm sorry they don't hit reply they hit forward critical difference they're going to hit forward on that message and they're going to blank out the message body they're going to blank out the subject
line and they're going to put you in as a Target now instead of having to create those goofy looking fakie you have an encrypted document kind of nonsense that could potentially be taken down because it is part of adversary infrastructure and all of that they are now landing at your big corporations encrypted messaging solution but what they're seeing is a holy new message and we have witnessed uh one adversary group literally make hay from one account and one message just blanking it out and using it over and over and over and over and over again and every time the recipient gets you've got a secure message they're not going to get any warnings and it comes
from a big Corporation and so the big Corporation is inherently trusted we're also seeing account takeovers uh in the Microsoft 365 space abusing Microsoft purview Microsoft purview is the encryption solution that used to be called something else I guess their branding rebranding worked because I can't remember the old one but it's basically when you hit send secure in Microsoft Outlook that's Microsoft purview messaging as a tenant administrator or as a exchange administrator you have very little ability to inspect what goes into that okay and and if somebody is in the Microsoft 365 world and they receive a purview message it actually gives the attackers additional credibility because you get this green banner across the top of your
outlook that says congratulations this message is encrypted and if you're the end user and you see a green bar with a green check mark in it how do you interpret that do you do you as a as the end do our end users go wow that's fantastic they employed Transit encryption on this I feel good about that no the way they interpret this is oh bar is green I'm safe I don't have to think about those security awareness messages anymore click if you're sending it outside of a Microsoft organization you're gonna you know to Gmail whatever you're going to get that typical log into the portal uh you'll get a message attachment that message attachment is fully encrypted
you can't inspect any of this and that really is is unfortunate so I talked a bit about what the the victim experience is like this is what it looks like the only thing that you're going to see different is the URL in the browser bar uh you can't can't fake that out um you can do tricky things right to left I don't you know uh you know all those obfuscation techniques that we know and love but again your security awareness messaging it's a lot less effective at this point why because if you're if you have a branded portal like I show off one on the right here um this is the one they see every single
day if it's a standard Microsoft logon it's the standard Microsoft logon so what's your click rate already on fake logons it's pretty bad right but when you have nothing to tell somebody I mean yeah you can tell them go go look at the browser bar but but again think about this whole attack chain from end to end you receive an encrypted message from a well-known reputable Corporation maybe it's something somebody you already do business with because they've done account takeover for that outside organization you're working with you receive an encrypted message the encrypted message has a green bar on it now it's trusted you click on that I do not believe that it is even fair
to ask our users to catch this
let's talk a little bit about the evolution of tradecraft um I'll uh one of my colleagues shares my same passion for for terrible clip art um so there's your Dolly generated hacker um thanks Chris um our adversaries are also evolving their tradecraft so we are seeing a lot more anti-inspection techniques okay so so so even if you'd gotten to the point where you were extracting all of the URLs from your email traffic um and things like that passing that through some sort of reputation service sandbox whatever it is doing that you know uh in bulk um you're probably not going to catch it why they're doing things like referrer checking right they want to make sure
that this looks like it came from an email click uh you know things like that uh they're doing you know basic sandbox detection stuff um like I said they're using uh encrypted email they're also um looking for uh egress IPS so if they're targeting uh you know a specific Corporation or set of Corporations and you're not coming from one of those egress IPS that's noted on uh the the Aaron uh or ripe or whatever net blocks you're you're just going to get redirected somewhere else um beyond that they're they're doing a lot of redirect chains um and and other obfuscation techniques right so your your typical automated sandbox that's going to look at a web
page um maybe it'll follow one referrer maybe two not six so that becomes uh a bit of a challenge to do any kind of inspection at scale um so let's talk about how we detect these things um I did put in the uh the description of this talk that there is one fatal flaw there is it's a bit weak so if you don't like it I'm sorry um but you know yeah I got to put butts and seats what are you gonna do um so let's talk about detecting attacks um first and foremost none of these things individually is going to be the tell the detection the one thing that allows you to catch 100 percent 80 percent
60 percent but if you treat these as signals if you have the ability to to look at multiple Dimensions if you have the ability to do any type of correlation you can build strong signals out of this um depending on the nature of the organization you represent uh impossible travel is reasonably accurate the problem is that all of your users who watch YouTube have now installed nor VPN after the three-month trial subscription um and it's all running on their phones all of the time um and not not just picking on Nord they're fine for for what they are other than uh snake oil for consumers um they do they go to Great Lengths to hide
their their egress traffic why because the only reasonable reason to use a consumer VPN service is so you can watch content outside your region right and so you're always in this cat and mouse game between uh your uh streaming video providers of the world and you know them having to comply with restrictions around uh uh countries and intellectual property and things like that um so so what that means is as a Defender you're gonna you're gonna get hits from weird places on the planet with really non-descriptive names and oftentimes if you do more analysis on asns some really sketchy neighborhoods okay um and and so you're gonna go oh hold on I'm under attack no no just bobbin
accounting was you know watching YouTube and installed a VPN um and then of course the other challenge is the use of uh uh proxies um proxy services so that you can tunnel out machines here in the US there are there are large and very um well-known ones that we see used a lot um so so what we try to look for is is a little bit of correlation right so um do I have a person logging in from an IP address that they haven't logged in from before and is there an authenticator change of some kind like a password reset uh in addition uh to their their multi-factor authentication options uh a change in those authentication options
things like that um also look for new device registration attempts I'll talk a little bit about conditional access policies later um but you know as of right now I believe that the standard default configuration in a Azure tenant is to allow devices to register themselves in Azure ad Ross the internet just because you're authenticating right so you're going to want to look for gaining persistence so going back to the uh the the hack that happened at EA um that's one of the things they did they they did an actual device enrollment as a persistence mechanism I believe they used a virtual machine so that's what you can do a lot on the the detection side
uh so let's talk a little bit more about um investigating these types of attacks responding and those kinds of things so first of all has a successful MFA occurred um Microsoft's logs if you're if you're taking these somewhere other than Azure Sentinel they are a nested Maze of terribly constructed JavaScript Java Json that um you know take you a long time to figure out and that sucks um do you know how to invalidate session tokens so in a lot of organizations if if you have reasonable confidence in your external MFA and somebody clicks on something and earns their creds you can issue a soft password reset it's nicer to the user because they just change their password
and next log on and you just kind of Coast by knowing that they're okay because there wasn't uh you've got MFA protecting you the problem is in in a in a session attack um against tokens um that soft password reset in Azure that does not invalidate the session token okay so so even if you detect the user clicking on a phishing email or and and entering their credentials you do that soft password reset the attacker's still in they still have that token now here's what sucks you go to a hard password reset and buy hard password reset I mean change it to an arbitrary value of your choosing and then make them figure out how to recover call the
help desk you know the self-service password portal probably not going to work because the attacker could use it right um that is really really um not great user experience um but that's where we are right now so beyond uh just Microsoft specific tokens think about the other things that the attacker might have had access to Slack um other other uh you know Federated systems or or other systems that might have provided their own token that's not tied necessarily to the uh the saml token that comes with Azure ID
um you're going to want to look at your Microsoft 365 logs for evidence of access so what did they do once they got here um the thing that I've seen most commonly is mail forwarding rules right because while the attack is a little more sophisticated these are people who are just using software as a service like evil proxy okay they are not the Geniuses who invented the technique they're the ones using the commoditized kit that they paid money for so the the adversaries are still the same lowest common denominator kind of folks so so so you know unless you're in a a more highly targeted sector with more advanced adversaries most of y'all are just dealing with
cyber crime and and what do they want to do they just want to compromise that mailbox so they'll set up like an exchange mail forwarding or not not an exchange but an Outlook mail forwarding rule um you want to look for user creation events right what kind of a count did they compromise um did they access data on on on one drive and and things like that um I should also point out that in the last 2 month or so um sisa successfully bullied Microsoft into giving full log access to people who have Microsoft 365 tenants um I don't know the full scope of what that means um but that is forthcoming um and shout out to sisa because nobody
should have to pay for logs that's [ __ ] all right
so let's talk about continuing to go on the offensive threat hunting um make sure you know your authentication endpoints um you can do Showdown hunts we talked a little bit about that um look for the usual typo squats page titles things like that um if there's a page title that's specific to the Microsoft logon flow you should never see that come from anywhere other than Microsoft so if that page title exists and you can detect that in your network traffic if you have the inspective capability to do so look for it so do you know where all your Authentication endpoints are you probably have some authentication endpoints that should be on a milk carton um
look at your Microsoft conditional access policies um it's almost cliche but they people say that identity is the new perimeter I tend to agree with that um and Microsoft conditional access policies if identity is the new perimeter conditional access policies are the new firewall and it's the new firewall in that it really is a pain in the neck to configure and you can screw it up easily in unintended ways okay because conditional access policies have the same concept of inheritance uh they have you can get order of operations wrong and the evaluation of a cap policy can go wrong and you don't realize it because it's staring at it on the screen it looks right just like your firewall
console so there are ways to to evaluate your cap policies the the more that you can do to strengthen your cap policies for users coming in from outside your network perimeter the better I know some organizations can can have more latitude in that than others you want to look for attempts to Target your organization so again look for uh type of squats related to SSO hunt inside your firewall logs um you know uh you know DNS logs passive DNS you can hunt externally um so in a previous iteration of this talk I was able to to demonstrate that I could find a whole bunch of people on Showdown running evil proxy and Evil nginx um because of the use of those redirect
change I talked about before that's getting increasingly difficult um but the main things we see are evil proxy which is the software as a service version of evil nginx and evil nginx itself now one of the things you can do is look for things like charm hashes um they're not rotating out the default TLS certificates that come with this software so um you know they're bad at tradecraft um so so and then I'm sorry there was one thing I wanted to go back to and point out here if we go back to the victim experience now there are if you get down into the Dom in Microsoft or a custom branded page and and this is this is that you know
here's that that one weird trick kind of kind of thing um there are elements in this Dom that only exist in this Dom okay for example on that Microsoft sign-in page there are links to a domain called msoff.net as far as I can tell the only time that domain is called is when you are doing authentication to Microsoft so if you see that in your network traffic coming from a source that is not Microsoft that means somebody is proxying the traffic you can catch it another thing that you can do is you can look for people who are faking your corporate identity that color blue is a very very specific hex code I know blue is the cool color for most
corporations but I guarantee you they're not all using the same shade of blue okay so the marketing department is going to make sure that that RGB code that represents our cherished corporate identity is unique okay so now again if you if this is that poverty line discussion so I'm sorry if you're not in this position but if you are in a position where you can inspect inbound web traffic you can start to profile for people using your corporate identity then you start to tune out and filter out the people that are known right that that SAS provider that made a copy of your logon that's totally okay right uh any any other additional logons you might have to your your sites and
systems and then you can catch them using your color or your you know logo specific size shape whatever it is so so that is one thing that you can actually search for and hunt for um that can give you a tell that your corporate identity is being abused regardless of whether it's coming through one of the proxies I'm talking about or if they're simply doing you know copy and paste into their their their uh adversarial infrastructure
so uh I've I've walked through a lot of um specific ideas techniques things like that um you can do this um the last thing I want to point out here is simulate attacks um you don't have to be a red teamer to do red teamy things okay it's not as technologically sophisticated as those cool red teamers would have you to believe okay um setting up evil nginx is as simple as spinning up a Docker container okay so run these types of tests against your organization to see what you see what do you see in your logs um you know what does the user experience like those types of things so I strongly encourage doing that
um so you got this you can do this all right so I've got I am going to post um slides within two to three days um you can catch those out there um if it's Mastodon I will pin it to the uh the top of my profile for for a little while so you can catch those there so I want to thank all of you for your time I want to thank all of you for taking time out of your work schedule to be here at b-sides um and uh I just want to say I greatly value uh the contribution each and every one of you make on a day-to-day basis to the safety of the organization and the
people that you're responsible for protecting so thank you very much and uh I'll take questions for about five minutes here if anybody has any
thank you
here so I I saw the Microsoft authentication window and this that came up like where it has someone looks at the URL and you said hey we shouldn't rely on our users to to look for that and I I completely agree but I I don't maybe it's just my organization that issue came up as uh in in another like uh cyber group that I that I attend I like hey they pointed this out I went back to my my company I didn't see a URL in in that authentication screen so that that's weird like just seeing the just seeing that Banner itself or having that revealed so I don't know like what's common or not but I was like well I so
so what I see commonly is um I'm not so much sure about the Microsoft side but but but oftentimes what we do with our authentication pages is we put them in pop-up Windows yeah that are like modal and they don't have an address bar in them yeah yeah like um that makes this attack even worse yeah just well just seeing it to me at my organization would be it but I didn't know if it was an option for like other people you know it shows up because yeah the fact that Sarah's like well this is weird but yeah I wouldn't trust my users for that either right exactly exactly and and I have another whole another
hobby horse talk about why putting all the onus on users is terrible as well
thank you hello hi so with the um the session token in that you described there's generally that um yeah a little closer than Mike yeah um so with the session token attacks um I guess one of the counter measures that you can take is reducing the validity of that token but how do you strike that balance like with user productivity because often users are quite lazy and your it managers are going you know I don't want to deal with complaints from users about constantly needing to authenticate and re-authenticate yeah absolutely so so the the Crux of the question is how do you balance uh user experience against you know uh the the life cycle of a token and making people
re-authenticate um I don't know I mean you know an adversary can get in and establish persistence very very fast um so I don't think the solution is shorter token life cycles um Microsoft is starting to roll out or they do they have um risky behaviors or risky sign-on events um I would strongly recommend implementing those um uh and this is actually where my gripe against Microsoft is I don't think they should allow logons from things like uh browser agents that aren't actual web browsers um I mean it's such stupid low hanging fruit but it still seems to work for some reason um the the one thing that I I think that should happen is there should be a match
up between um the token and the actual browser agent that was used for the token okay so if your browser agent changes and you have a successful off with that token it should just Auto invalidate the token Microsoft has not implemented this to my knowledge they have some token protection stuff but it doesn't quite go this far um but but frankly there's no reason why you should be able to Pro for the same token between Chrome and Firefox even on your same machine because that that token is just sitting there in a cookie in the local cookie store so it should never be proffered from anywhere other than the first place but that's that's what they're doing
I guess first first more of a comment I have heard Microsoft has something in beta that's exactly what you're talking about yeah they're working I know they're working on it my question is Microsoft authenticator's passwordless authentication Microsoft claims it's uh fishing resistance is it according to cisa yes I do believe it is thank you so if you're using the the right configuration Windows hello for business um and Microsoft's authenticator um if you're using in the right configuration it's storing the uh the certificate in uh in an enrolled device uh whether it be your Windows laptop things like that yes um there there are a lot of great great options there um I'm a Mac User in an Enterprise none
of those are available to me um so so the so the problem is uh that's great if you're in all windows Fleet um it starts to break down when you have any type of BYOD Max situations things like that so I've gotten the stop sign I want to thank all of you for your time again and the Fantastic questions thank you very much [Applause] [Music] thank you [Music] thank you [Music] foreign [Music] [Music] thank you [Music] foreign [Music]
[Music]
[Music]
thank you [Music] foreign [Music] thank you [Music] thank you [Music]
[Music] thank you [Music]
[Music] foreign [Music] thank you [Music]
[Music] foreign [Music] foreign [Music] foreign [Music]
[Music] [Applause]
[Music] foreign [Music] thank you [Music] [Applause]
[Music]
[Music] thank you [Music]
baby [Music]
[Music] don't leave me alone [Music]
don't wanna overthink it baby [Music] maybe you'll kill me giving me Wind and Rain some kind of butterfly baby [Music] [Music] but I don't wanna miss you baby [Music]
[Music]
maybe you'll give me [Music] away [Music] don't leave me alone baby [Music]
[Music] baby you maybe you'll get me [Music] don't leave me
[Music] oh [Music] okay guys this is the talk on the dark playground of cicd a tag delivery by GitHub actions given by speakers yosuku kogu and you go to Yamamoto we would like to thank our sponsors especially our Diamond sponsor Adobe Anna gold sponsor black cat Toyota and conductor one it is with their support along with other sponsors donors and volunteers that make this event possible these stocks are being streamed live and as a courtesy to our speakers and audience we would ask you to check to make sure your cell phones are set to silent if you have any questions use the audience microphone so that YouTube could hear as well please welcome our speakers
[Applause] uh hello everyone today we talk about attack techniques and related to GitHub actions I hope it will be useful and excited for besides Haka community let me tell you about us I'm yusuke and he's kirsto we are from Japan and work for NTT communication Japan telecommunications company we work as offensive security researcher sometimes works us Internal Revenue this is our presentation agenda our talk goes like this first we talk about basics of GitHub actions GitHub action is a share CD platform provided by GitHub that allows us to automate build tests and development pipeline this is a component of GitHub actions in our presentation Runner and action importance so I I will explain a retirement detail
there are two types of Rana guitar posted Runner and self-hosted runner in short the difference is a resource owner GitHub hosted right now with resource provided by GitHub sir Jose Rana done with resource provided by user Rana supports some of operating system Mac OS Linux windows but our research focused on Windows next let's look at custom actions actions are immediate tasks in the workflow we can use a combination of action to suit our purpose and we can also create and publish on actions custom action has a concept of type and location three types are available but we have Target JavaScript action and compose detection that supports Windows so much for the basics and deep dive
into the main contents as the beginning of the research we analyze the behavior of JavaScript action the figure is visualized process Behavior as you can see worker.xa executes a and gives index.js as argument this index suggests it defines as entry point in the metadata file and the script engine no.text there is included in the run application package observing this behavior and specific specification we came up with new attack techniques what do you think of when you hear JavaScript in general it is a script language the core technology of the World Wide Web on the other hand some may think of jscripts there is a basic compatibility but J script and JavaScript are different our research focused on both jscript and
JavaScript I mean we considered two techniques for each firsts we introduce the jscript version we call this technique maisha's descript custom action this is a technique for executing the script from JavaScript custom action via binary hijacking and buscar reading prepare workflow and custom action for the attack as shown the velcro has two steps first step we place the binary and second steps and execute the custom action this custom action is implemented in J script not JavaScript the behavior of the endpoint when the workflow is executed is shown in the figure copy double script.exe to node.x and index.js is executed from nodal text there at this step node.xa has been replaced by w script.exe we hereby at composite action
this one allows us to bundle multiple steps into the into one action the figure on this slide shows how it changes with and without composite action with composite action the velcro is one step I'll explain the reason for this better finally the JavaScript custom action combined with the composite action looks like this we call this technique as monisha's J script composite Direction and here is the actual port code we developed the velcro calls compose reduction and composite action copies WS crypto.exe in Step run and cause custom action in step 2. index.js is written in jscript in which the attack is implemented DACA can arbitrary attack by changing the jscript implementation I have a quick demo of infecting covers
like [ __ ] zoo from this technique let me show you the rest size attackers View and right size is big team View around Now application is already running and git push to GitHub tutorials events um then you can see that velcro yeah velcro it starts and some child's process starts from
and finally see the connection is established and at the end of the workload the process is terminated therefore.xa this process is a sheet process it is started by early bird injection and PPI disproofing Chrome run application so far I have talked about jscript version and next I will describe JavaScript version we have discovered attack techniques that exploit node.js extension node.js has extension called Shipra addon this one can be used for attack we call this technique Marshall's JavaScript custom action there is a package called memory.js which implemented functions like open process and inject the area we use we use this library in our book here is the actual code we developed the velcro calls custom action this custom
action includes simple DRL and node modules for the attack text.js is implemented in JavaScript and is designed to perform a dll injection using memory.js Library an open process and inject DNA function already implemented in memory.js so our JavaScript JavaScript code is very simple we used memory.js in our research but we can Implement on Shipra add-on for arbitrary attacks unfortunately there will be no demonstration today as my final part I talk about some consideration first let's discuss the attack scenario Mauritius action disguised as a regime attraction could be published in the marketplace by attacker then users May mistakenly use the measure section A thinking it is a estimate action or users of GitHub actions are potential
big teams so this is a big baby threat a composite action makes this threat as realistic without composite Direction step one is required but who set up this even if users accidentally use a fake action but they will not accidentally reset step one so as attack aside it is very important to add composite action it can also expand the attack possibilities in advance technique run application thorax launch chart process and here's our child process when the workflow is finished this means that the process launches for the attack will also be good it is tracked by process environment variable called run networking ID we can deserve tracking by rewriting this variable like this this makes the process persistence
next let's consider how to protect it GitHub recognized these attacks as threat and published best practice it says that it is important to audit the code on the creators of course it can be detected by Security Solutions like EDR or antivirus maisha's just script composite direction is detected by EDR for defensive patient via masquerading this is unchangeable part this techniques so it is a good detection point also js5 is sometimes detected by antivirus however this is not effective because it can be easily purposed like this right with a few changes it's easy to bypass of detection so antivirus is not not effective measure detecting JavaScript custom action is a bit more difficult because this technique does not require
to replace the script engine so there is a bonus detection point we think it is important to detect the attack Behavior being carried out not this technique itself and this technique may also be effective against amsi chroma quick analysis it appears that the nodal takes that does not support MSI
so in my part let me introduce GitHub action C2 GitHub action C2 is implemented by utilizing the GitHub actions run application as an agent this is an abstract image of GitHub action C2 the run application runs on target machines connecting to GitHub especially attacker's own Repository this connection becomes C2 to operate this attack attacker push a marshes configuration file outside GitHub this C2 has two features or threads the first is The Source process is a resume application so run a DOT worker.exe exit and second point is that the destination is a resume domain and IP either explain or precise so how to establish C2 so as actually stated Runner application when launched performs a long pouring
wrong pouring against GitHub to await jobs to manage Runner as a C2 agent assigning a unique level is needed this arrow is a target to send the command individually when establishing a city connection with multiple Target without level attacker end up sending same command to all connecting runners how to send the commands so any shared commands can be executed via GitHub options workflow workflow jobs are described in a configuration file on attack the Repository so we can trigger the workflow by GitHub events like push and in this case we can we can use webflip event called repository dispatch we can call this event using call or passing instruction from outside of GitHub plus using just introduced Marshall's
custom actions more advanced attack can also be performed I will show you short demo to make it easier to image victim runs the Run applications and then after that the attackers and the commands character.exe and through my now you can see the park.x is launched and we can see the who are my result
so adapters attack scenario is almost the same other C2 firstly attacker create a public repository for tax and then send their runners application to Target using techniques such as fishing after that Target the user on no inquiry runs it as shown in the demo about detections so since Source process and the destination domain is legitimate C2 establishment is hard to detect so it's very ordinary way but don't miss a lot for activities related to GitHub action C2 just they just the theme regiment last part is auto thread related to GitHub actions I am briefly introduce three threads fast thread is free jacking free jacking is a process of using free Cloud resources to perform Crypt mining
operations attacker execute mining script using GitHub resources and they get Rewards to utilize GitHub resources the attackers and prerequest triggering Mining using others Repository to prevent this GitHub change specification and approval has been required to run workflow from a public folks since April 2021 so attacker have shifted their focus toward automatically creating GitHub account executing GitHub actions and exploiting GitHub resources secondly Mauritius public Focus request the idea is that equal self-hosted Runner is being used with a public repository GitHub actions may become a potential entry point for external attacker to gain some football GitHub recommend using cellhost run only for private repository however it was nothing that there are numerous organizations still utilize their hosted
Runner
lastly steering Secret GitHub provides a feature to create encrypted environment variables called secret as an organization or repository level secret can be accessed by GitHub options when they are set up for repulsory these pose a potential risk of secret being stolen if accessed by external attackers through GitHub auctions so I ended up here conclusion there are three points by UT rank that feature of advice cicd such as a command and script execution it's possible to launch attack within the resume to cicd process Mauritius code May infiltrate the cicd pipeline without notice so publicly are available malicious problem there's a risk of enabling C2 through the resume feature offered by crcd which are the user owned host for executing
the task this is a feature work so thank you for GitHub security team for cooperation also thank you beside Las Vegas for accepting our talk [Applause]
if anybody has any questions you guys have five more minutes to ask hi uh so I work for a corporation that has a number of publicly available software development kits hosted on on public GitHub we use CI CD to run unit tests and show test coverage that hopefully our customers can trust the efficacy of the sdks deployed and we've observed a couple of malicious public PRS you're saying that we're working on locking down those repos so that doesn't happen again you're saying the current industry wisdom is to Simply disable CI CD for public repos is that the case or is there another is there a way I can have my unit tests and not malware
question uh how do I configure CI CD with GitHub actions for a public repo and not be subject to malicious public PRS are there other access controls that we can implement
all right sorry it's me how to set sets p uh cicd pipeline yeah so the the GitHub suggestion is to only use CI CD on private repos but our customers will only use our sdks if they have unit tests run and we can show a certain amount of test coverage so we need to use CI CD on public repos in order to get that assurance and get that testing there's there's just no way of hardening a private Runner against malicious public PRS is that
Brothers
right but the cicd is trigger prior to review and merge
that is
well so the secret stealing that they were showing works on self-hosted or uh public hosted Runners either way
okay okay thank you
thank you for the insightful talk foreign
[Music] thank you [Music] foreign [Music] thank you
[Music] foreign [Music]
[Music]
foreign [Music]
foreign [Music] hahaha [Music] [Music] foreign [Music] thank you [Music]
[Music]
[Music] foreign [Music] foreign [Music] thank you [Music] foreign [Music] thank you [Music] of embedded devices and the best thing on Amber is it is open source so right after talk you can go straight to a GitHub repo download Amber and start analyzing firmware and at the end making the internet of things a little bit more secure so my name is Martin Messner I'm a penetration tester and security researcher at Siemens energy and during my daily business I perform penetration tests on critical systems and environments and this is the area where we developed Amber and where we are using Amber on a regular base but first things first so in today's environments everything is now connected
it starts with your private iot at home or better it starts with your iot on your body probably have a smartwatch the Smartwatch is connected to the smartphone and the smartphone is connected to the rest of the world or you're going to work by car your car gets updates over the air which means over the Internet the traffic lights are connected critical infrastructure highly interconnected if you're working in a big company or in a small company it doesn't matter your I.T infrastructure your OT infrastructure ICS everything is connected and on the bottom level there are running these little gray boxes yeah I know there are not always little and not always great but they are
running some operating system on it this could be in rare cases a full-blown Linux operating system uh in more cases it is a it is a stripped down Linux operating system you will find some real-time operating systems like VX works or in very rare cases you will also find some Windows operating systems the generic term for these systems is firmware and according to a firmware expert company called eclipseum femora is the most exploited category of the over the last few years and if we take a look at the Microsoft digital defense report from last year we will see that they found out that 30 more than 30 percent of the analyzed firmware has more than 10 known critical
vulnerabilities so from our perspective it is time to take a deeper look at firmware security but how to start um in my environment I typically have the the real device on the on hand and we're trying to move away from the Black Box only approach to a more great boxes and Analysis Style with firmware analysis with the firmware we are able to understand the inner workings of the device and we are starting with some information gathering like doing strings on the camera binary or doing some some entropy analysis we're using well-known and established extraction Frameworks like unblock we're doing configuration analysis find some files do a lot of research and at the end we try to
identify the juicy stuff of the firmware we try to find weaknesses configuration errors and vulnerabilities and to understand the inner working of the firmware all of this takes time and time is typically the problem the limited factor in such a penetration test so Amber think about Amber it is like an automation framework for firmware analysis so you just need the firmware binary you drop it into Amber Amber extracts it Amber is doing the analyst analysis for you and finally generates a nice and shiny HTML report but how do we get access to the firmware the easiest way is just go to the window website download the firmware and you're ready to go but sometimes the firmware
update files are incomplete sometimes they are encrypted or they are just not there or behind the payroll so you need other mechanisms to access the firmware one possibility if you have the device you and you have shell access to the device just copy directly from the device or you can probably use other command other vulnerabilities like command ejection vulnerabilities that you can exploit and get access to the device another possibility is getting getting access while the hardware you can try to identify debugging ports like uart or JTAG you can try to sniff the communication between the flash storage and the CPU because the ephemera needs to get transferred to the CPU for execution and finally the more invasive
attacks like desoldering the flash storage and extracting the flash storage then via a specialized Flash Reader at the end you will have access to the firmware you have the firmware and now you can start analyzing the firmware and you will have a lot of questions for the firmware you've just a few examples um where are the binaries which binaries where the libraries which configuration files are there other configuration mistakes which kernel is running on the on the system and so on and so on one of these questions is probably regarding your software inventory the world speaks around s-pom um this s form or software software inventory is really interesting from a penetration tester perspective because
there are probably some some vulnerabilities and exploits for free there so at the end you need to identify the software components you need to find the exact version details and then you're ready to go you can or queries vulnerability databases like the CV database then you can query exploit databases and you get probably a one day for free the goal for every penetration tester is not doing this manually the goal is that this should happen automatically in the background and Amber um has a hybrid approach implemented for doing this so first first of all we are trying to analyze the package management system if it's available on the firm room we then use static analysis like we are doing
strings on every binary or we're doing hex stump or um query the kernel modules for the for the exact kernel version and additionally we try to run every binary in an emulated environment we will see it on the next slide and finally we also try to run the whole firmware in an emulated environment we will see this also later on so regarding the emulation of every binary Amber runs through all of the binaries identifies the architecture of the binary it chooses the Right emulator and then it starts or with the trial of every binary to emulate it to run it in an emulator with different parameters like minus V minus capital V minus minus help and Amber collects the outputs that
it's generated now and here you can see an example from BusyBox BusyBox spits out a lot of output with minus minus help and in this output you will also find the version identifier we are currently able to detect more than 600 different version identifiers and we collect all of these version identifiers we aggregate them together and at the end we have the software component we have the identified version and now we can start querying databases like the CV database with the CV database we get the known vulnerabilities with ferrating the CBS score and finally with the CV identifier we can then query further databases like the metasploy database for quite stable exploits or the exploit database exploit
DP or packet storm or whatever we want and you get a quite good overview of the real world exploitable exploitability of the firmware and all of this happens in the background and you're now able to focus on the interesting stuff so analyze the framework for fresh exploits for zero day vulnerabilities usually this is the goal of every penetration tester to find currently unknown vulnerabilities and you want to spend as much time as possible on this so um I will show you a little vulnerability we found ages ago in a home consumer device it is in the command.php file the interesting thing now is that the command.php was not interlinked from the web interface so from a black box approach there is the
risk that you are missing this file and you're missing a critical vulnerability as soon as you have access to the fender you can walk through the firmware you can find all the script files and you can analyze them and this is quite time consuming again but Amber can do it also so Amber can use this uh can check all of the PHP files and use same grab for example to do static Source static code analysis and as everything is interlinked on the HTML report you can just click on the on the possible results for your further teardown and you get direct access to the source code with the suspicious area highlighted and you can see there that the vulnerability
is easy to detect is the CMD parameter is passed to execute function and then it is just executed on the operating system so hooray we have found our first zero day but Amber can also help you in not only have and but cannot only help you in um scripting languages it can also help you in identifying the juicy stuff on on binary level so you can see a typical output from Amber um this is from a firmware that was analyzed a while ago from a security researcher and guessing which binary there were the vulnerabilities yes in the NCC binary and Amber checks every binary for legacy C functions like string copy and if there are Legacy C
functions very often used there is the risk that something goes wrong and then we are matching this with with some other POS or some other interesting criterias like um is the um like an educated guess if the binary is a known Linux file or is it it is not a known Linux file if not then it is probably something from a vendor that won't take a look at it and it shows the binary protections are the symbols in the binary and finally um is are the network capabilities in the binary but now we have a zero day vulnerability we have a binary in the next firmware that is quite suspicious but till today or till now we do not know if something
of these vulnerabilities or possible vulnerabilities is exposed and exploitable so we fought a while ago how can we improve this now and we introduced the full system emulation framework and with this framework we are now able to move away from the static-only approach in finding vulnerabilities to a more Dynamic approach so in an Ideal World we can just boot up the device and we can verify our vulnerabilities we're using a technique that is called qar that is called emulation um we're using qaml for this and according to Wikipedia an emulator is Hardware or software that enables one computer called The Host which is our for example our Kali Linux to behave like another computer called the guest
the guest is the firmware the embedded device and we are not trying to execute code or binaries from the guest from The firmware on our host system on our Kali Linux system at the end we are trying to boot up the firmware on on a different architecture with we we're dealing with with a lot of issues there we have a different architecture we have a different kernel for example and so on and so on nevertheless um it was 2016 where as a research project called thermadine showed us that it is possible to automatically boot up a lot of firmware to a state where we can interact over the network in 2020 the successor project firmai
improved the success rate massively the problem now is that both projects are not actively maintained anymore and so we thought about this issue and we decided to do a complete re-implementation as Amber modules with this re-implementation we can now further maintain our this emulation engine we were able to improve this emulation engine in multiple areas like we can now we are now supporting more architectures than before and now it is possible to automatically use this full system mode emulation during our firmware analysis during our automated firmware analysis and now let's go back again to our serial table nobility that we that we found out a little bit before um we know that there is a vulnerability
but we do not know if the vulnerability is really exposed and we can exploit it so now Amber is doing its magic it is trying to emulate the firmware and it shows us the final emulation state or it tries to to Ping the or the emulated firmware it tries to do an end map port scan and if there are web interfaces detected it shows us a nice and shiny screenshot of the web interface and we can see that there is a configuration interface probably working because we got a nice and shiny screenshot from it but Amber is doing much more now and by scrolling the whole web interface or the web server for the whole family for all
of the femoral files so at the end we know which files are exposed via the web interface so we can see that the command PHP file is exposed by the web interface and now Amber is doing um cross checking to or if there are further results already available and MSC is okay we have already a possible vulnerability um identified via our static source code analysis and now you can just again click on it do some manual analysis on there Additionally you can now use this for system mode emulation to further exploit this vulnerability you can write your own proof of concept you can write or exploit for this vulnerability now in emulation without owning the real device
and for this talk we thought about um if we should show the exploit development process now in emulation but on the other hand we thought okay then you can see that it's possible but we know that it's possible so probably it would be much more useful to give you the possibility to not just use one exploit or one proof of concept in your firmware analysis in the future but to use I think more than 2 000 exploits in your firmware analysis process in the future so we integrated the meter Sprite framework as one of these analyzer modules for the live live testing and now um during our firmware analysis process it is now possible or Amber is doing
again some cross checking um I've mentioned before that we are doing Port scanning on the emulated system um so we can now cross check against the meter spray database with the default ports we are doing cross checking regarding the operating system and then we're using the Metasploit framework in an automated way to first try to check or use the check functionality from every module if there is no check functionality available we are trying to really exploit the vulnerabilities and you can see it here that the vulnerability the command of PHP vulnerability now was identified and verified with an successful exploitation attempt at the end you get again a nice and shiny HTML Report with all of the
vulnerabilities that we were able to verify um via exploitation and probably the seasoned security guys here can remember the DB Auto Pound feature from your display which was removed years ago this is more or less DB Auto Pound for firmware but in a safe and secure environment because it is an emulated environment now so um we have shown now that firmware analysis is not only configuration analysis it is not only generating an s-bomb it is more in hybrid or mechanism it is static analysis it is dynamic analysis and at the end you get a much better understanding of the real world risk of the firmware and um last week we have released version one three zero from Amber and Debbie
introduced a quite interesting nice and new feature the AI assistant firmware analysis so from now on you can also use chat GPT to get a second or a second opinion a second meaning of the of the possible vulnerabilities so thank you very much I think do we have some questions now
so most of the discussion was on Linux does this support analyzing other real-time operating systems or other situations other than embedded Linux um so so Linux is quite a nice Target because you can do a lot of analysis but automatically we also support the real-time operating systems but with a limited um analysis mechanisms so we we are able to generate an s-com for these real-time operating systems we are also supporting um UAV analysis via um the FW hunt Mac open source project from brinally so we are trying to also support other operating systems other than Linux yes
yeah so I'm curious about the dynamic part of the emulation so if you have firmware for an MCU which is not emulated in qmu for an example um are there any future plans for how you would extend it if it's not possible to emulate it at the current time or um yeah currently if qaimo is not able to emulate it then we are definitely failing um and um if qmo is able to support these uh controllers in the future then uh they're definitely uh plans to make it also possibly number so we are actively maintained the firmware emulation mechanism because it is so great and so helpful thank you very much it's very cool project thank you
so I do need to ask what is the AI assisted Edition so um we can query chat GPT via the API so um currently we get results from semgrab and other static source code analysis tools and if you enable the AI assistant mechanism then we upload these files to Jet GPT and ask GPT for an analysis and the results are quite interesting and quite good so you get a quite a nice second opinion on the vulnerability that are already found via other tools and in the future we are planning to also include this possibility for not analyze uh static already found vulnerabilities so that we can extend it massively in the future would it do it also with some binary
code you showed us a string copy or this kind of thing and maybe trying to get the assembly around the function to figure out if it's somehow available we have already we did some tests on with the um with the decompiler from radary2 but the results were not that good um but probably in the future if the results are getting better then probably we can also do some some interesting analysis via VIA GPT there thank you that's very cool you're welcome
so um you talked about binary analysis what about compressed binaries like UEFI images and stuff can you give us more details on that um um you mean what analysis mechanisms we are using for finding interesting stuff for binary analysis yes um first of all we are we are counting the usage of Legacy C functions um how many times they are they're used and this is not by itself a vulnerability but the more often they are used the more likely it is that there are some problems um in there we are we are testing the binaries regarding um regarding the binary protections we are testing the used functions in the binaries um if there are functions used that are
that are indicators for network activity and um we we are doing some further analysis where we are generating from every binary uh the the whole output via by the strings command and we are analyzing this output for private keys for passwords and um some some other stuff like this so multiple things that we're doing in the background here
more questions then I think we are ready to go for lunch thank you very much
[Music]
thank you [Music] thank you foreign [Music] thank you [Music]
foreign
[Music]
[Music] foreign [Music] [Music] foreign [Music] foreign [Music]
[Music] foreign [Music] foreign [Music] foreign [Music] thank you [Music] [Applause]
[Music] thank you [Music] thank you [Music] [Applause]
[Music] thank you [Music]
baby [Music] giving me winter rain some kind of butterfly baby
[Music]
I don't wanna overthink it baby [Music]
some kind of butterfly baby
your whip up my appetite [Music] but I don't wanna jinx it baby [Music]
[Music]
[Music]
maybe you'll give me [Music] directions [Music]
[Music] some kind of butterfly baby
[Music]
[Music] oh oh [Music]
[Music]
[Music] thank you [Music] foreign [Music]
[Music]
[Music]
[Music] foreign [Music]
[Music]
[Music] foreign [Music]
[Music] [Music]
[Music]
[Music] moving up
[Music]
[Music]
foreign [Music]
[Music] thank you foreign [Music]
[Music] thank you [Music] foreign [Music] foreign [Music] oh yeah [Music] thank you foreign [Music] wow [Music] thank you [Music] foreign
[Music] thank you [Music] foreign [Music] [Music] foreign [Music]
thank you
[Music] foreign [Music] thank you [Music] thank you [Music] thank you [Music]
[Music]
[Music] thank you [Music] [Music] thank you [Music] foreign
[Music]
[Music]
[Music] thank you [Music] foreign [Music] thank you [Music]
[Music] foreign [Music]
[Music] [Applause]
[Music] thank you foreign [Music]
[Music] thank you [Music] foreign
[Music]
[Music] baby [Music]
[Music] don't wanna overthink it baby [Music]
[Music] baby
everything don't leave me [Music] but I don't wanna jinx it baby [Music]
[Music]
oh [Music]
maybe you'll give me five years I'm gonna butterflies baby you'll give me foreign [Music]
[Music]
[Music]
oh [Music] oh [Music] [Music]
thank you [Music] foreign [Music]
[Music]
foreign
[Music]
[Music]
foreign
[Music] foreign [Music] [Music]
[Music]
move it up
[Music]
[Music]
[Music] thank you
[Music]
foreign [Music]
[Music]
[Music] thank you [Music]
[Music] you're ready [Music] oh yeah [Music] thank you [Music] thank you all right
[Music] foreign [Music] foreign
[Music] thank you [Music] thank you [Music]
thank you [Music]
foreign
[Music] foreign [Music] foreign [Music] foreign
[Music] foreign [Music]
[Music]
[Music] foreign [Music]
[Music] thank you [Music] [Music] foreign [Music] foreign [Music]
[Music] thank you
[Music] foreign [Music] foreign [Music] thank you [Music]
[Music] thank you [Music] [Applause]
[Music] thank you [Music] thank you [Applause]
[Music]
[Music] thank you baby [Music] you're giving me wind away [Music]
[Music]
[Music] don't wanna overthink it baby [Music]
[Music] don't leave me [Music] jinx it baby foreign
[Music] [Music]
[Music] oh my God [Music] don't leave me alone baby you give me rain it's some kind of butterfly baby
[Music]
[Music]
oh [Music] forever [Music]
[Music] foreign [Music]
[Music] foreign [Music]
[Music]
thank you
[Music]
[Music] moving up
[Music] foreign [Music]
[Music] [Music]
[Music]
[Music]
Move Along
[Music] foreign [Music]
[Music] thank you [Music] thank you [Music] thank you
[Music]
[Music] thank you [Music] foreign [Music]
[Music] no no [Music]
thank you [Music] foreign [Music] foreign [Music] awesome [Music] foreign [Music]
[Music]
thank you [Music]
[Music] foreign [Music] foreign [Music] foreign [Music] foreign [Music] foreign [Music] thank you [Music] foreign [Music] [Music] thank you foreign [Music]
[Music]
foreign [Music]
[Music] foreign [Music] foreign [Music] [Music] thank you [Music] foreign [Music]
[Music] foreign
[Music] foreign [Music] foreign [Music]
[Music] thank you [Music] [Applause]
[Music] foreign [Music] [Applause]
[Music] foreign
[Music]
[Music] you're giving me wind away some kind of butterfly Maybe
[Music]
[Music] don't wanna overthink it baby [Music]
[Music] don't leave me [Music] but I don't wanna jinx it baby again
[Music] but I don't wanna miss you baby [Music]
oh [Music] foreign [Music]
foreign [Music]
[Music]
[Music] oh oh [Music]
[Music]
[Music] foreign [Music]
thank you [Music]
[Music]
[Music]
[Music]
move it up
[Music] foreign [Music] foreign [Music] [Music]
[Music]
[Music]
[Music]
[Music]
thank you [Music]
[Music] thank you [Music]
[Music] thank you foreign [Music]
[Music] thank you [Music]
[Music] foreign [Music] oh yeah [Music] thank you [Music] foreign [Music] foreign [Music]
foreign [Music] foreign
[Music]
[Music] thank you [Music]
hello everyone we are going to start
hello thank you thank you for joining uh our lecture for today is Google workspace forensics insights from Real World hands and instant response my name is Don Carmi and on my left is my friend Ariel shelf we are both senior Cloud researchers at mitiga and also we have some something in common and today we are going to talk about Google workspace forensics we're going to share a little bit about what is Google workspace a little bit about the log structure and some challenges that we found in in the logs while performing forensic investigations we are going also to talk about what we did in over to overcome those challenges and share some real cases that we found
during instant response and threatens at the end we are going to show a particular visibility Gap that we found in Google Drive so let's start so first of all what is Google workspace as you may know Google workspace is a cloud-based collection of tools that were designed to make collaboration between individuals and organization much easier it included it includes many services such as Google Drive Gmail Google Calendar Google keep and more what you have to know for this lecture is that this is a very very popular platform there are more than six million paying businesses all over the world which makes it a high Target for threat actors to uh to exploit and steal data
a little bit about the logs before we dive into it into them so first of all they are divided by the service once you enable Google workspace logs they are divided by the services that you have enabled they are collected in near real time and the typical retention period is six months with some exceptions today we are going to be focused specifically on Google Drive but everything that we are going to say apply to all the logs in Google in Google workspace but before we start we would like to share with you a story cool hi before though on dives into the log structure I want to share with you a story one of our customers saw that internal
data was published publicly and want us to investigate it so we got to work part of our investigations investigation within basic anomaly detections and found suspicious activity we saw an external user from gmail.com that performed approximately fifteen thousand thousand download events at the same time step it's a lot and it was really interesting and you had a lot a lot of questions about that for example what are the file paths who created them who shared the files externally in order to answer these questions and more now we're going to learn about Google workspace log structure with Dawn so let's talk about the log structure on your left hand side you can see you can see a typical log record from Google
Drive we can see many pieces of information that are relevant for forensic investigation for example we can see the caller which is the email which is the entity that performed action you can see that we can see the IP address and we can see the application name and more what we also can see is a list of dictionaries which called events which we expanded here on the right hand side for example we can see an event of type upload and its parameters the parameters is another list of dictionaries that represent the parameters of each call so what we can see here is actually two lists of dictionaries for each log entry this is quite quite challenging
as you may agree but other than the other than the log structure we found other challenges in Google Drive and Google workspace logs that we are going to show you right now for example the first thing user agent field is missing there is no user agent across all the logs of Google workspace we can see many types of information again such as the IP address the type the event name and more but there is no user agent and it might be challenging when you want to perform some anomaly detection without having the user agent another another thing that we found is IP addresses inconsistencies here we can see three download events coming from the same user which is blurred but this
is the same user from Google Drive all in the very same second but from three different IP addresses this could be misleading the investigator while performing performing the investigation another thing that we found is that there are no path in file related log entries for example in download event you cannot know from where this file was downloaded you can see some information about the documents such as the document title the document type the document the name and ID but you cannot know the path so we cannot agree right now that this is quite challenging but what you can do in order to make it much easier to read to investigate and to be ready for an
attack so let's talk again about the log format and specifically about the events what we can see here is a list of dictionaries that represent the events of each log entry and why there is a list of diction a list of events this is because the way Google tied different events is the following there is one event that will be marked as a the primary event and this is actually the action that was taken by the user for example in this case this is upload but following this event this event triggers other actions in the in the background that are all all related this is quite difficult to understand so what we do in order to make it easier to
investigate we split each sub-event in the events field into a dedicate dedicated row so here we can see three different events that originally were part of the same chain and we splitted them into different roles try to think about the case that you would like to search for all the files that were publicly at some point in your organization you don't care if it was created it's public or was private and then moved to be to be public and you just would like to know that it was public at some point with this technique you just need to search for the event name that represent Public Access and you will know all the files that were public
another thing that we would like to highlight is the talking about the parameters the parameters represent the parameters of the call that was taken for example here again we see the upload event and we can see a list of dictionaries that represent the parameters of the call for example we can see that this is this is a primary event we can see the document ID for example and we can see whether this file is encrypted or not what else we can see is that each dictionary has two types of keys first one is the name the name of the parameter and four other keys that represent the type of the of of this key of this power of this parameter
for example the first one primary primary event is Boolean the second one document ID is string because the value is populated and the third one is also Boolean in this case for our research we found out that whenever everything is null it means Boolean set to false so again this is quite challenging you would like to investigate something you would you would like to be fast you wouldn't you would need to understand the logs uh right away so what we do we restructure the data format we actually omit all the type related keys and we leave only the parameter name and its value this is much easier to read much straightforward and the investigation can be much quicker
the 13th third thing that we do is we enrich the data under the parameter in time some occasions you may see the originating up ID not sure if you can see here originating different parameter and this idea to present the application the two connections on behalf of the user so in the log you will see that the email address or the actor that took the action is someone in the organization but actually this action was taken by by an application and sometimes this is important to understand during an investigation for example in this case this is the slack application remember earlier that we talk about IP address inconsistencies this could be one of the reasons why we see those
inconsistencies because sometimes the IP address would be the IP address of the hosting provider of the application but the actor will be the user and this mismatch could be uh confusing so understanding this is coming from an application would help the investigation now back to Ariel to tell us more about the exploitation case thanks Don now we are going to talk about that excitation from Google Drive let's start from the basics there are six event names that may be related to that acceleration in Google Drive the most obvious of course is download set actor also can View files they can send them in email as attachment they can print them you'll note that they don't need to physically print them in
order to accelerate huge amount of data they can print them to PDF files of course they can play with them and the least intuitive they can copy them to more convenient location for example to a public folder when you suspect the user you can search which acceleration related events this user performed and also in threatened when you want to generate leads you can search for anomalies in these events appearances for example this is an anomalies graph based on these events each line is the user and as you can see we can see how many acceleration related events each user performed over time for example the green user here performed on February 27th approximately 20K of exfiltration related events and it
might be really interesting to investigate it now let's talk about sharing files in Google Drive we are going to talk just about about sharing files so when you share a file of folder in Google Drive this window pops up in this window under the general access section you could choose a group anyone with a link your organization or restricted restricted means just user users that you explicitly mention in the upper section get permissions to this object and also you can choose access scope viewer commenter or editor in our example we changed the the group from our organization to anyone with a link this click actually generated four events to change document visibility events and two change document access
scope events in this table also you can see three parameters we extracted from the parameters column Target domain old value and new value at First Look it may be really confusing but when you are looking back you can see here a pattern in the first couple of events the start State people within domain with link and can view access scope changed to the clean State private at none after that in the next couple of events the clean State changed to the end State people with link and can view access scope you'll note that even though we didn't change the scope it still goes through the clean state to none now let's talk about how share file or
folder with a complete principle looks like in the log when you share a file with concrete principle it's straightforward there is one event that's called changes or access and the act of this event is the user that actually performed that easy but when you shall folder with conclude principle something interesting happened for the main folder there is one event that's called changes access and the actor of this event is the user that actually performed that but after that for each file and folder recursively under the main folder there is special event that's called change user access hierarchy consiled and the act of these events is system and you'll note that all of these events is by our primary events and not part of
a chain of events like don't describe Delia now back to our story from the beginning just a reminder we saw an external user from gmail.com that performed approximately 15 000 of download events at the same time stamp and it's a lot one of one of the questions we asked ourselves was what are the file paths the straightforward solution of course is using API but there are two problems with that first using API to use API you need proper permissions and when you are an external investigator you don't always have them and always and there I'm sorry and the second when you when you use API calls you get the the current state of the organization and when you investigate
you want the historical state so we try to think what what we can do in order to get the paths based on the log records only in our research we saw that for each file or folder creation there is a great event of course but also there is a to folder event there are the document I did title and also there are the destination folder ID in title based on these events at folder events we built this table in this table you can see the document ID the destination for the title and the destination folder ID now if you think about that if all the destination folders you have also the etfolder events you can try to search these ideas the
destination folder IDs in the left column doc ID and try to build the paths recursively so that's what we actually did this table is from our lab don't worry here you can see the event names the document title and the calculated document path we build in this cool technique you'll note that in this technique the paths might be partial of course depending on the log time frame if you don't have the relevant at folder events you can do that back to the story research just to do to to close the story research in the logo which user share externally the files we saw this user was an admin user long story short this user was compromised by
phishing attack and after we understood that we investigated the logs in the relevant time frame finally I want to share with you a visibility Gap we found in Google workspace logs it two months ago when we investigate when we investigate we assume consistency in the logs what do I mean all of us already know that there is an event about download file so every time a user download file there is a log record about that right so it's not as simple as that let's talk a little bit about licenses in Google workspace each user has the free license Cloud identity free and this license enables basic features in addition an admin can purchase all the
licenses in order to enable more features in this example you can see that in this in this organization there is a paid license that's called Google workspace Enterprise Plus but this license isn't assigned to this user in other research we found that if a user doesn't have any paid license there are no log records on their private drive at all not about download files copy files create files and so on it's crazy to think about just with the free license there are no log records on their private Drive in organizational Google workspace based on this finding we'd like to think how to attack them can exfiltrate not just the private drive but also the share drive with
minimum logo codes now we want to share with you a use case house attack though can perform something like that in this use case the compromise user is an admin user because an admin user has the permissions to revoke and decide licenses so whose case the Theta token evoke the paid license to the compromise user copy all the files from the share drive to the private Drive download all the files from the private Drive and finally reassign the paid license to the compromise user to be discarded as possible now let's talk about the logs for the revoking we assign the relevant log records under admin audit log user license revoke and user license assignment for the copy files actually it's
interesting in general for each copy file in Google Drive there are two log records Source copy on the original file and copy on the destination file these events are almost the same so usually it's not interesting to monitor both but in our special case there are no copy events at all because there are no log records of the private drive so they are just Source copy events and for the download of the files there are no longer called the top based on This research we understood that in our investigation we should search also for license revoke and design in showtime and also we should search for Source copy events without related copy events don't think
so let's talk about what what we are today in this talk so we talk a little bit about what is Google workspace how the log structured and what are the challenging in those logs we talked we talked about the challenges in the structure itself but also about some pieces of information uh that aren't presenting in the logs for example the user agent the inconsistencies of the IP had the file path and more a real cool use case of data data exfiltration and the visibility Gap that we found in Google Drive but now you might ask yourself what now what what do I need to know to do now so first of all we think that the first
thing that you would you would need to know you would need to do is to know the logs to understand the limitation of Google workspace logs to understand what it gives you what it doesn't give you and to be able to to know it before an attack before you need to actually perform an investigation in one week time we recommend you to start and facilitate the Google workspace logarithability exactly what like we showed you to split the rows to split the events into different rows to flatten the parameters and to make the logs ready for an investig investigation and once it's ready in one month's time we'll recommend you to start proactively monitoring for data
exfiltration cases from your organization to understand if someone somehow was able to actually exfiltrate data out of the organization this was our talk on the left QR code you can see a link to our blog where you can find more more information and the right queer code is for our advisory that we shared with Google thank you very much [Applause]
I think it already the mic
check check foreign well thank you for the presentation that was great uh I'm curious so I've done uh I've I've set up uh event logging where I could have a notification or an alert if stuff happens in gcp uh in like their logging they have a logging product where they can actually show you know kind of more or less the same thing but for you know different gcp applications do you know if if Google workspaces and gcp share the same resources for the back end where I can actually query uh those logs from the logging platform gcp they usually Google Shares a lot of resources on the back end they share a lot of platforms
store them in some other solution to be able to query it cost query those resources but we need to check that I'm not completely sure
[Applause] [Music] thank you [Music] foreign [Music] foreign
[Music] foreign [Music] foreign [Music]
[Music]
[Music]
foreign [Music] foreign [Music] [Music] foreign [Music] thank you [Music]
[Music]
foreign [Music] foreign [Music] foreign [Music] foreign [Music] foreign [Music] [Applause]
[Music] thank you foreign [Music]
uh good afternoon welcome to b-sides Las Vegas uh this talk is discovering RDP vulnerabilities by reading PDFs presented by door Dali uh we'd like to thank our sponsors especially your diamond sponsor Adobe uh as well as our gold sponsors bluecat simgrip conductor one uh it's uh with their support that we're able to do this event along with other sponsors and donors um a reminder that cell phones are distracting these talks are being streamed live uh if you have as a courtesy please put your cell phones on silent uh even the vibrate is really annoying uh YouTube can hear them if you have questions at the end there'll be time there's a mic right there behind the projector and feel free
to come up and use that thank you very much all right so um welcome to my talk uh the talk is going to be unveiling day then uh discovering RDP vulnerabilities using PDF files now I know it might start sounds very interesting what is the connection between RDP and PDF files and this is what we're gonna talk about so first let me let me introduce myself so my name is door Dali I'm the head of security research at Cielo um I have some experience in the app sack and the infrastructure security award uh over 10 years in the cyber security field are pretty much vast background from all different aspects and now I'm the lead I'm the head of
security research at CEO now what we do in ceolo pretty simple secure access we help companies to secure their their high risk access between their clients and the Target computers and servers and this is what we also research we focus on research on remote access applications and remote access protocols now I do have a question for the audience before we start how many of you when you do some security research just really read a big PDF file from A to Z and just by that manage to find five different cves in a high use protocol because this is what I did and when I talk about highly used protocol I talk about something that we are all familiar with
which is RDP so if you are not familiar with RDP this is the the windows the dialogue that you most likely seen in your when you used your computer and you use it in order to connect to a Target server so if you want to have some access to a remote server you just use this dialog and you connect to a Target server that you want now RDP has been researched for quite a while but it has all different aspects and this is a pretty big protocol now one part of this protocol is something that is called RDP Gateway now what is the RDP Gateway RDP Gateway is a role in Windows server that allows you
to have some kind of RDP proxy that you can install in your DMZ so this is a server role and it allows you to create an RDP tunnel between a client that is outside in the wide internet to a server which is inside your on-prem environment and this works by creating a TLS tunnel between the client and the rep Gateway and then the rep Gateway connects to the Target computer so it's very important to remember those names the RDP Gateway and the target server because those are the stuff that I will use throughout the presentation so now that we know what is RDP Gateway we need to think of a research methodology how we can research
this thing that is called rep Gateway so I decided to take this quite a unique methodology for the research I decided no to use jitra not to use Ida not to use windy BG and of course not to use the crowd favorite caulk I just decided to read the manual this is what I did just read the manual from A to Z and understood how it works so when we talk about the manual we also used a little bit more stuff for the research we used in previous research and there was a little bit amount of research on the RDP Gateway so we used that we used some protocol analysis tools with the one that was famous is
Wireshark just to understand how exactly the protocol looks like we did use a lot some open source implementations now RDP is a closed protocol right it's built by Microsoft and it's pretty closed but there are all different open source implementations for this protocol such as free RDP is the most known one but the one thing that we used the most was protocol specs now if you are not familiar with protocol specs so actually Microsoft releases on their website a list of 1000 protocols a PD list of PDFs for one over one thousand protocols that they are using in their systems and one of them is the RDP protocol so all the information about how to implement and
how to use the RDP is over there on their website so also one of the protocol specs is the RDP Gateway protocol spec which is called msdsgu which means terminal Services Gateway server protocol so that's what we did we just read this protocol and this is a small glimpse for what this protocol looks like and how it works so first of all we do see that the connection starts with an HTTP connection so in order to create a TLS tunnel between the client and the RDP Gateway we start by creating an HTTP a TLS connection between the two now after we created the TLs connection we have some version and capability negotiation in order to tell the RDP Gateway what we
support and what we don't and by the end of it we get two things we get a tunnel and a channel now in order to understand what our tunnels and channels it's pretty simple I'll use some I'll use some analogy um so the tunnel is like the road that you have and the channel is like the Lanes on the road that you have so that means that you get one tunnel and you can get more than one channel on that tunnel why do you get more than one ton one more than one channel you'll have to bear with me for the end of the presentation to understand it because this is very important not for the first
vulnerability but to the second one so this is how it works now we need to find our first culprit right we need to find our first vulnerability that's what we are here for right we are all looking for vulnerabilities so we looked at the PDF file and we did notice a very nice pattern each and every time that the RDP Gateway sends a message to the client that connects to the Reb Gateway it explicitly says that the message need to be null terminated each and every time they specifically says null character but there were two different instances where they didn't say it they actually said just to send a message with no null character so we did it we as we said we are
following the manual right we are just reading the manual and we are following what Microsoft told us to do so we sent those two messages with no law termination and we actually noticed that the client kept on crashing pretty interesting just by following the manual I can get an RDP client to crash that doesn't make sense of course but that's what that's what happened now I'll talk a little bit about what are those messages so the consent message is a message that shows up once you connect to the RDP Gateway it allows you to to accept the Euler for example and the service message allows an admin to send for example to tell everyone hey this
server is going to be shut down in couple minutes so those are the stuff and let's see it live right now so as we can see on the left side on the screen we started in our AP Gateway and here we are connecting to a server using the RDP Gateway so we got connected to a local server on the Target on the place where we are running the server and now we sent a server message with an alternation and now we're going to send a message with no termination and we see that it crashed I mean I just followed the manual I didn't do anything crazy up until now so I know that I promised that we didn't
use any of the big guns like queen dbg or g-draw and all that stuff but when I saw it I was like I must check what's going on here so I looked into the stuck trace and I actually saw that there are some problems with the Heap now because because this was pretty easy to do first of all I decided okay I need to report it to Microsoft before I try even to exploit it I first need to report it Microsoft before someone else will find it because I mean that was very easy so I did send it to Microsoft and while I was sending to Microsoft I tried to try to exploit it but before I even got to
do it Microsoft told me that I got an rce on RDP client I mean pretty simple pretty soon feeling pretty insane okay so that was pretty nice I actually then later tried to understand what's going on and what was going on there and actually there is a one and off by one vulnerability in the Heap that kind of like messes up with the hip headers and stuff um that's the story and it's actually exploitable and can you can actually get an rce by sending this message so that was nice right but that wasn't enough for me I wanted something cooler I wanted to find something even bigger just by reading the manual I mean I just
heard the manual and I got an rce I can probably find some some more stuff there so that's what I did so remember that I told you about the lanes and the road so one reason that they allow you to have multiple channels on the same tunnel is to support UDP now as we can always as we probably all know UDP is a lot a lot more faster and allows you to create to have better performance connection so what Microsoft allows you to do with the RDP Gateway is to create a secondary Channel which is a UDP channel so by the end of creating the main Channel which is the TCP Channel you get something which is
called a cookie now this cookie is the one the only authentication mechanism that you are using to authenticate to the ud to the UDP socket of the RDP Gateway so all you get so you get the cookie and you just send it back over the UDP socket and once it verified the cookie you got authenticated so I was like cool what is this cookie contains right so let's let's understand what it contains so the manual told us this is inside and encoded byte blob that contains a struct called authentic cookie data um and destruct contains a lot of very interesting stuff first of all it contains the username it contains the scheme which we are
using which is UDP it contains the expired time the server AP that we want to connect to the server name and the port that we want to connect to so this mean that we have a lane which we can actually control and divert to a different tunnel to a different Road so if only we can manage to understand how we can forge this cookie so that's what we try to do so we looked at the byte stream now they didn't tell us how it signed or already encoded but once I looked at the byte stream I noticed that from I noticed a pattern that was very familiar to me I noticed that the two first bytes were
0x30 and 0x to 82 in HEX now if you have ever played with certificate and pcpkcs so I noticed that this is an ASN sequence now it was it's also very easy to understand if you convert it to base64 and you see that it starts with Mii which might sound a little bit more familiar to the audience and then when once I understood that this is an ASN one I need to understand what type what exactly type of encoding it was and this was something called CMS cryptographic messaging syntax now what is cryptographic messaging syntax it's pretty simple to understand essentially you've sent the data the sine data along with the signature and the public key of
the private key that used to sign the data so this means that anyone who gets this byte that contains the data and the signed the signature and the public key can actually verify the design data was in tempered with right so what do we do if we want to try to check if Microsoft worked correctly here we just take it we do something very simple we take the part we create a self-cent certificate and actually put our own public key that was that was signed using our self-cent certificate in that byte and sign it with our on-sell signed certificate and that actually worked we actually managed to get an authentication bypass now what it gives us
first of all we get Event Viewer log forgery second of all we get the full ssrf from the RDP Gateway into your internal network over and all of that over UDP protocols such as dhtp SNMP DNS system whatever UDP protocol you can think of we can get an ssrf into it the third thing is Network denial of service because this is a UDP what Microsoft decided to do in order to work correctly in order for that to be more reliable I would say for each time that you are connecting they are sending three packets over the network to make sure that at least one of the packets will get there so by sending one packet I get
an amplification of three times that is pretty cool and all of that have been vulnerable since windows 2008. which is pretty insane now let's see it live so as you can see on the left string I'm saying I'm creating a cookie with all the data and if the researcher and I'm trying to connect to besides LV server now of course and all of that import 5514 which is a syslog port and I'm connecting to a broadcast address now what happened is that I'm sending three broadcast messages into this log in the network with some data which means that I'm going to spam all the syslog servers in your network and here you can see how I created a
simple Event Viewer log which can which can make all the same people in your organization go wild to understand what the [ __ ] is besides LV and we will always be who is if the researcher so that was pretty nice
so overall during this research just by reading the manual I managed to find four four different cves we add the RCA that we talked about we add the authentication bypass that we talked about we had another cve which was actually the fact that they were using an old version of TLS they actually used a TLS 1.1 over UDP in their system and there is one last TV that we cannot hear this close because they haven't fixed it yet and if you ask me what should be your key takeaways from this presentation first of all the fact that widely used protocols such an RDP such as RDP still have many vulnerabilities in them and you need to be aware of that second of
all closed Source isn't necessarily closed I mean we have the manual we can use it and just find vulnerabilities with it right third thing read the manual I mean please if you're a researcher just read the manual before you try to open all those crazy tools and all the reverse engineering tools and if you're not a researcher and you're just a Defender read the manual as well to understand where it could be the problems in your system and the last thing patch your software because those vulnerabilities have been existed since Windows server 2008. thank you very much [Applause]
all right if anyone's got any questions feel free to come up and use this mic
all right thank you [Applause] foreign [Music] foreign [Music] [Music] foreign [Music] thank you [Music] foreign [Music]
[Music]
[Music]
[Music] foreign [Music] all right [Music] foreign
foreign [Music] thank you [Music]
[Music]
[Music] thank you [Music] thank you [Music]
[Music] thank you foreign [Music] foreign [Music] [Applause]
[Music] foreign [Music]
thank you [Music]
[Music]
baby you'll kill me [Music] baby
[Music] us back [Music]
[Music] don't leave me oh but I don't wanna jinx it baby [Music]
[Music]
[Music]
maybe you'll get me [Music] away [Music] don't leave me alone [Music]
butterfly [Music] baby
[Music]
[Music] oh my God [Music]
[Music] foreign [Music]
[Music] foreign [Music]
[Music]
thank you
[Music]
[Music] thank you
good afternoon uh welcome to besides Las Vegas this talk is by Aaron it's a it is powerpc emulation and transition I got a few announcements before we start we'd like to thank our sponsor our Diamond sponsor Adobe and our gold sponsors Prisma Cloud send grip Plex track it's with their support that we're able to do this event and keep going next year we also want to thank all of our donors the volunteers everybody a quick reminder about cell phones if you have them in your pocket please put them on silent uh even the vibrate's really annoying so please make sure it doesn't show up on our recording later um if you have questions at the end uh
there's a microphone by the projector there feel free to come up and ask them uh Aaron said she'd be happy to take questions thank you so much
five seconds while I remember to turn my phone on Vibe on silent
there we go all right so let's get going I am Aaron I can't wander oops sorry bad habits we'll see how this goes uh I'm Aaron Cornelius or Acorn uh take your pick and if you want to know what the hell the thing in the bottom left is go to onionchark.com it's got a little background story there um pronounce see her um I am and I've been uh seen I'm currently senior staff security researcher or something like that at Grimm I've been been at this job for a little over six years in this particular job um I in this particular job I do a lot of uh you know uh reverse engineering Hardware software do vulnerability research I do
training development I give training one of the things that is one of my key thing key Passions sure that'll work key passions is helping to teach people and helping to Mentor people and now I'm going to give all of you one of the things that I tell you something that I tell the people that I Mentor which is if somebody knows what they're talking about and somebody sounds like they know what they're talking about there's only three possibilities one is that they're full of [ __ ] which is an extremely popular option second one is that they've been working with that system you know software whatever for like three plus years last option is that they made the thing right they
created it so they know everything about it probably everything um so that's one thing I wanted to say up front here if it sounds like some of these things are confusing and you're not familiar and it feels intimidating or for me or for anybody else speaking anywhere this week just remember that after you have had experience doing the same thing for like three plus years you'll also know things and you will know way more than people who've not had experience with it so just some level setting there so this talk is about emulation and powerpc and transition we'll get to that um we'll get to the story time in a little bit but before we do that I need
to lay some technical uh Baseline here uh why emulate something um there's a couple different reasons uh in my line of work I specialize in cyber physical systems uh so it's bare metal systems we don't get quite as much visibility with the debugging and so trying to attach jteg debuggers to embedded systems can be problematic at times or it can just be slow I've worked with a lot of you know really weird architectures and really weird piece of crap debuggers and they're slow and they make everything kind of more annoying than it feels like it really should be so that's one reason that uh you know having an emulator can be handy also full system emulation provides a
lot more opportunities for collecting information and for uh you know especially for doing reverse engineering the slide mind notes it can be in reverse engineering for example if you have a program that has tamper Protections in it right anti-debug features on Linux one of the common one of the ways that you get debug access to a program is with the p-trace system call so programs which p t r a c e the p-trace system call um so if the program itself wants to stop anybody else from debugging it it can just call the p-trace system call for itself and register as a debugger which means then that nobody else will be able to debug that particular
application they'll be prevented from it uh then so then that program itself knows that if once it registers for that nobody else can attach after that if it attempts to register and somebody else is already debugging the program then the program knows immediately that it's being debugged and it can take the appropriate action to exit or whatever to make my life annoying basically but if the if you have full system emulation you don't have to rely on the real system calls you can just return whatever information you want to return right you can just say yeah there's nobody else debugging this program no problem carry on and the reason that it works fine because you're not debugging it right
you're not actually debugging you're emulating everything there's a tool called vivisec which is going to be key in a little bit and for vivisec uh it uses emulation in some interesting ways it's actually one of the ways that it finds the finds what is and is not a function it takes a block of code and attempts to decode each instruction as it goes and if it you know hits a proper return at the end of something then it knows this is a valid function that's how that's how it uses emulation to actually do disassembly and also you can do kind of more targeted type reverse engineering where you take a function that has been found
and you start emulating it and you fill up the registers they're the emulated registers and memory with what's called a tank value which then allows you to track what results as the program goes are affected by those input values and if all of a sudden you get a program counter that's set to like the tank Value Plus a certain Value Plus some other thing then you know immediately you've got code execution in this function if you can provide if you can you know manipulate the input parameters so basically if this all sums up as it's useful so there's a lot of other emulation tools out there where I'm making your new one there's one called qmu which is
very popular open source emulator uh it's code base is terrible this is my personal opinion sometimes things just don't mesh well like if you look at a code base that people have written if any of you work do any sort of development whatsoever you'll know that people have different development Styles and sometimes they don't really play Happy Together Like the Way one person does things doesn't make sense to somebody else that's just the way it works people think about problems in different ways so existing code bases don't make me happy um if you and also they can break like if you've ever tried to emulate a full system Raspberry Pi image last I try to
do that it was actually fully broken even though it was trying to do a Raspberry Pi 3 and it's because the uh the ethernet device the emulated ethernet device that tried to have you the instructions I'll tell you to add doesn't work um so essentially you know in the end also there's because I want to right there's something very valuable about making a new tool you learn a lot about how things work you you know and you gain new knowledge and expertise in the process uh and also it was uh my job so you know that one kind of overcomes the rest of them but there's other good reasons too power PC I said this talk is emulation
powerpc and emulation powerpc why in the world does anybody using power PC who in the hell uses powerpc anymore uh and the answer is in embedded systems uh a lot of you know cyber physical systems if you must be that way uh and it's very common in automotive it's very common in Aerospace these industries have used it for a long time um you know at the time they started using these power PC chips back you know in the you know mid 90s whenever they first were created I didn't look up the timeline it's been too long since I've done that uh you know why were they using power PC versus something else maybe there's a
good technical reason at the time uh maybe it's basically because they want to right if somebody if an industry uses a particular thing if a company uses a particular platform and Tool they will just keep using it because that works right A friend of mine says if it works don't breathe on it don't [ __ ] touch it otherwise it's going to break right so if this system works and they know how to debug it and they know how to develop for it they will keep using that same exact platform short answer is they use it because they have used it um there isn't a whole lot that actually does emulation for power PC uh there's a
few things qmu does provide some but you can emulate like an early 2000s Mac with qmu but when we started this process I haven't checked recently but when we started this process any of the standard open source debug emulation tools they did not support some of the newer I should probably put some quotes around that newer power PC features like vle vle stands for variable length and coding you can think of it very much like arm thumb 2. um so there's an entire instruction set that's not supported in the emulators and also there's additional custom features for some of the embedded controllers that is not really addressed within current emulation tools um and um again you know uh very much
like the last slide it was also part of what I was assigned I was assigned to make an emulator for power PC so here we are right what project was I assigned to oh yeah the amp program I almost forgot I put it down there so DARPA amp program itself um I'm not going to go and read a bunch of stuff to you you can look at the web page right there if you want it stands for assured micro patching and you can read about it if you want more detail because I've already gone on too long without getting to the story I will give you a very quick summary of what amp is about
the goal here is to that this part project is to the goal is to make create or Advance the state of the art and tools that will take a binary and lift it up into higher level language let you modify that high level program whether it's C or something like C and then take that modified program recompile it back into a binary and then take the original binary and then patch it in in a way that's unobtrusive to the actual program execution the very last and most challenging in my opinion part of this project is to provide to make those tools be able to provide assurance that the uh you know the changes that are made don't
negatively impact the behavior of the program so that's you know what's been going on um I'm not going to be going into details about these particular tools because that's not really what I've been working on right that's the overall goal of the program there's a lot of other tools that have been talked about this year previous years um like ofrac from Red Balloon last year uh fish is going to be doing a talk about anger and anger's been making there's been a lot of changes going on with that I think that's a Defcon talk but there's a there's a bunch of different tools that are used in the industry for doing disassembly and reverse engineering and as well as new
tools being created as a result of this program doing some really cool stuff and that wasn't what I was working on what I was working on is testing those tools so part of the DARPA project is that there's a team that's actually does a um so there are companies AIS Cummins and Grim where the team that actually is testing and also um you know with uh some people from CSU are the team testing those tools the end goal is to be able to um find bugs in a real embedded controller because Cummins is a partner and that means that the end goal the end test is going to be on and a real Cummins engine controller a bug is going
to be placed in it needs to be found and patched and they need to be able to verify with all the complexity of a modern engine controller that the patch made does not negative does not negatively impact it but this Pro the tool the processor in this controller is a modern you know forgiven value of modern powerpc chip which means that there were no emulators that actually did the two did what is necessary for this emulate this Beast this is the particular chip and that engine controller and it doesn't really you don't have to look around and pay too much attention to this eye chart this is just from the reference manual from nxp um the things probably to point out here
are that in that little yellow I could probably let me see here here hey it works in this little yellow box here um you can see that it's got a couple different things here's vle we talked about that before mmu memory management stuff virtual memory there's also this block here called spe2 and that's really annoying spe2 is an nxp proprietary component that implements custom floating point and Vector instructions which means even if there was a standard emulator out there that did those things we potentially need to be implementing new custom instruction decoding and emulation also so you know there's there's there was work to be done at the beginning there and I'm going to show you this here this
is like the I didn't add I should have added up the number of pages all these reference manuals were I think it's probably around 5000 or more pages loads of fun but thankfully it it was a few year project right this is like three years I've been doing this um but I'm not going to dwell on here too much if you have questions about if you have technical questions about any of this stuff obviously feel free to get a hold of me afterwards and I'm more than happy to talk about it so almost a story time but we're going to real really quick mention this here Conway's law I don't know if anyone's familiar with this particular
um you know I not law idea concept the idea is that any organization that designs a system will produce a design whose structure is a copy of how that particular organization communicates if you've got four groups that are making a particular product then you're going to end up having four individual sub components and the way they talk together where they work together is based on how those teams communicate with each other so and you know in terms of at a lower scale I know this is true you know this should be fairly obvious because the way I write tools and you know great tools for myself is all based around how I think about problems right so the tools that I
write work well for me and they let me look at the pieces of information that I find interesting during the pro during the you know while using the tool this is also one of the reasons I encourage everybody to make their own tools because oftentimes the tools that are out there don't really mesh well and work for the way you think about problem solving all right let's try to solve particular problems so you know and like I said before making your own tool will help you you know learn more about it in the process anyway so it's designed for some story we'll talk about uh kind of how the you know how what the work that was done on
the emulator um the different challenges that were there not by now I you know assume it's obvious that I'm trans and I say figuring myself out um whoops what was that figuring itself out do I miss one oops oops
all right I'm not sure what that note was all about I must have typed out something here I'm not going to go into the nitty-gritty of a bunch of things because I don't have the time for it um and things can get a little personal um and also this talk is already way more personal than anything I've ever done before and I've been a little bit nervous about that hopefully it's not coming off too badly we'll see I'm sure I'll find out afterwards so what do making an emulator and power PC and you know coming out as trans have to do with anything they're not related well okay I suppose power PC in the emulator related to each
other um you know coming out as you know coming out as trans is more of a just a thing that happened at the same time as the other things I was started on this project well let's start here right so start here's the beginning this is the only surviving selfie I ever actually tried to take here take of myself before coming out I have no idea how it wasn't deleted um it's only really relevant in this particular situation because this is around the same time that I started thinking more in depth about you know myself and like gender do I have one Etc um worth mentioning right here that I don't mind old pictures of myself and
this is uh this is can be extremely personal for Trans people and it's as uh so don't assume anybody you talk to is okay with you know looking at old pictures or showing people pictures of themselves right it's an extremely personal thing um personally it doesn't look like me and it never felt like me so you know it's kind of more of an abstract thing only reason I find old pictures kind of interesting is because uh how much how different they look for me now it's also worth mentioning that this is probably around the time that at this time I'd already figured learned that I was ADHD diagnosed as an adult and I was slowly learning how that impacted me in
terms of you know how how that impacted life I guess for me it's probably the most generic way to say it um you know the story is about kind of emulation in power PC it's also essentially a part you know that's undercurrent of all this is related to reverse engineering we're developing an emulator because it needs to be emulated for this particular program that we're working on but we're doing an emulator in a particular way because it also should help us be able to um you know be able to use this tool in the future for doing reverse engineering right this is the one reason we structured it the way we did so we can get additional
analysis and information from it part of this is essentially me looking at the patterns in myself you know and mentally and slowly reverse engineering essentially myself and my own brain so enough of this let's move on summer 2018 uh you know work started on the power PC support in vivisect which I talked about vivisec is a reversing uh vulnerability research tool vulnerability research toolkit something like that I always forget the specific words there's also another co-worker at this time created a powerpc vle disassembler plug-in for binary ninja that's up on that GitHub address there vivisec which I should have put the address there also is that like github.com so uh let's see oops not that uh the next thing here
winter summer this you know kind of improvements were made the initial project not a whole lot of progress was made initially because Works seemed to keep getting in the way go figure um you know eventually then kind of the next year uh Grim got an award from NF theater to officially add an improved power PC support to vivisec because powerpc is so common in transportation they were trying to kind of push forward a bit of the state of the art in terms of you know tools for doing vulnerability research and um you know disassembling powerpc and embedded systems I didn't help with the initial stuff but I did make a load of unit tests for it I
made it the way I always do things which is I script it so I took an existing program I found all the instructions in the program and then I dumped those out with an existing tools and I made a giant file of tests to run through and like verify that the disassembly was correct and I found a lot of bugs uh in both our tool and in the commercial tools because there's bugs everywhere all right anyway so next uh here's the last picture I have before the pandemic started before I started growing my hair out um I had always wanted to grow my hair out but it for some stupid reason I felt that uh I should look professional
instead um which was stupid but you know that's kind of where I was at the time uh with regardless of all the other completely terrible things that have happened due to the pandemic um at the very least it was a good excuse for me not to grow not to cut a haircut I'd been working at Grimm for about three years at this point um and I've mostly overcome my initial imposter syndrome I hadn't fully grown in confident in my own abilities to like teach and Mentor at that time um also more mental progress type stuff I'd already figured out that I was probably autistic at this point um as a result of that it didn't really
change anything too much it's more along the lines of helping me realize that there's certain situations and like physical you know noise levels or whatever that become over completely overwhelming for me and um instead of trying to like power through it and push myself through those things I realize it's much easier and healthier and in the long in the short term you you and long term I recover quicker if I just take a moment go find a quiet corner to chill out in you know kind of reset myself and then I can get back to whatever I was doing right so it's more of a self-care type uh process um you know say uh poor air in here had
no idea what was in store for her uh in the next few years uh at this point I was pretty sure I was non-binary at some point um but I'd also decided that gender was all [ __ ] anyway so who the [ __ ] cares um I do want to emphasize very much right here that just because somebody feels says they're non-binary it doesn't mean it's like intermediate State it's very much for me at this time it was just easier for me to accept non-binary rather than accepting that I was you know trans woman and you know that's just because honestly I wasn't I wasn't ready yet it took a while for me to be able to accept
myself but you know thinking I was non-binary was less scary but yeah very much so and just want to repeat that one more time again that you know people who say they're non-binary you know they're non-binary right it's it's not like they're not yet decided that was just for me the way my brain was thinking so summer 22 um the DARPA amp program finally kicks off and my some of my colleagues started working on the emulator work I had not joined yet I was working on a different project um my colleague uh Matt created the initial emulator framework along with defining like the memory mapped i o reads and writes that would allow plugin like generic peripherals to do like
certain actions to happen when you read and write memory if you're familiar with how low-level embedded systems work when you read certain memory it lets you like read a message that's been received if you write memory to a certain specific memory address it allows like messages to be sent over a network the type of network and the addresses you read and all that junk are part of the reference manual so you just got to look it up depending on what you're working on um late fall I started helping Matt with the powerpc emulator which was really at that point was a lot of learning how vivisect itself works and how the emulator is capabilities in Vivid
sectors worked was working in December we hired a new Junior researcher also who didn't have a lot of Jordan who didn't have much experience with doing assembly or even or programming python but he said he wanted to learn and so uh a lot much of the next year so I was helping Mentor him to you know teach them how to do Python and how assembly Works how to decode instructions and so on and so forth I don't think he's too pissed off at me for teaching him power PCS his very first Assembly Language I hope I do feel a little bad about it so software Watchdog timer that's what swt is that's kind of the first peripheral
that was made one of the benefits is that it forced us to kind of come up with the way we're going to manage the tracking of time when a part when a system boots up it tracks like how many you know system clock ticks have occurred so this was the first go at it we tried to come up with something the way that was efficient it did kind of bite us a little bit in the end but it was good enough to start with that's often what you need to do when you're developing a complex system next thing that was done in SIU which stands for system integration unit and fmpll which is frequency modulated phase
lock loop again these are just words garbage words from the reference manual you don't have to care what they mean but they're related to basically getting the system to power up and getting the initial things to behave properly and getting the initial system clock to be set so with those three things done we were ready to actually start emulating the real code right so I have a 2350 the same 2350 uh that was on my desk cracked it open attached a debugger to it and ripped the firmware out of it I mean strictly speaking I did have another firmware image already but this one was useful because it was um I like having duplicates of things it helps me
confirm how things behave and also because I got the debugger were hooked up I was able to even though Hardware debugging is annoying and can be clunky and less easy than having an emulated thing to run it is a nice way nice to be able to actually hook up a debugger and confirm that when these particular values are set that the correct things that my emulator is doing the same behavior as the real thing itself like I'm able to read the correct values I'm able to see messages be transmitted and so on
so let's see that year so that was kind of the you know that was lead up to May next thing that happened here I had a talk at escar in 2021 I was held virtually obviously this was a professional headshot that my partner took for me my girl growing my hair out was going pretty well at that point um you know at this point I'd kind of come around to the idea that yeah maybe I wanted to be a woman at some point but like so what gender's all [ __ ] anyway right it's not like and my thought was very much I'm old now so who cares it's not like it's worth doing at this point in
my life um at least that's where my mind was you probably know how this story ends but I'm going to emphasize how very very wrong I was about that thing about it being too old and being not worth it next there's a lot of work that was done in May it was kind of easy when we didn't have a whole lot of framework to worry about because the more things that were added the more complicated adding new things in became and we also kind of chose what things to add initially because those were like the basis of how a lot of other things needed to work in the system so mmus how you configure uh for virtual
memory addresses it's also on powerpc fun fact that the um these are special purpose registers because everything in powerpc is special purpose registers it's one of the things that makes it kind of annoying but over here this mmu assist register two something like that yeah these flags here are the ones that control whether not a certain page of memory is vle and so if you want to know which parts of memory and a particular system are vle versus the regular 32-bit instructions you have to have the mmu configuration it doesn't have like a simple bit flag like it doesn't arm also fun fact in power PC systems you can configure different pages of memory to be big or a little endian this has no
real relevance whatsoever this is just one of those things that annoyed the [ __ ] out of me when I was trying to emulate this damn thing so over here you see this e flag that's the Indian flag just you know just in case you ever want to mess around with things and make terrible terrible power PC ctfs you can have a lot of fun with these things because nobody knows this or realizes these things next made flash you don't have to worry about the you know again you don't have to worry about this flash block layout this is just kind of the layout from the reference manual if you're not familiar with how flash
works in embedded systems you generally have to erase blocks before you can actually write new values to them which meant that to emulate how the system works because this is an engine controller and the way it takes updates over the can bus and there's a dealer tools that then are able to send new programs to it and if we're doing vulnerability research on this thing we absolutely want to pay attention to how new programs are written into it and how we can manipulate and affect that so I wrote flash emulation now this is um to kind of track and follow the proper process for uh erasing and you know rewriting memory this is also how I very first my bricked my very
first virtual image because I started everything up and ran it and then uh everything was going great everything the last program configuration worked perfectly and then the program that you know that the engine controller program reached a point where it was looking for some information for something I hadn't implemented yet and so it took some error path and the error path had it go down and update flash to indicate that this image is bad and it needs to take an update and then every single time I booted the system after that my you know say booted ran the emulator after that every single time it stopped working like I couldn't get the same code flow that I had before
and it took me an embarrassingly long time to realize that wait a minute something's different here and I took out a new Fresh Image off of the real hardware and compared it and realized that yes in fact this one here has some flag set that are not set in here um so that was kind of fun because it helped confirm that yes I was emulating Flash correctly it was also kind of frustrating because I didn't even consider that possibility at all I really should have but go figure uh interrupt handling exceptions these are things like if you had to divide by zero error or if you get like if something tries to read memory address
it doesn't exist right those you get these kind of low base level exceptions in standard operating system those exceptions are kind of translated into error signals that are sent to the program that's running at the time bare metal systems it's not it's like there's just one giant application running which means that it might install handlers for those things most of the time for many of these things uh the people programming the system assume that you know know that this isn't going to happen because it's not doing anything you're not like loading new things on there after the fact right you load the program on and it does the same thing every time you boot it for however many
years you're going to turn you're going to use this particular controller so these don't all have to be implemented for embedded systems um but the framework has to be there because that framework is still used for things like getting notifications when a message arrives on a particular you know Communications bus few more peripherals here um if you don't know about can feel free to stop by the car again Village of Defcon we have lots of fun teaching people how to do can um SPI serial peripheral interface um the Wikipedia page is pretty good for uh for learning how to uh spy Works ADC is analog to digital conversion that's basically taking a voltage value and
translating it into something that can be read um typically a fixed Point number of some sort ebi stands for external bus interface is how like you can add external RAM to the particular part um most embedded systems that have all these different peripherals they're called system on chips or socs and they also usually have internal memory some internal SRAM but typically they also have a way to add in extra memory and then I've got here summer to Winter lots and lots of integration right this kind of you know this kind of uh goes through a bunch of these things here from August on trying to take one of the real challenges that was developed and
given to the different teams or demonstrated to different teams and trying to make sure see how it works all together in the emulator itself it was basically a lot of work developing these things and you know um like a month or so in August and then after August going through and you know trying to fix all the bugs that I wrote or things I did incorrectly how do we test these things make coming up with the test because these are it's all fairly complex probably worth noting I think it was around October 2021 that one of the people I follow on Twitter just made a random mention that uh having trouble with long forming long-term memories is
can be a symptom of cptsd up until that point I had kind of assumed it's because I was ADHD and just didn't pay close attention to things um but you know uh it was now my mind was open to the fact that there is another possibility right um so I started trying to figure out grapple with the idea that maybe I did have cptsd and what did that mean uh some addition to vivisect analysis to make it easier to do raw PPC firmware analysis with the tool and then February 2022 the emulator was moved to GitHub and released for the teams to use it wasn't complete yet but it was complete enough for people to start using it and
manipulating it and playing around with it um I should probably I didn't forget to mention these slides will be up on my web page that was at the very beginning on Union shark.com um but they're not there yet they will be um so also in early February I finally accepted myself you know that yes I probably was trans came out to my partner uh this is one of those things that when looking back on this particular timeline of how the what the work that was done in the emulator was realized that it was very uh there's some weird time coincidences here right in this particular case this is one of them so you'll notice from here we're kind of
getting a little bit less populated with technical stuff honestly it's because most of the hard technical things were done earlier on but also because uh we're running out of time this talk is very much in danger going on too long so 22 March February I came out to my partner that's the picture of myself that I took the day after I came out um it was really weird like a switch flipped and all of a sudden I looked at myself and I was like maybe this isn't the worst thing I've ever seen in the world before that it definitely definitely was so then March I had the very first week of March I had a training to teach at
the company I work for grim and I had to decide wow I wanted to be did I want to be me did I want to be the you know person that people had been seen for three plus years at this particular point I very much knew how my brain works and that if I didn't come out immediately it would never feel like the right time um so I was kind of felt like maybe I should do it for that and also because I was happy I was happy for like the first time in my life um and I wanted to share the reason for it I knew there were some people I follow on Twitter who came out publicly and I
got a lot of inspiration from them so I thought maybe if I can be more public that it might be inspiring to other people also my first work trip after coming out was at the end of March which was for the amp program where he went to CSU and we're testing a bunch of tools and it laid right on top of March 31st trans day of visibility so that was kind of also a very symbolic thing where it was like well I hadn't actually worn like a skirt you know professionally at that point that was very much a let's just [ __ ] do this thing um I was extremely nervous but you know I'd already come out professionally came
out LinkedIn Twitter to my company so you know all the people I've been working with for you know at that point about one and a half years you know I came out to them and it was incredible I've actually got a smile on my face as you can see here it was a good day uh next thing uh at my I had a talk at escar so again a year after that one professional headshot I had another one that had to be taken um and uh so my partner helped me with makeup I'm stil
Related talks
51:51
32:48
33:48
54:33
31:06
20:51