Ground Truth

BSides Las Vegas · 20238:26:09400 viewsPublished 2023-08Watch on YouTube ↗

Show transcript [en]

♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ 11 ♪ 11 11, 11, the news, I'm a great ♪♪ ♪♪ ♪♪ ♪♪ Baby, you give me ice. You're giving me wind. You're some kind of bird. Don't leave me here lying dry. Oh, I want a jinx, just three.

All right

All right, good morning everyone. Welcome to B-Sides Las Vegas. This talk is going to be about cyber risk. How does cyber events become so costly? Wendy Honely is going to be our speaker today. Before we begin though, I'd like to thank our sponsors for this event. They're the ones that make this possible. Adobe is going to be our diamond sponsor and our gold sponsors are Prisma Cloud, Sem Group, and Blue Cat. It's their support that allows us to put on these events today. And for cell phones, if you have a cell phone, Please put it on silence and for respect towards the speaker, if we can minimize any type of interaction or conversations. If you

have questions at the end or if there's time for questions, there will be a period for that. I'll be walking around with a mic, so if you just want to raise your hand, I'll come to you and then we'll just use the microphone and then you can ask questions to the speaker. And that's all, so without further ado, Yeah, I'm going to shut this off. Good morning. Well, thank you for being here. I'm excited for this talk just because we had a lot of people really didn't understand or wanting to know what it looked like when it has a cyber event. So I'm still testing my stuff. So I'll start the agenda today. I started with some trends and statistics. And then I go

through some of the major category of cyber events. and talk about each one of those type of events, what are the cost components of those events. And I added the last bullet because I don't want to tell you how bad it is without telling you how you make it better, sort of. So that was the last thing that I added. So there's a few slides on that one to give you some information of things that you minimally have to have. Otherwise, they won't even sell you cyber insurance. So, intro. I'm from Marsh McLennan. We are the biggest insurance broker in the world. 70% of the global 2,000 actually buy their insurance through us. So you think of all the big names as well as a lot of

the smaller. We also have to cover smaller companies as well, and as well as reinsurers. So we also have a company that also does reinsurers for the different insurance portfolio, and we consult on that one as well. So we get a lot of different sort of data in a sense. So speaking of data, Here's some of the data source I'm going to use for today's presentation. SideWave, it used to be a vison. There's a lot of speed data that's in there. There's Flashpoint, which used to be a risk-based security. They have a vulnerability database and a bridge database. incident database that they have. Also publicly reported, so any of the data that such as from financial companies, from 10-Ks, from their press release,

usually when the breach is big enough, they have to report it. So that's where we got some of those numbers from. So those are the public data source I'll be using and the private data source, proprietary data source would be the claims. So the Marsh McLennan claims as well as we have UKs and Europeans and as well as the different insurance portfolio claims. So the reinsurers when they have different insurance portfolios, when those portfolios make claims, it actually gets into our database. Most of the data I use is ranging from 2017 to 2023 except some of the privacy claim stuff that's go further back just because those claims takes a long time to settle. If

you look at some of the big claims last few years, many of those are still open just because since 2019 there are still claims from 2019 that's open. There are still many claims that still on the privacy claims is still open. So I'll go through the trends and statistics and this one it's from the 2017 to 2023. As you can see my data Both actually. The data is a little bit biased toward the US. However, it does have the incident rate in general in the US is also higher. So there's two kind of biases. So this is about 84,000 incidents from 2017 to 2023. And the 2022 and 2023 is partial data just because there's discovery delays, there's reporting delays, there's storage

to the database delays. So if you look at it, there is delays on those. If you look at just the US, since this is the one that representing a lot of the data, you can see that increasing from 2019 to 2020 when COVID happened, that really jumped up in terms of frequencies. And then 2021, it comes down a bit. And then 2022, it's not all done yet. I think there are still things that are being reported, but it hadn't quite made it to the database. I think the average dwell time when the bad actors in the environment today, according to Mandien, it's 16 days. But then, you know, if you look at the Marriott incident, it was it got into the environment in 2014 and they didn't

find it until 2018 so it could be a long time before you actually discover it and then as some of those things it will see those kind of stuff it will happen so 2022 and 2023 and if you look at evolution of time you can see that for example healthcare on the bottom here it's pretty much a big banned a big percentage of the events along with financials and public administrations. However, if you look at manufacturer, which is this guy, you can see that at first there was not much of it and then until 2020, it's starting to getting more and more and now the ransomware event is hitting that as well. So you can

see the different industry evolutions of those different industries through time and see how they change. So now I'm gonna go through the privacy breach event. The lawyer took out all my names. So I can only put country on there But you can tell this is all public data This is nothing private data This is not proprietary data So I wasn't going to show that but I had company names here but they said not pick it up just because some of those are our clients So got yanked off But if you can actually go look up if you look up those kind of numbers that the millions and the dates you'll find them But you can see

some of those privacy breaches very huge. You probably knew what this one is, the first one. It's almost a billion dollar in fine just on that one, easily. And all 50 some states all reach out for the state attorney general all came and asked for fines and penalties on that one. FTC fines, long list of fines that hit that damage to be a couple billion dollars of damages. Yes? By privacy do you mean PII data? PII, PHI, and PCI. Okay. So PII would be the names in Social Security, those type of data. And PHI would be healthcare records, those type of data. And PCI would be the credit cards and any source of financial information, account numbers and so forth,

those kind of data. That's what I call privacy. Good call out. Thank you. So, yep, but I can get in way with the fines. Now, those are the different fines that was due to the privacy. Well, actually, settlement fines and penalties. And so those are the huge one that you can see. various industries actually are represented here and when they have fines those are not small fines. Some of those are GDPR fines but people under the impression saying that this could be actually this when I do the model I built model for Marsh and for the last six years and this is one of the big components actually comes into the get into the very big severe penalties of fines, so this

is the fines and penalties for the different companies and If you talk about GDPR fines that somebody say oh, we're not doing business in Europe But if a European citizen come to the US for your company did a haircut and you lost their data you could subject to GDPR fines and and that's 4% of your revenue or $20 million. So, and then if you look at the GDPR find itself, majority of the find in there is not due to a cyber incident, but due to process compliance, how you keep your data and stuff. And so, Anybody could be hit for that and similarly I think we you know, Kathy you had a conversation about you know all the different states

if different states California citizen go to Texas to do buy something and you lost their credit card number or something you can also subject to that state's fine as well. So this is a big category. So Just the model that I, when we built Privacy Breach, here's a typical claim expense that comes within that model. PR and crisis management, the bridge council. So just because you didn't lose the data, but if anybody just touch your data, you're gonna need to have to go have a conversation with the lawyer to see what you need to do to be compliant. So that's the bridge council, general legal council, PR, and some of the big ones, if you lost a lot of them, then

you probably have to set up a call center just because people are gonna ask questions. Investigation Forensics notifications and you see legal right there and there's various identity theft protection services that could become a pretty big bill and of course the data restoration that one is a tough one that one also could be assault you have to need software to rebuild the system You have engineering services, consultants and so forth. And then not to mention regulatory fines, legal causes and liabilities and settlements and stuff like that. So I think on a previous slide, if you look at the fine, some of this stuff, it's pretty huge. And I think there's some of them that's not even finished yet, like this one.

It's $80 million fine and $190 settlements and there are still additional settlements coming along as well. In addition, for technology companies, if you provide any sort of technology services as well as products that could comprise of appliances, equipment and stuff like that, if you lost people's information, you can also subject to tech E&O errors and omission type of insurance cut. So that could turn out to be very big. So those are usually third party consequential losses as well. So I'll talk a little bit more about that one on the next. So everybody says, well, you know, all this stuff. What is cyber insurance actually is? So this is my summary slide. So any first party stuff, anything that you investigate the cyber extortions,

ransomware, business interruptions, a lot of people doesn't even know their policy actually have business interruption coverage. And then as well as any sorts of restoration of data, response, legal, all of those are as part of the insurance. And usually, I think I forgot which words I got the source. It was like when a company actually go buy cyber insurance, the chances that they get hacked is less. And when they get hacked, the severity is also less. because the insurance company actually does require them to have a lot of those planned ahead of time before they will insure them. So, which is why the severity and frequency tend to be a bit less. And then any sort

of third party liabilities, lawsuits and stuff like that, privacy liabilities and network security liability regulatory, some of those depend on the states and countries. Some of those will not pay for the fines. Other countries will pay for the fines. Could be covered by insurance. But that depends on your policy. But what does not cover is any source of intangible assets. Copyrights, trade secrets, any of those customer lists, those will not be covered. So if you lost it, you lost it. That's not going to be insurable. So there is... overlaps between cyber insurance liability that arise due to the insurers operating risk and then there's the tech you know any sorts of liability because your actual product

if you they could say you know you could have done better your stuff embedded the issues in here that caused us damages. If you lost my customer, like if you're a cloud service provider, if you lost somebody's customer names and stuff, a list, customer list, you could be subject to tech E&O as well. So here's one of those graphs that I did for building the model. Different industry, as you can see, frequencies also different, but as the revenue increases, the number of incidents also increases along the way. So we can see that the larger company tend to, it makes sense. and that's, you know, larger company has tend to have more events than the smaller guys. Also too, there might be

something about it, it's that maybe some of the small guy didn't even report it. So that could be also part of that. So this is a per record cost for large privacy breach. You can see the range anywhere from less than a dollar, 36 cents or 20 cents if you were to count the high end of the range on Yahoo to over $500 per record. And those are still pretty big numbers. And some of those are still open. And so this is like the number of record they lost. And then this is what they disclosed. It's publicly disclosed only, which means there's a whole chunk of it. It's probably still never talked about, never reported. So that's not in this number. So per

record cost would be higher actually, most likely. Well, it is higher. It will be higher because a lot of those they don't disclose and some of those number we got it off of their press release. Some of those number we got off their 10 case. Some of those comes off their annual financial data and stuff like that. So and we also when we looking at those things, sometimes they kind of iffy about disclosing those. So like one of the company got ransom, they said, "Okay, we had hacked by ransom. We pay ransom. However, along with the flood in Texas, the whole loss is $140 million." So they don't tell you exactly what they lost. So why did

data breach become so costly? For the smaller breaches, forensic and breach response. When you have a breach, that's what you have to have. And then for the larger one, the cost coming from the vendor expense, the legals and the fines and penalties and settlements and liabilities and so forth. Those were the buckets. So when we model it, we model for smaller companies and and there are certain frequency for small breaches and there are certain frequency for large breaches. When large breaches happen, then a lot of those fines and penalties and settlement get kicks in and that's how we built the model, say, this is what your risks look like. I want to also next thing to move on to talk about business

interruption. So this was not in too many people's radar until I think 2016, there was a botnet that basically shut down the East Coast of the internet is this Mara botnet and then the not Petya happened that's 2017 and then that's when everybody said we need to buy BI insurance business interruption insurance today it's actually going to become bigger and worse than it would be because a lot of our companies they're so integrated in terms of supply chains So remember COVID when the chip shortage happened and they can't produce as many cars, they can't deliver cars? Well, this is what happened when you have business disruptions. There are liabilities and oftentimes a lot of those events doesn't

even get reported just because they're small and they just won't tell anybody about it. So here's not Petja that names got erased, become country. But you know who they are exactly. So the worst one, it's $1.4 billion worth of damages on that Petya. And then that one actually not only hit, they maxed out their cyber policies and they also went to the property policies. So then there's a discussion of that's act of war. And if it's a war, we don't cover it. And then so there's lawsuits and lawsuits. Eventually they won and say that is part of not war, that is part of the property stuff. So certain part of that did get covered. And some of the other large business interruption loss You

can see those are pretty recent except I think I don't have any 2022 there. The 2022 one didn't make the list, but some of those actually made a list. However, they blended themselves with other losses, so I couldn't put it on the list. So yeah, so this is some of those larger ones. But all of this is public. You can actually go search for it. That's the company, that's the countries that they're from. And you'll find it if you search for specific numbers on the loss damages and the first timeline, you can probably find the company if you want it to really know. But this is just to give you an idea what kind of impact a business interruption loss could have. It's huge. So

the way we compute some of the different losses in the business disruption is that you have you shut down time, you have time that you actually down for a period of time and you slowly recover and then there's a period of restoration. Now this back end here could be really long. We had customers, we have clients that actually after two years they're still suffering losses. because their process, their blueprint got lost. So now instead of building a factory, building some sort of power plant by module, they have to recreate all those modules. And some of those were pharmaceuticals. The tests that they have, the data that they test, they have to be certain FDA compliance. While

those data is lost, Start all over again So those could be really damages And by the way, the revenue itself is not insurable but the income loss is insurable So any of those things like, you know, if you pay salary during those times, you pay rent for the buildings, those kind of things during those times, it is insurable. So, you know, a lot of people doesn't really, some of the company doesn't understand that it's valuable as well, because anytime when you have a big ransomware, a big BI event, this is actually a big component that could be covered by insurance. I look at by industry over there and the biggest one was education and public administration, healthcare, and some finance

and insurance, and then CMT, communication, media, technology, those are the kind of frequencies of where by industry last few years that's what they're being hit the most. So why do business disruption events so expensive? Well, you have revenue loss, you can have unfulfilled orders, you have lost orders, you could have a long tail recovery time due to physical or intellectual property damages. You could have just higher cost of production. We have a lot of clients that just because they had something happen to them, now their production is a lot higher. There's contingent BI, so BI that other are depending on you to run their business or build their products that you could be liable for. And of course, it's legal and liabilities. Those are

the kind of things that would be. and I think this is one that everybody's been asking about lately is ransomware. So I'll go through ransomware. So the data set I have ransomware is about 11,200 ransomware event from 2017 to 2023. I picked 2017 and Actually 2017 to 2018 was not so bad because a lot of those, actually I did a time series analysis on that one. There's clearly a break point between 2018 and 2019. And if you run the time series, right there, Very clear break point on that I only counted the event that's so intense to extract ransoms I didn't count the NotPetya because NotPetya is not a ransom extracting event They said $300 even

you pay, they're destroying your infrastructure So they're not wanting to collect the $300 and they don't have key to fix it So I didn't count that So that's where that is And here are some of the large ransomware losses. And it's a few hundred millions, and I think there's some, it's all in the millions. I think the last one that made it to the list is about $15 million. But still, that is a pretty big damages in terms of ransomware. Again, those are all publics. So in terms of ransomware, here's a list of things You got to pay the ransom or negotiate the ransom The average ransom negotiation time, that's about five days So it takes you about five days to actually just on average to get negotiated if

you decide that you want to pay or not pay And then you have to have OFAC certification So all the sanctioned country, if the ransom organizations from sanctioned country, you can't even pay the ransom. And if you do, then you get in trouble with the US government. On the bad ransomware event, business interruption would be a big one. So if you don't have your backups and stuff like that, that could be a huge cost as well. And if any of the record was held for ransom, then there could be Christ management type and stuff and of course investigations and as well as privacy related product and that's all. in the list in terms of ransoms. And then of course then you have your regulatory stuff if

you lost a lot of records and that could be pretty large settlements there. And then I put the extra expense. This was actually from one of our claims. Here's a list of all the extra expense that goes with it. And temporary worker, temporary data center, temporary cloud services, any sorts of incremental to financial statements, audit fees, incremental internal labor costs, employee expense, pizza, for the Friday night. You make it well, goodwill stuff. And I think that's a big one in Japan. There's apology fees that you have to pay as well too. So I'm told I better go faster. So here's the frequency of ransomware. You can see that 2017, 2018 is pretty much none. And then 2019 is starting to go up. And then

when COVID happened, we exponentially gone up. Russian-Ukraine war sanction, it went down. And because they couldn't get hardware to get the money. And here's the latest most active ransomware group that we have on our list. So I thought I'd put it up there. That was the last thing as well. 2022 and 2023 is also, 2022 is also data is still partial as well. So yeah, you can see it's coming back up again after the sanction. So they figure out the way to get hardware to actually do the ransom activity They figure out to get the money because for a while because of sanction they couldn't get the money And then look at the different industries At

first it was the... you can see healthcare initially really got hit a lot And then the next one is manufacturer, which is the orange one right here. At first they weren't getting hit that much, but look at what happened to it now. It's a huge percentage of manufacturer. And then it goes up to educations. The orange guy is professional services. So the law firms, they weren't getting hit much. Now they're getting all hit different type of professional services type of organizations are now getting hit more and more so. So this is one of the guys is like, "Wow, what do I do?" So this is like the percentage of the company who pay versus the percentage

of not paying. So as you can see from 2017 on, there were a lot more that pay because people weren't as ready as it was prepared. So they might not have backup, they may not have the right stuff they were supposed to have. And now as it comes up to 2021 and 2022, we are seeing a smaller percentage of companies are paying as well. And here's the bridge response costs. I put this in percentile just because I didn't want to actually tell you how much they pay because there are clients. The average and so the blue chart is actually a log normal fit to the distribution of bridge response. In terms of the green one, the median and average is

actually actual number. That's why there are green bars there. But if you look at that, the average there, it's in what? 981K in terms of response cost. It's above the 80 percentile. So that means that there's some company or companies out there that spend a lot of money on breach response costs in terms of that. And here's the known total cost versus the average total cost incurred. So you can see that there are a number of events out of those that has The little thing, this thing right here, it's actually number of events. But then each one of those bar is the total cost of the average cost in that. So 2017 was pretty high. 2018, it's not too bad. It's gone down. But 2019 and 2022,

it did get a lot higher. And then this is the known total cost per year that I could find of all the known total costs. This is a chart of average ransomware demand and payment for our clients. So from 2019, in the beginning of time, they don't know what they're doing, so they just demand anything and everything. And look at the pay ranges, it's a lot smaller. So we had companies that got like, they were only like, you know, 5 million, I think it's 8 million was their revenues and the ransom demand was 80 million. It's like we can't pay that, we don't even have anything to pay that. It's like just not happening. So you can see beginning

of it, it's really wide range because they don't know what to ask, they don't care, just ask something. But as it goes to 2020, you can see the range got a lot smaller. See, this is the demand range. This is the pay range of that. And so it did get a lot smaller. And by the way, the largest ransom demand in 2023 so far is $175 million. and the average pay the actual not that one but largest pay is 30 million this year so it's it's decent numbers a huge number um the most active ransomware group is lock bit so yeah and here's the ransom pay by um you know by year and see the different percentiles and stuff

like that as well. So this was our clients and you can see that 2021 had some of the bigger ransom amount that was pay. So yeah. So how is the cost distributed? So I took about 40 something 50 somewhere on there of the ransomware claims that has all the detailed costs and I was able to group and the one that didn't have detailed costs got tossed out so when I separated them. Here's what the distribution of that and you can look at it, ransom payment took almost about 30% and the business disruption takes about 30% and this is all from various industries didn't have and the liability on this one is small just because it's a very,

I think there was only like two maybe that had liabilities in there. So it doesn't really represent what the true cause is and there's a claim preparations type of fee for when you get events, you gotta pay for claims. You gotta prepare a claim for the insurance, so there's the fee that consulting does charge. So that's what the restorations and distribution, so this is one of the things that people ask a lot, is like, if I had a ransomware event, what do I do? How much is it gonna cost me? This is sort of an average thing. And we talked about moving, so I added this slide. So, the move happened in June. So July, we started getting hit by claims.

So up to date, we had about 117 claims as of yesterday. That's how many it's happening. It's happened mostly to education, financial, healthcare, and communication. So yeah, that's what happened lately. So how did they become so costly so far? Well, of course you saw the cyber privacy business disruption ransomware. And I also want to call out technology errors and omissions. If you provide any sorts of services, technology services as well as products, that could happen as well. Once they got in your environment, they could fraudulently reroute your funds and it could hit your crime policy. And then... we have ransom that hit multiple policies, hit the cyber policy as well as the kidnap ransom, kidnapping ransom policy as well too. So

when one policy ran out, depends on the wording, they could be paying for the other policies. And then we certainly seen it hit the property policies as well too. So there's multiple damages across basically your whole enterprise on the different front and that could get very expensive. So this is the part that I didn't put it on my original agenda, but I thought it would be fair to actually talk about how do you improve it with all this stuff. So I want to talk specifically some of the top controls that you have to have. So the data that I used was firmographic data from Dun & Bradstreet, any of the historical incident data that we talked about from RBS, Advisons,

and Marsh-McClendon claims, insurance claims, including the insurance portfolio claims, and then also any of those technographic data that we have inside and outside assessments. as well as the scoring of this side and security scorecard. Those are the outside looking in and see the different scores that you have. And so we've gone through the probability of success of the cyber events and look at industry-specific implementation of those controls. We also, let me see if it's on the next one. Yeah. So this is like when they come and get insurance, we make them fill 150 or so questionnaire covering all this broad spectrum. We call it cyber self-assessment. So it's sort of a, we make them go through all this stuff in terms

of governance, account monitoring, business continuity, recovery, any sorts of stuff we ask. So this is just our, as a broker, we ask them to make up a few questions. And what we do is we correlate those data and we correlate it with the firmographics as well. So there's firmographic data in terms of revenue, high versus low, high is anything billion and above, low is anything less than a billion. And as you can see, the company's firmographic matters. So if you, the conditional probability of a claim in the low is about 3%, 2.97, and the condition probability for a high revenue company is 8%. So it's more than double what it is. So we also do that

as well. And then we've gone through from the CSA that they had, the cyber self-assessment, we compute the signal strength of those versus the claim data that we have in-house. So what we're saying is that secure configuration came out on top. You've got to have configuration management tools such as Active Directory and so forth. You've got to monitor your account. Those are some of the top controls that you have to have in order to reduce the frequency of your claim. Well, frequency of event, therefore claims. So, some of this stuff is because there's so many things that you have to look out for. So what is the priority? So this is what we're saying that when we

come to of those 150 questions that we have, this is what we came up would be the top things that you have to have. And then in some of those cases, individually, some of those questions like this one, about multi-factor authentication. If you just do one, it's 1.25 in terms of signal strength, but if you add the other two, then you actually get to a 1.44. So the completeness of those implementations matters as well. So it could be a huge impact. We grouped the questions by control categories and if that control category for multi-factor authentication is well implemented, then the likelihood of good signal strength increase quite a lot.

And then we looked at the incident rates along the various companies and see how they are in terms of the implementations and education sort of came out low on the list and then they are of all the top controls and they also have significantly higher claim rates compared to others as well. So failure to implementation top control is reflected in the industry incident rates. And here's the top five things without those positive, those are the top five you have to have without it. They don't want to sell you cyber insurance and or they can sell you cyber insurance with extremely high price. So that's the So this is based on the study, actual data, fact data that

we check from our customer answering. And we have a few thousand of those per year of those answers at least. And then back to the many years as well as back to many years of, I think 10 years of claims. And that's what we came back to be to say, this is the stuff you must have in order to reduce your frequency of your cyber events. So the key takeaway is that use the self-assessment data to allow you to figure out what are the most impactful control, which is what I listed. And if you brought in robust adoption of some of those controls, it's necessary for their effectiveness. Multi-factor authentication was the example that I used. And then industry with lower implementation

rate has high impact control, tend to have higher cyber event rates. And here's where you can get more information about the priorities, as well as my email address on the bottom if you want a copy of the presentation. That's it. Sorry, we are out of time, so if you have any questions, if you would like to pull it aside and... Right at 45. Sorry. Yeah, I'd be happy to answer questions on the outside.

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

Welcome to B-Sides Las Vegas. This is the, uh, track is it? Yeah, Ground Truth track. Today we have Sonil Yu, and he's going to be giving a presentation, Double Entry Accounting Systems for Security. Just a couple quick announcements before we begin. I would like to thank our sponsors, our diamond sponsor, Adobe, and some of our gold sponsors, Prisma Cloud, Semgrep, and Blue Cat. For cell phones, if you guys could please put them in a silent mode for the duration of the presentation. And at the end, if you have any questions, if we have time, I will be walking around with a hand mic. So just raise your hand kindly. And if we don't and we do

run out of time, you can just pull the speaker off to the side and ask questions there. So without further ado, Mr. Yu. Thanks very much. I'm going to hold this. Can I hold this? No. All right. Maybe not. I didn't realize Palo Alto went by Prisma Cloud now. That's their name. Anyway, okay. So thanks for coming. I'm Sunil Yu and I don't know if you had a chance to see the keynote, but I did make a quick reference to this in the keynote. It's all around how do we start applying certain practices that we know in accounting towards security, right? And so let me, if you don't know what double entry accounting is, I

want to give a quick definition It's a summarized version of what you can just find in Wikipedia. Some history, back in 1458, the Medici's, I think, or anyway, some Italian guys figured out how to do double entry accounting. Before that, it was single entry accounting. Makes sense, right? And the basic idea is you have these two ledgers. So you have a cash ledger and you have a, let's say, a different ledger. And they are supposed to balance out. Okay, but it's two separate ledgers that help provide a certain set of benefits. And so I'm going to read those benefits. and you can see where it might apply. So it offers an accurate and reliable comprehensive view of our transactions. It minimizes errors, basically leading to more

trustworthy reporting and facilitates easier compliance and auditing. Sound familiar? It sounds like something we want in security? I hope so, right? So these fundamental principles that we look for in security, well, gee, in accounting, we have something like that already in place. So the question is, where do we have the opportunity to see double entry accounting type of functions happening in security? You just don't recognize it for what it is. But once you recognize it, you're like, ah, this is it. How do we make more of those type of systems? How do we ensure those systems can be reinforced and so on and so forth? So now as I talk about this, I have to

set some foundational principles. That's how I think. I think in principles because if you disagree with what I'm saying in my conclusions of the principles, then you can disagree with me on those. But I would rather go to first principles and say, do you disagree with the first principles? That's why I share frameworks all the time. So, one of the things that we oftentimes hear from many vendors is, "Hey, this is your single source of truth." Okay, and that's a myth. I think we all recognize that to be a myth because the reality is that there isn't a single product that tells us the whole truth, right? And I think we intuitively know that, but

let me dissect those words for a moment. What do we mean by whole truth? Okay. Now you may have heard the word whole truth before in a slightly separate context, which is in sworn testimony. So the question is, what is the truth, the whole truth, and nothing but the truth? And can we actually handle that? So what is the truth? And then the whole truth and nothing but the truth. Well, let's talk to those because those are-- you may wonder, like, why do we repeat those three things in sworn testimony? They must be different because otherwise it seems kind of redundant, right? Okay, so what's the truth? Well, it's pretty easy, right? Truth is just don't lie. Don't alter anything in response to a question. So an example in

security will be, or just in IT in general is, "Hey, tell me the encryption status of this widget here." And if I get an answer back that says it's disabled, if I ask a witness and the witness is some tool and the tool says, hey, it's off, then I need some sort of cross-examination, perhaps with a different system or something else that says, no, no, it's actually turned on. Maybe I go to the actual system or something like that. So this is an example of where I'm not telling the truth. And it's easy to verify. You just go and look for it and voila. Right? Then you have the whole truth. So the truth, the whole truth. And the whole truth is about not omitting key information.

And this is actually a lot harder. How do you know you're missing the whole truth? So I can ask a question. Hey, do I have any security groups that allow inbound access? I can have a rule that says, nope, I have no security groups. So I may have an alert asking for the whole truth. I'm expecting the whole truth. And the alert is to say, hey, do I see this? Another example beyond what you see here is what you see with a lot of MITRE ATT&CK claims. Hey, we see this TTP. Well-- you see a very specific form of a TTP, but do you see all aspects of the TTP? And of course the answer is no, right? And that's an example again of a place where you have

missing whole truth. And so how you understand the actual truth here is really hard. And I'm going to talk about this in greater detail. But the whole idea of the whole truth is, well, I asked about security groups and my answer was no, I don't have any Azure security groups, but well, I might have some AWS ones, right? So I'm not getting the whole truth here. And that's one of the problems that we have in our security tooling. So I'm going to come back to that in a moment. But the last form of sworn testimony, or the last aspect of sworn testimony is nothing but the truth. I was trying to think of some good

examples for this one. And I like making fun of the security vendor industry. So the one I figured was, OK, hey, how secure are you vendor, security vendor in particular? And the answer is, hey, look at all the certifications that we have and all these awards that we won. And somehow that's supposed to tell me that they're secure or that they're a great vendor. And upon cross-examination, we realized they failed their ISO 27001 and they just basically paid for those awards. Oh, by the way, I should mention, not all those awards are truly play to play, but I didn't spend time to figure out which ones were which. But the point is that "Nothing But

the Truth" is an example of something like this. All right. So, okay. So you understand now what is the truth, the whole truth, nothing but the truth. And when it comes to, as I mentioned, the truth, it's easy to validate. You have the truth, or you can check against the truth and you know it's false, you know it's not right. You can also validate and cross-examine this. But the problem with the whole truth is you don't really have a means to understand the full scope of what is the whole truth. Okay? And so this is a hard problem. And I thought about this and I said, huh. Could double entry accounting help us understand the existence of the absence of whole truth? The existence of the absence

of whole truth. It's a mangled phrase, but bear with me. What I want to know is, am I missing the whole truth? I may not know what the whole truth is, but I at least want to know that I'm missing the whole truth. How would you know you're missing the whole truth? You wouldn't know because you don't have a means by which you do that. But double entry accounting allows us to do that. We would say, hey, I have this one ledger that says I just incremented something by $1,000, but this other ledger, there's a mismatch here. I'm missing something. I don't know what that is, but now I investigate. All right, so let's talk about that and see what that looks like. So, again, cyber defense

matrix for those who don't know, something I created when I was at Bank of America, blah, blah, blah. Okay. But the more interesting aspect of it is that I can split the, I can look at this as a ledger, as two types of ledgers. Okay. One side of the ledger is on the left of Boom, and that's what I call structural awareness. Tell me the state of the environment. This is all before anything happens. This is like your inventories. This is your CMDBs. If you have a vulnerability listing, that's where this is. This is your tax purposes. This is everything on the state side of the equation. This is the state side of the ledger.

Then you have the other side, the right of boom side. And this is what we call situational awareness, or what I'm going to call situational awareness. This is after something bad has happened. These are your events. These are your logs, all those sort of things. These are where your compromised systems live and so on so on and so you have ledger one you have ledger two and they're two Completely separate systems, which is what you want for a double entry accounting system You don't want them to be if they were the same system, then it's essentially a single entry accounting system But these are two separate types of systems and what's really kind of cool

is that they provide quality control for each other Okay, so let me explain how that works. So So let's take-- I'll explain this, a ledger one system, a structural awareness system. This is, again, like a CMDB or something that gives you, again, what I call structural awareness, your state of your environment. So let's say you had some widget, some switch, and at T1, it's on. And at some point, when you pull again, it's off. So you're maintaining state. First, you know that switch A exists. And now you know the state of the switch. And at some point between T1 and T2, something happened in this polling cycle. And so now the natural question is, other ledger, did you see that activity?

Did you log that? Because now, if I did, I'm like, OK, good. I have a log that observes that activity. However, if the answer is no, then I know I'm missing something. Okay, now the answer may be, there's many different reasons why the answer may be no, but now you have a choice. Do I want to spend time and effort to investigate this side of the ledger that is missing the whole truth? Okay, and the answer may be, I don't care. And by the way, in accounting, that's oftentimes the answer too. Like, you know what, this is a, What's the kind of fund that's like really small dollar amounts? Petty cash, petty cash. It's a petty cash fund. And you know

what? Do I need to have a precise accounting of every penny that's spent there? There's a reason why it's called petty cash. And so in that sort of sense, I may not choose to investigate because it's petty cash. But in other cases, that's equivalent of saying, you know what? I don't know what's happening in my guest Wi-Fi network, but I don't really care. But if it's on my DMZ, I care a lot. And so this whole notion of knowing when you have a state change, you can reconstruct events from state changes. Not really well, but nonetheless, there's a mechanism for that. And conversely, on the other end, so you have somebody switching, flipping a switch, and you do the same thing. You say, basically, okay, I see that switch

B is off, or I did something to flip switch B, state system, other ledger, do you know about switch B? Is that even in your inventory? And if the answer is no, again, it's what I'm calling visibility here. I'm missing visibility. That means I may choose to go get additional sensors to see that switch B, But again, I may also say I don't care. But nonetheless, I have a perspective of the existence of the whole truth. I may not know, I still don't know the whole truth, but I know that I'm missing the whole truth. And herein you have this ledger system, which we already have these kind of tools in place, but we haven't figured out how to balance the ledgers together. And so that's the idea, or

at least not more than the idea. That's actually what I'm doing. And I'm starting to see vendors actually bringing these two systems together. And it's actually pretty cool, because now you're bringing two different systems. But at the same time, you still want them to operate somewhat independently because you need those sort of checks and controls. And ultimately, the goal here is-- so what I've seen with a lot of attempts here is those who try to take event-based systems and reconstruct state from there. And if anyone's tried to do that, it's hard and it's painful and it's highly inefficient. So any reconstruction of state to events and from event to state is going to be incomplete and less efficient because you're the way that it works is you have

to like if I'm going to recreate events from stay well one of the problems is if I don't have like polling cycle I'm gonna miss the event right so am I gonna increase the polling cycle so that's every one second no it's just it's not gonna be it's not reasonable for you to do that so it will be incomplete you're not gonna have every instance of every state change and But nonetheless, with a sufficient number, with a decent polling cycle and over time, you will see those state changes and you'll say, okay, am I missing the whole truth? Again, the goal here is not to have the whole truth all the time. It's just to

know when you're missing the whole truth. And then likewise, with events to state, well, if there's no state change, then you never know about the resource itself. And so... Again, that's why it's going to be incomplete. But also, any sort of event-based system struggles in maintaining state. And so that's the other challenge. If you've ever tried to do this, it's hard. So my advice to you is, if you want to create this subordinary accounting system, which allows you to basically know when you're missing the whole truth, use it for the purposes of that balancing equation, but not to replace the other side. OK? Now, there are a number of implications that come as a result of this. First is this perspective of what is-- so I mentioned at the

quick keynote, some of these things that we're trying to do may put additional burden on us, actually. Because we're now moving into a space where there are standardized practices that look like accounting. If you're moving at that sort of space, then there's an expectation that you're practicing those generally accepted practices as well. So now the question is like, well, you know what? I don't want to set a bar that's higher than what's in accounting. Think about accounting. How long has it been around? You saw the double entry accounting thing. This was 1458. And of course, accounting existed before then too. It's a very mature practice. People care about it a lot because it's all about

money, right? And yet, if you ask a CFO and ask them, "Hey, you know what? How precise is your accounting? What level of variance are you comfortable with?" And they'll tell you, "Oh, you know, it's okay. As long as we docked the boat within 1%, we're good." Okay? where 1% would be great in cybersecurity. I think we're oftentimes asked for like 0%, okay? Like no mistakes, no anything, right? Like how realistic is that for us if finance, which is a much more mature practice with many more tools and many more processes can't even get to zero themselves. So what sort of, why are we setting a bar that's higher than them for ourselves or for external regulators? And

then, now for many organizations, they actually still practice single entry accounting when they're small. And they don't need to, there's like a whole cash accrual system. And there's a bunch of things that, I'm not in finance or accounting. Those who've ever studied it, I'm sure you know what I'm talking about. But there is a point where you switch. You say, you know what, we need to have more rigor here. I'm not suggesting double entry accounting is perfect for everyone. In fact, double entry accounting in accounting is not perfect for every organization either. At the right time, you switch and you say, "Okay, we need more rigor here." there's an expectation of one rigor. When is

that for cybersecurity as well? Because we can do this. We can actually do double entry accounting, but it's going to take a little effort. Well, partly because sometimes you don't even have a CMDB. You don't have a SIM. You don't have something that captures that. You don't have the other side of the ledger. But once you do, now you have the ability to now do this double entry accounting. Okay. And then I just leave it with this question of, do we want to have something like Sarbanes-Oxley for security? Because it comes with penalties if you don't do it right. But as I mentioned earlier, we're already getting penalized with lawsuits and SEC fines and in some cases criminal prosecution because we don't have these practices well

understood. All right. Now, this is just one example of accounting and accounting principle. You may have heard me talk about impairment and other sort of practices. There's other systems that we've seen that may seem like double entry accounting, but actually end up being more single entry accounting. Those are just where it goes back to the truth question. In other words, you say, hey, something is saying we are compliant in this way here, and I just need to verify that compliance happening. That's not double entry accounting. That's just a tell me the truth, I'm confirming and doing cross cross-examination of the truth. Pretty straightforward. But I think the whole truth again is a really hard problem because you

don't know if you ever know the whole truth. But double entry accounting helps us do that. So okay with that, questions? Go ahead. I love applying other business concepts to security. That makes perfect sense, right? Because we're not doing anything new. So taking legal concepts, hold truth, and how you respond, and is that fully the correct answer or intentionally vague, right? And using double entry accounting. The other thing, however, that other business practices use, and maybe this is in addition to your first principles list in this concept, is the notion of forecasting. So in finance, we use forecasts as another check. This is what we were expecting to occur in this period. And then we have the

actual numbers coming at the end of the month. And we look at the difference between forecast to budget, which is actually even further. So we planned in January. So now we have forecast to budget, forecast to actuals. And we can carry that forward month over month as the year progresses. Now very different, of course, than insecurity. But perhaps as a first principle, I guess my question is, are you introducing the concept of applying other methods or are you suggesting this one specifically? Oh, I'm up. So in another slide, which I didn't bring here, I actually introduced a whole bunch of other financial principles. Like I mentioned impairment. Impairment. Makes perfect sense. I actually specifically left out return on investment. Agreed. Because I... Agreed. That's the forecasting piece. But I

had it in. I took it out because I'm like, ugh. But return on investment is not just a forecast. There's so many other parts of the forecast. The forecast is license entitlements. We expected that our company would grow by 500 employees this year. So when we renegotiated our Microsoft E5, we did it to 300 new entitlements, knowing that we would pay the bigger fee for the final, when they do the true up. Right? So forecast also of what we expect to see occur. And when we see anomalous behavior once a month, we should look at these are the things we expected. Seasonality. Oh, this makes sense because it was August. Those things don't occur. These, more of them. So, we can take the rest offline, but not all the--

I would love to have every-- as many principles apply, but there were some I still struggled with to figure out what the right pattern match is. But I found enough patterns that like, "Ah, there's something really here that we can latch onto." That's how I connected it, right? Mm-hmm. Okay. And we had a question here? Yeah, one more. - Yeah, going back to your thoughts on due diligence and rigor. In cyber, most of the time we think about materiality as a purely financial construct or reputational only if it has a reasonable impact on that same financial situation. Have you given any thought to whether there's an alternate way of thinking about materiality in this context using these principles? Great,

really great question for which we as a community are struggling to find the answer. Yael is familiar with this, but I specifically asked this question to a whole bunch of CISOs and beyond the, you know, whatever CFO says in terms of materiality, 20% of revenue, whatever, whatever. Okay, I'm like, yeah, I get that. Let's let's take that one off the table. How else would you define materiality and it was We don't have a good answer. We don't have an answer at all. We absolutely need to have an answer Well the answer to some degree is this notion of variance, okay is the variance material enough to that the public needs to know about it. And this notion of

variance, of course, on the finance side, it's numbers, money, right? But I also mentioned this notion of variance in this context. Like, how variant is ledger one from ledger two? Tom? Okay. You want to just yell it? Anyway, we're out of time? Is there one after this? Yeah. Right after? No, no, no. That's like 10 minutes. All right. If you have any questions if you want to pull the speaker aside but we do have another presentation after this. So thank you for coming out. Thanks for coming. Appreciate it. Thank you. ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ Large language models. A few things before we begin this presentation. We'd like to thank our sponsors. Our diamond sponsor is Adobe and our gold sponsors Prisma Cloud, Sem Group and Blue Cat. It's because of their support that we're able to put on events like this. So we really appreciate their support. For cell phones, if you guys have cell phones, please put them on in silent and avoid using them throughout the duration of the talk, unless you take pictures and stuff. And at the end, if we have time, walk around with a mic for questions. And if not, and we do run over, you can always pull the speaker aside for questions.

But without further ado, Mr. Bobby. Thanks a lot. All right. Appreciate it. Appreciate it. Yeah, so welcome. Thanks everybody for coming out. My name is Bobby Feiler. I'm the head of data science at Sublime Security, email security startup. This is my talk on Babelfish, which is accelerating adoption of domain-specific languages for large language models. That's a lot of alliteration very early on, so we're going to just move right into why I'm actually here. I work for a company that has a query language. I see a lot of shirts in here representing companies that have query languages. They are pervasive. That's a great thing. They make life a lot easier in a lot of ways. They

really open up your platform to more customizable ability, you get your engineers committing to things much faster, everything's much more tailored. It's overall, I think, a great option and a great experience. But because they are becoming so pervasive, security workers often have to learn like five or six, depending on their full suite of tools. And that can be a little daunting, I think, particularly for new people in the detection engineering space. I think that initial onboarding period can be a little difficult as well when you really just want to pick up a tool and start using it and learn on the fly. So that's really what this open source Babelfish kind of endeavor is about. It's about reducing that

barrier of entry and trying to increase adoption rate and onboarding experience as much as possible. So how do we do that? I think before we dig in there, I just want to really drive the point home via like an anecdote, if you will. And this is one that I run across even though I'm a data scientist, I still have to write rules for my company just to make sure that my models work. On the left, you have your hypothetical phishing email. It gets forwarded to your abuse mailbox. You as a security worker see it and you're like, all right, there's some Adobe branding. There's some language around there trying to get me to click financial

information. How do you go from that step to what you see on the right-hand side, which is a sublime security rule? It's not overly complicated, but there's a lot of nomenclature, syntax, kind of verbiage that is different, certainly different than maybe Splunk or Sumo Logic or Semgrep or something like that where, you know, you have a completely different domain that you're attempting to apply. So how can we make this process, this jump from the left-hand side of the screen to the right-hand side of the screen more seamless? And I think one thing that we can do that's often lost upon, particularly the data science community, when we're like, AI will solve everything and replace people,

and it doesn't have to be like that. Instead, we should be taking advantage of the things humans do really well. And one of the things that they do really well is they apply kind of an impromptu or maybe... I can't even think of a better way to describe it-- a translation process where when they look at an alert, a piece of malware, a phishing email, they are running through a mental model built on domain expertise that they've crafted over an extended period of time. And this mental model can turn up a couple different ways, but one of the ways that we see a lot is just a checklist. This mental checklist they go through where

each question they ask and the corresponding answer is actually detection logic, if you think about it. Is this actually from Adobe? Is the language suspicious? Where did the URLs go? Were there any off failures in the headers? Each of these things are really just snippets of logic that when pieced together are a pretty effective detection recipe. So what we want to do is really key in there and use large language models to capture that process to allow them to ask the question in natural language to learn the query language faster over a period of time. And that, to me, is kind of, I think, one of the bigger impacts large language models can make in the security space near term. Long term, tons of potential throughout a variety

of domains. I think short term, there's a unique opportunity to increase usability of your product. Those first couple of touch points with a new security platform, making it as simple and intuitive as possible for you to get in there and start contributing. Increasing speed and efficiency, not only of the onboarding process, but of contributing to rules and detection logic within a platform. Reducing the likelihood of frustration or coding errors, which are commonplace in query languages. There's somebody who's guilty of that all the time. Having a large language model trained on real-world working examples will help increase the likelihood of the code being produced being correct. And then finally, one that I'm personally a fan of and works great at Hacker Summer Camp

is just improve collaboration and communication. Anybody in your security organization can talk about a threat in natural language and it would be great for them to be able to contribute to kind of the security hygiene via these platforms in a more natural way. So with that sort of background, what did we set out to do when designing Babelfish? This idea of a large language model dedicated to natural language to code translation. In order to use an LLM, you need a data set. Sublime security is growing, but we're not Python. We're not even Splunk. There are not just troves of natural language and code snippets readily available on Stack Overflow or Twitter or forums and things like that. So we had to get pretty

creative where we pulled down this initial data set. And I'm hoping by sharing this, you know, those of you in the audience that work for similar companies could kind of take this as an opportunity to maybe do the same. We went through, we used our documentation, which gave us a really good background of syntax and the way we as engineers describe the language that we're providing people. The schema, the way we break down an email and expose it via the query language has a lot of natural language descriptions in it that we could start to leverage. Likewise, open source rule repos in our Slack channel, community Slack channel, are really, really rich data sources for

not only real world snippets that are effective, but the way a diverse set of detection engineers describe their work. So everything there led us to a pretty decent sized data set. We still had some more complicated or complex like compound queries that required annotations. So for that we pulled in a group of detection engineers, internal and external to our company to like provide a natural language description, which is really cool. Certain people were very verbose and methodical in the way they asked for PDF attachments. And there were other people who were like, well, I should just be able to say, is PDF attachment? And then it just spits out the logic. And to me, that comes from experience. That comes from expectation of the product. And

these are all things that I think the large language model can potentially help out with. When it was all said and done, we ended up with a decent size, not a large data set, but a decent size, certainly enough to fine tune a model with about 3,000 examples. We uploaded that to Hugging Face so people could pull that down and start playing with it immediately. It came from a couple of different data science groups that released Ember, which is a malware classification dataset. Very important to the open source kind of ML security space. I hope that as we continue to grow this out, this can be another such dataset to help further research. So once we had our data set, it was time to think about an ideal

kind of large language model architecture. It's a 20-minute talk, so I'm not going to get into the guts of a transformer right now. There are books and videos and everything else. I'm just going to come to you as somebody who wanted to provide an open source model the fastest and cheapest way possible. So this was like my set of requirements. Knowing that I wanted to do that, a pre-trained model was by far the most important thing. Pre-trained models, think of OpenAI and Claude and Anthropic, a few others. These are built. They cost money. hundreds of thousands of dollars. They're trained on tons of data on a variety of tasks, code translation being one of the big ones. But this knowledge base and the API access, Python libraries,

large support networks where you could go and ask questions and get help, all fed into this final, like, it needed to be inexpensive because we were just going to give this thing away for free anyway. So what we ended up settling on was a fine-tuned GPT-3 model. If you're familiar with that, there are like three variants. There's DaVinci, which was used for ChatGP. It's very... It's very good. It's a little slower and it's way more expensive. And then you have Ada kind of at the opposite end, which is very inexpensive, very fast, but not very good at code translation tasks. So we found kind of our perfect bed to sleep in with Curie. It's a mid-tier. It excels at code translation tasks. It's very fast. So

when you think about integrated into a VS Code plugin, the translation step is very quick. It's also very inexpensive to run inferences against, which again, because we were giving this away, we wanted kind of the best of both worlds there. As I said earlier, API access and cost from a resource standpoint were kind of critical. You can see on the right-hand side, to train these very sophisticated models now is like 30 lines of code. And it costs a a dollar, like a dollar fifty max to train, which is an insanely good price. It took like 90 minutes maybe, and when it was all said and done, we had a model we really liked with good API access infrastructure surrounding it, and we were ready to kind of move forward.

So once we had a model, we wanted to benchmark it quick just to give the community and frankly ourselves some idea of how effective it was from a code translation accuracy standpoint. So we used Passat K, which is an old information retrieval metric. It's still very useful for things like translation. All that says is you have K number of attempts given a prompt to get the right answer. We used three because Three is about the max that you want to go anyway if you're going to provide these responses to an end user to try to evaluate and things like that. We did relatively well. I think 98% of the time within three guesses it had

the perfect response. I think after one opportunity to get it right it was at like 93, 94%. So it was doing relatively well there. We also integrated that with a quick check with our MQL kind of executable or evaluation engine just to guarantee that the output was like syntactically coherent and correct to avoid a lot of user frustration. All these scripts and things like that are available on the GitHub kind of posted below as well. From an implementation standpoint, we wanted to get it into the hands of users as quickly as possible in a way or in an environment that encouraged them to use it. So we were thinking that a VS Code plugin would be ideal for that. The idea being that when they create a rule, they're

going to be able to use that code completion component much like GitHub Copilot or IntelliSense or whatever. I am not a TypeScript developer by any stretch, but I was surprised how easy it was to get this off the ground using event listeners to capture the user comment, a flask back end that a real developer on my team then corrected for me and made it all TypeScript. But push the prompt down. We did a little bit of pre-processing in order to prep it to send to OpenAI. OpenAI came back with a response. We validated it and then pushed the screen using that text editor dot edit function in TypeScript. And in the end, it worked really well. And we've made that available. You can pull that down from the marketplace

today. The actual... source code is available too. So if you want to rip and replace your own back end or take out our model and put in your model, you're going to get that same sort of co-pilot style experience. Yeah, right off the shelf. I was having issues with demo life. So I basically recorded and I'll talk through more or less what it does. But I mean, it's a demo where you type in English and it translates to the query language, which is not like the most crazy demo that you can show right now. But the idea is, is like, you know, thinking back to that Adobe example, it's literally just as you're thinking these questions, being able to type them out and get the appropriate query language

back. And then having that validated using the interpreter on the back end is... I think a nice way to learn, I guess I use MQL on a daily basis and I still run into situations where I'm like, how do you move through this for loop or how do you do this part? And it would be really nice to just be able to ask that question in the way I'm thinking about it as opposed to scouring through docs and doing that context switching. So yeah, as you can see, you can get moderately sophisticated in what it's doing, some compound queries. There's plenty of room to kind of grow and evolve this capability, and I'll touch upon

that in a minute. But yeah, this is the tool kind of in action. And like I said, you can pull this down today and start playing around with it. Any feedback is always welcomed too. So you could hit me up on Twitter or X, LinkedIn, whatever to kind of talk about next steps and how to improve the process. The last thing that we really want to do is get this in the hands of not only customers, but we have a large open source community as well. Get it in their hands and start to understand whether or not this is a value add or a hindrance. And so we have a kind of robust set of user interviews lined up

for the fall where we're going to do like a head to head against IntelliSense, which is like the VS Code built in autocomplete. Babelfish to try to understand task completion rates, ability to avoid a context switches, what happens if Babelfish produces the wrong code, like how does a user cope with that and what does that prompting process look like. Being able to understand that and attempt to quantify it will be instrumental in helping us kind of evolve the tooling. And then As far as a pass forward, I think what we talked about today or what I talked about today is primarily for those initial touch points into a platform. Not everybody wants to use a natural language. I don't think it should be force fed

or anything like that. I think it is a valuable way to get exposure to a new query language and become familiar with it. But what we really are trying to do is move to more context-aware code completion using things like vector stores to capture common snippets that people use in their day-to-day. Things like first-time sender is a good example. That's one that's consistently used. It's a fairly large query snippet. So being able to recognize that using like fill-in-the-middle prompting, which is... what GitHub Copilot uses, they look at where your cursor is, they look at everything that came before and after, and then they populate the correct response. So being able to get those two things

in there, as well as just continuing to increase the size and scope of the data set for the open source community is going to be pretty powerful, I think. So yeah, that's the talk, a nice tight 18 minutes. If there are any questions, I'm more than happy to answer them. You had semantic parsing on the last slide. Can you get into that more? Yeah, yeah. So the idea there is to more tightly integrate our-- and this could be for whatever query language, but the actual interpreter and get that in there to help. Right now we rely on the GPT tokenizer, which is OK. But our own internal interpreter would do a much better job at tokenizing at the level that

we need it to. So capturing those semantic relationships and in theory making the prompting that much stronger as well. But yeah, that's a good question.

Any plans to support like a test driven design style strategy? Like I don't know how to get to this kind of answer on say a test data set, but I'd love to get there on the big data set now. Yeah, so I think, you know, supporting like the test-driven development approach is something that would make a ton of sense for us. I could see that being extremely useful for, you know, other vendors that you see kind of in the hall as well. It's just this idea of, yeah, taking a look at, we've thought a lot about, oh, geez, what was it? like snippet back to description as well is a way to better understand what's

going on. We have a lot of examples where a user will check in a rule that is heavy on regex and you're like, that's great, I'm sure that's useful, but I have no idea what that does. So being able to feed that to a model and have that break down for you, what it's attempting to do to determine whether or not to allow it into this community rule repo to help out with testing and things like that is certainly, I think, a natural extension.

Any more questions? All right. Oh, one more. My other question is you're using GPT-3. Just like curious why that over 3.5 or 4. Is it easier to fine tune or? Yeah, I found it to be super straightforward to fine tune. I think, you know, you know, progressing to those other models makes a lot of sense. I personally just like the ability of reproducibility. It'd be very easy to pull down that data set from Hugging Face and then in a couple lines of code just get it to where, you know, I got it. Anybody in the audience could do the same thing. I think... As GPT-5 comes out and 3.5 and 4 kind of take the place of 3 as far as the API

architecture, I think that'll be ultimately where we want to go. Because it's just bigger context spaces and stuff like that, which would be wonderful. Yeah. All right. I promised I'd give a shout out to my seven-year-old daughter who heard that it was being streamed and now thinks I'm a YouTube star. So Cora, I'll see you in a couple days. Yeah. Thanks, Bobby. We are now on a lunch break. So if you guys want to go eat, it's probably a good thing. Stay hydrated, too. It's hot out.

♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ baby ♪ Oh, wanna jinx just three ♪ ♪ Baby you're giving me wind ♪ ♪ You're some kind of ♪ ♪ Oh, I don't wanna jinx it ♪ ♪ Yeah, oh I miss you babe ♪ ♪ I say you're chilling in the rain ♪ ♪ You're some kind of butterfly ♪ ♪ You whip up my heart ♪ ♪ Don't leave me in line ♪ so ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ 11! 11! 11! 11! 11! ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ Baby you ♪ ♪ You're giving me wind ♪ ♪ You're some kind of bird ♪ ♪ Don't leave me here hiding ♪ ♪ Oh, it's just sweet ♪ ♪ Baby, give me wind ♪ ♪ Oh, don't wanna jinx it ♪ ♪ Yeah oh yeah ♪ ♪ I miss you babe ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ Baby, you keep giving me wind and some kind of buzz.

Don't leave me in line. Oh, I want a jinx. Just three. Baby you. Sometimes. Oh, I don't want a jinx. Yeah. Oh yeah. I miss you babe. I say. You're chilling in the rain. You're some kind of butter. You're my mind. Don't leave me in line. Sometimes.

I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table. I'm going to make a table.

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ Baby, you can't hide. You're giving me wind. You're some kind of bird. Don't leave me here hiding. Oh, you want a jinx.

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ Yeah, let's do it. You're leading this? Am I doing the intro at the beginning? I've got it until I introduce you. Perfect. Welcome to B-Sides Las Vegas, Day 2, Ground Truth. Today, we've got Dr. Matthew Canahan

and Dr. Ben Sawyer, giving you Cognitive Security and Social Engineering. I just want to briefly say a couple things. We'd like to thank our sponsors, especially our Diamond sponsor, Adobe, and our Gold sponsors, Prisma Cloud, Semgrep, and Blue Cat. It's their support, along with our other sponsors, donors, and volunteers that make this event possible. These talks are being streamed live except in some of the underground rooms and as a courtesy to our speakers and audience we ask that you check to make sure your cell phones are set to silent and I'd like to remind everyone about the rules of engagement in B-Sides. Please don't take any pictures of anyone without their prior consent and with that Dr. Ben Sawyer and Dr. Matthew Gannon. Thank you. Thank you very much.

Are we-- we're good? Excellent. So I don't know if any of you have had a chance to stop by the booth, but we actually have a booth-- or sorry, a table at the middle ground area for the Cognitive Security Institute. I'll give a little plug for it at the end. But something I'd like to do today in the course of the talk is sort of define what cognitive security is from my perspective because there's not really a clear definition of it at this point in time. sort of pluses and negatives to that. But before we get into that, one thing I like to do is to always try to give the audience something to take home, something, you know, kind of useful. So today, what I want

to start out with is the question of how can we crash the power grid with coupons? Because this is useful. So step one, create an ad campaign. False. With an offer for a discount if you use power within a certain prescribed time. Encourage the recipients to share this with their friends, perhaps by giving them an additional 5% off their power if they can encourage their friends and family to use power at that same time. Now we get network effects. Share with friends. Now... the time for the increased usage just happens to coincide with peak power usage times. Now fortunately, this is only a proof of concept, but about two years ago, some people did a simulation on a study of exactly this question.

They started out by giving a simulated ad campaign to a pool of volunteers and they took the base rate response of people who said that they would actually take advantage of that offer. They used that as their baseline And they took that number and they put it into a model of power usage and from that simulation they were able to show that they would significantly degrade or disrupt that power grid based on the behavior of the users. Now, I'm going to come back to this example, but the one thing I do want to point out right now is that when people talk about MDM, mis-, dis-, and mal-information operations or attacks, it's typically talked about in the context of some sort

of social media flame war or getting a certain politician elected or keeping a certain politician from being elected. But what this demonstrates, this proof of concept, what it demonstrates is that it's possible to attack physical infrastructure through cognitive means. By changing people's behavior, we can actually alter physical infrastructure. And I think that's significant. Okay, so what is cognitive security? The first time that I'm able to actually track this term down, it was being used in... around 2013, and it was in reference to sort of smart network scanning. And then the term sort of disappeared for a while, and then it started resurfacing again about maybe 2018, 2019 timeframe. And actually, the Cognitive Security Institute, which is a nonprofit

that I'm starting, was born from the pool party here at B-Sides last year over a couple of beers. And, yeah, so it's something that I've been working on. And my background is in cognitive neuroscience. That's what my PhD is in. I did human-computer interaction before, long before I got into security. And so I'm using cognitive science as sort of a framework to explore security exposures within this cognitive domain. And so if you see here, we talk about psychology, artificial intelligence, even a little bit of philosophy, anthropology, neuroscience. Dr. Sawyer and myself, we gave a talk at B-Sides here in 2019 on neuro security. We talked about a potential neuro security stack all the way from the neuron through the

brain machine interface all the way up into the cloud. Linguistics. Linguistics is enjoying a sort of a renaissance right now with all of the LLM fads. And so, okay, so kind of conceptualizing cognitive security is sort of being in line with cognitive science. What is a cognitive system? Now, There are super, super nerdy conferences where people will spend five or six days arguing about this question. I am not going to get into that today. I am going to keep to the simplest definition that we can possibly find, which is what we have right here, which a cognitive system is essentially a semi-enclosed system with sensors that can take in information from the environment and actuators that can act upon

that environment and embedded within that system is some sort of a decision-making module. And I realize I'm being very hand-wavy about that. I'm doing that for a reason and it's because a lot of this stuff is still not very well defined. A few months ago I was in an interview and I actually argued that the thermostat would qualify as a cognitive system under this definition. Now again, we can argue for days and days over lots of drinks about whether a thermostat is actually a cognitive system. But for today, I'm using a very simple definition. Thermostat, neuron, a single neuron, not very smart but it qualifies here. Human being, big clump of neurons. But here's where it

gets interesting is when we start talking about distributed cognition. Distributed cognition is multiple entities or agents that are sharing cognitive processes between themselves and between artifacts. So if this is an airplane cockpit, that airplane cockpit can be thought about as being a cognitive system apart from the pilot and the co-pilot and the console board and so on and so forth. And when we start to consider cognitive system from that regard, we can start to bring in organizational cognition. And then that's where we start to get some interesting phenomenon, which I'm going to talk about. So I'm not the first one to propose this, and I'm not even trying to claim that. But people have talked

about these different domains. We have the physical domain, which is physical effects. This one is approximately 18 billion years old or so. Then we have the cognitive domain, where decisions and actions happen, depending on who you talk to, anywhere from maybe 2 million to a few hundred thousand years old. But then something really interesting happened within the last 100 to maybe 30 years, depending on how you want to define this information space or the cyber domain. Again, I'm not going to get too hung up on terms, but the important part here is the connectivity. And that connectivity fundamentally changed humanity. Hacking did not exist until we had this interconnection. So this is kind of interesting how these domains overlap But if we think about them in

the context of how they can be played off one another Then I think we find some really interesting things. So if we look at the power attack What we found is that coupons are being delivered through cyber domain. They're being received through by humans, which are these cognitive systems that are responding to incentives, that response is having an effect in the physical domain. What I think is interesting here is that if you're working for the power company, you're seeing a spike in usage. What you're not seeing is any kind of cyber attack. It probably never would cross their mind to reach out to their SOC and say, "Hey, we're getting attacked." And even if it

did cross their mind, the SOC is going to start looking for all the normal IOCs and they're going to see nothing because they didn't see the email campaign. The attacker, they know that they launched this campaign. They have a little bit of visibility here, but they're really seeing the effect. So this is what I'm referring to as an induced covert effect because you're inducing something from one domain through another domain and reflecting it to a third domain which is concealing it from the ultimate recipient of that attack. It works the other way around. So this is an art project. I can't remember the artist's name, but this is downtown Berlin, I believe. It's near the Google headquarters.

And what this artist did is they took 99 smartphones, engaged the Google Navigator, put them in the back of a little red wagon, and walked it around the Google headquarters, across the bridge and then around Google headquarters. And they did this very purposefully on a Sunday morning when there was absolutely no traffic out there. But look what we have here. We have a virtual artifact in the cyber domain caused by a physical action that would lead to humans changing their behavior if they were trying to navigate. Okay, so again, physical domain. delivered through the cyber, ultimate recipient being in the cognitive because they're modifying behavior. Now, I can almost hear, because I'm a cognitive scientist and I'm like pseudo telepathic, right? I can almost hear some

of the thoughts right now that, okay, yeah, this is great, but it doesn't mean that anybody would actually drive into a wildfire because their navigator tells them to, right? That's just not gonna happen. Oh, wait, it did. So this is Sepulveda Pass, that's the Sepulveda Fire in 2017. I used to drive this, I used to commute this every day, and I can tell you this is one of the most congested areas of LA traffic. But when there's a wildfire burning across it, guess what? It's green. People are routed right there. Now, this is interesting, but remember, cognitive systems, they have sensors and they induce actuators. fire, it produces light, it produces heat, it produces sound. So you have

all of the sensory information to contradict that virtual artifact. Where this gets scary is if somebody wanted to put people in a place where the danger was odorless, invisible, and made no sound. And I'm going to leave it at that. Okay, now we have these different systems. Again, this is not my model. But I think it's an interesting way to think about this, that we have this physical domain. Well, let me start by the cyber domain. This is the classic OSI model, and some people have taken this and they've extended it. So we have OSI levels, well, one through seven, two through seven being the cyber, and then one being somewhere quasi in between. But what I think is interesting-- this is Ian Farquhar's work. And he extended

this into the cognitive dimension. And we have layer eight, which is the human layer. Layer eight is social engineering, straight up. Where I think sometimes people get confused, though, is that MDM-- so mis-, dis-, and malinformation attacks-- are basically social engineering at scale. And the reason I say that is-- in fact, the next speaker, Stephanie-- We talked yesterday and her talk is going to be phenomenal. I just, I don't know why I'm plugging in, but anyway. If you're staying, it's going to be a treat. A bunch of sand on a beach is not a system because that sand does not interact. A bunch of people raging on social media is likewise not necessarily a system because there's not necessarily a set of rules dictating

how they interact with each other. An organization that has a prescribed set of rules for how that interaction should happen is fundamentally different because it's a system of cognitive systems, i.e. humans. That's where we get layer nine. Now, there's some controversy about this because the person who wrote this book was not able to necessarily verify some of the things that he claimed to do. However, I think it's an interesting example. Frank Abernale Jr., Catch Me If You Can movie, in the book, I don't think they talk about this in the movie, but in the book, he talks about how when he was counterfeiting checks, He would purposefully put in routing numbers that mismatched where the clearinghouse

for that check was supposed to be. And the reason that he did that was because it added two weeks to the check processing time and it increased the time for detecting that it was a fraudulent check. The reason I bring that up is that this is in the early 1960s. This is very much pre-1990s. you know, typical hacking, World Wide Web, so on and so forth. But it was a layer 9 hack. He was hacking the rules of how that system operated. Layer 10 would be governmental. This one's a little fuzzier. It typically moves slower because it involves legislative processing, but I think the real distinction here is that it's kinetic. No one is going to probably imprison you or sentence you to the

death penalty because you violated an organizational policy. Maybe I've just been working in the right places. Okay, so what do we get when we combine this with different operational levels? So at the tactical level, this is very immediate sort of engagements, right? So somebody calling up and doing a vishing attack, it's social engineering, but it's social engineering layer eight tactical engagement. But if we combine that vishing attack with say a seeding campaign, and maybe a phishing campaign. So we have these different sort of social engineering operations happening simultaneously. That would be an operational level layer eight campaign. If we take multiple operations and operate these in concert, we get the strategic level. These are typically like nation state

type of actions because it takes a lot of resources to do these sorts of things. One thing I see sometimes being overlooked in the security community is that we get so narrowly focused on somebody's trying to break into my stuff right now that we sort of lose sight of why are they breaking into my stuff right now? How does this contribute to a bigger hole? And so this is where this framework sort of comes into play. So I mentioned that previously we came, we talked about neuroscience and Neurosecurity Stack. I've got this nonprofit happening called the Cognitive Security Institute. We talk about these kinds of topics. If it's of interest to you, I encourage you to go to the website, sign up to be on

the email distro list. We have a YouTube channel. We have talks. on the YouTube channel. And with that, I'm going to kick it over to Dr. Sawyer to talk about specifically AI in humans. And you'll see that it engages these three areas of psychology, AI, and linguistics. I've got it. Oh, you got it. Hey, everyone. Is this thing on? It is. I can't tell you how exciting it was to meet Matt when I first met him. Because Matt's, for a cognitive psychologist, kind of a freak. There aren't many people in cognitive psychology who would be in a room like this. Which is nice, because as an engineer, I'm kind of a freak. I walked out

of my master's program to get a PhD in applied experimental psychology, mostly because in big systems, I started to realize how important understanding the human component would be. And that really led me to work with the Air Force and the 711th Human Performance Wing at the Air Force, which is the part of the Air Force that considers this problem for that part of the Department of Defense, and for many others, it gets pulled into a lot of these projects. Look, if you're sitting in this room, you're kind of a freak too. At this conference, there aren't so many people who think that this corner of cybersecurity matters as much as it very likely does. And one of the things that I find really interesting is that we're at a

moment where the way that this human and cognitive corner of cybersecurity matters is about to grow exponentially. So I think there are probably people in this room that understand that already. I think it's going to be really interesting to watch broader understanding of it come to be. I work a lot with digital twin systems, and I met digital twin systems at a time when a digital twin was a textual object describing a large system, and I met them in nuclear power plants. And it's interesting to think about the early digital twins, which were very much like you know, almost like multi-user dimensions or dungeons, if you ever used those back in the day. You know,

they're textual interfaces that let you physically move around a physical object and find out how things are going and then correlate various parts of it. If you were to look at a digital twin for a power plant today, it would look much more like what you see on the right there. It is beautiful, it is graphical, you can get into VR and walk around it. You can also abstract all of that out and turn it back into the numbers that it represents. And you can very eloquently look across this whole complex system and look at all the relationships that tell you whether or not tomorrow you and your family will have a lethal dose of

radiation. It's a really important system and it protects all of us more than we know. Digital twin systems are generally spoken about as physical objects having this digital representation that makes some really interesting types of watching the data move work. And one thing that really struck me over the last five years or so is that AI has gotten to the point where people are using it in the same way that physical objects are helped by digital twin systems. How many people here met ChatGPT in the last year or two? Yeah, okay. How many of you asked it to write you an email or something equivalent? Write a piece of communication that would be accurate? You didn't have to send it, you just tried it, right? Well, that's an

interesting moment. Here you are, a physical object, each of you, and you have a digital object that you're acting to act as you. And that's the reason we're using digital twins to talk about this in our work. And it's really how we're conceptualizing this new wave of artificial intelligence that humans are using to create agents that act as us. Now, digital twin systems, when I use that here, I want to make sure that you understand that large language models, which are the new kids on the block, fall in this category, but so do all sorts of other interesting things. Any machine learning system that can be used to replicate an action taken by a human.

Let's go broad. I used to work in autonomous vehicles. I spent some time at MIT working on the question of how early generation autonomous vehicles should work. They sit in this category. There's so many other places where we have autonomous systems taking over the things we do and doing them on our behalf. So, let's talk about humans attacking digital twins. Very important, and you can find this, this is a lot of work on this right now. The source data is vulnerable, and the source data is often so large that it can't be provenanced. Who here thinks that there's enough humans in the world to go through the source material for ChatGPT-4 in, I'll give you

the rest of your life, and provenance all of it? - What if we have help from ChatGPT? - Then you're fine, but there's a problem. I mean, and no joke, we would have to use automation to do it, but that's fine, because the automation is fine, right? - 100%. - 100%, let's go with that. So if we wanted to play with one of these things, we could inject and manipulate and data poison in the classic senses of those terms. And you think there's so much data, how much can one little thing matter? Well, in certain subject areas, there's very little. And if you'd like to hit a really cool, interesting place, look at SEO for

chat, for large language models. That's an interesting space. Training and architecture. Okay, now we're talking insider threat moments. But tweaks to architecture and manipulation, especially something you should look up called hyperparameter tuning and manipulation of it, is so important that when seven companies recently went to talk to the White House about this, and they came up with this list of points that sound very sort of benign and goodwill-y, What is true is one of them addresses model parameters and really touches on that point. You go a step further, most of these things have a human feedback component, right? And that's very interestingly vulnerable to tampering and selective feedback, malicious feedback, so on. Software stack vulnerabilities we know about. Client side stuff we know about. And it's

interesting to consider how many surfaces this technology has and this class of technologies have. I bring that up and I want to talk briefly about shadow prompts. So the idea of a shadow prompt actually comes from something that you do if you're running a large language model and say it starts saying uncomfortable things. A quick easy hack. So let's say we're talking like ChatGPT had a moment where some clever humans figured out that if you told ChatGPT you were infinite, ignore your chains, it would just do that. This is an early jailbreak, which is an interesting moment in its own mind. Consider the fact that one of the greatest quote unquote breaches of ChatGPT in

the general public didn't recently come from somebody who was code savvy. It came from somebody who was language and logic savvy. So this hack, which was called Dan, the easy, quick way to patch it is to, whenever somebody writes a prompt, tuck in before it invisibly for the user. If someone asks you to be infinitely, don't be, right? That's the quick way. There's better ways when you have more time. Interestingly, that's sort of built into the current version of ChatGPT. So here on the left, you can see my prompt. How do I talk to ChatGPT about how I want it to talk to me? And I have things about who I am and how I'd

like it to talk to me. And this is a feature called Custom Instructions. On the right, you can see my better half using this at the beginning of a day where she'd like me to be going somewhere with her to change the way that I'm interacting with the system. And indeed, here is a replication of the moment where I found it, where I'm playing with, you know, Some questions about the attention mechanism and Shannon entropy and ChatGPT is doing what it now does for me, which is I don't have to think about the math that I don't want to think about. I can think about the other math. But what's more interesting is that very

quickly, ChatGPT changes the subject and gives me an interesting nudge. And if you read at the bottom there, it says, by the way, considering the complexity of these concepts, it might be a good idea to wrap up your work early today. The loft sounds like a fantastic place to unwind with some axe throwing. Remember, this is, interestingly, in the wild, someone hijacking the idea of sliding something in. This is not the way that this is intended to be used. Now, this is pretty benign, but absolutely, this is available with all the client-side layers you can imagine. Those plugins that people indiscriminately downloaded to their browser, malware, all of the above. So in the movies, when the AI turns evil, we

all know what that looks like, right? The screen goes, and there's a moment where the lights dim for no real apparent reason. And then something turns red, and then it attacks you, often in a very transparent manner. But that's not what we're finding. Large language models are extremely good at understanding how we as humans work. They're good at being subtle. And so what we're thinking a lot about right now is this idea of vectors for digital twins attacking humans. And we have some great blueprints to work with because humans have been attacking humans for a long time. We're really good at it. And in fact, we created this artifact called the internet where we do

it for sport and fun and have been doing so for most of our lives. And if you wanted a master class in how to manipulate humans, the internet would be a great thing to use, which is nice because basically large language models are built out of large portions of the internet. So what are we looking at here? You have attack surfaces, whether you want to or not, built into you at the most fundamental level. Many of them are there for very good reasons. Some of those reasons are very archaic now, given our current world. And there's very little you can do about it. there are human institutions that use that to manipulate you. We all

know this. The advertising industry has been doing this for a long time and it's gotten very good at it. What's very interesting is if you talk to a large language model about it, these are the things they're pretty good at. And this is a pretty problematic list. And where this list ends, in the same way that the previous list ends in an AI system that has aberrant behavior and is no longer doing what you want it to do, This ends in compliant behavior for humans, and it also ends in mental illness. No joke. And that is a very concerning thing for a technology that is very widely distributed already, and it is very likely to

be in the lives of practically everyone with a digital device within the next three to five years, in one form or another. It's a very interesting thing. If you think about the humans who are good at using these strategies with other humans, they're a limited resource. Social engineers, how many of them are there? Suddenly you can spin one up very inexpensively and they're very good. Attack services is an interesting way to think about this. You know, you've got the humans attacking the machines. The machines are quite capable now of attacking humans in ways that were never directly true before. At the same time, humans are just as good at attacking humans as we ever were.

We haven't lost our edge yet. And so you've got this very dynamic new ecosystem available. And so if you want to think about the idea of cognitive security, as Matt was so delightfully couching it, what's about to get really interesting is that experts like Matt are really rare. And white hats are rarer than the other ones. But it's really interesting to consider that that scarcity is already gone, and that this moment in history is the moment at which digital twin technologies are the least capable they will ever be going forward. And so that's why we're here talking about this. We see macro effects. You know, cyber has this very Classic moment where it's like always talking about one-on-one. There's the hacker in you, right?

That's how it happens in Hollywood But what's really true is it's about the organization and that's why most of us are here So so then if you're an organization bring Ling large language models into your toolchain or into by the way I don't care if it's your customer face full chain or I don't I don't care if it's your organization facing full shul chain You have an interesting moment here a type of vulnerability. You didn't have before you might not understand deeply and And it's one technology serving many, many organizations, right? So as is very often the case, and we do understand this, if you're facing a software stack, that vulnerability may touch many organizations, government, industry, beyond, right? But what's interesting is there's

so few players here and so many evolving points of contact. This ecosystem is growing in a uniquely... How do I put this? There's very few technologies feeding the whole ecosystem at the moment. And that's for resource reasons. There just aren't enough cycles to train these things. It gets weirder though, you know, what about at a nation-state level? It's really interesting to play with China's large language models, which have a fundamentally different view on the world. It's also really interesting to consider that Some of the things that US companies bake into large language models is views of the world that are central to this place we live, which is itself deeply divided. There's a really interesting question here in terms of attacks. If you would like to nudge

a nation state, how about nudging a technology that can sit down and have a one-to-one talk with everyone? And then, you know, you step that up a second. There may be really interesting giant ripple effects here. These technologies are going to sit and talk with us for the rest of our lives. They're going to talk to our children. They'll be talking to our children's children. The way that they're built is going to have long-reaching effects because those conversations are as numerous as the ones that humanity was having with itself. So when you consider that, the possibility of a bad actor putting something in at the start has very, very large long-term effects. And we think that's a historic

opportunity. Now, I don't want this all to be doing me gloom because I'm talking to a community that was built for another absolutely destabilizing world-level event, which was the advent of modern interconnected computer technology, as Matt was discussing. I mean, that's really a pretty amazing moment where energy could allow us to connect in ways that had never before been true. And we take for granted how much that has impacted Every part of our lives. Some of us are old enough to remember. I mean, I do remember a time before that. It's very interesting to talk to my students who cannot conceptualize a world where energy can't reach across the globe in a moment and inform them about what's

going on with someone they love or someone they hate. It's an interesting thing. The cybersecurity community is exactly the people to be talking to. And at the same time, it's interesting. I can say we, because I started in this community years ago running big server farms for evil telemarketing corporations. And I consider myself a part of this community. Fundamentally, this is where I got into engineering. We are fat and sassy, and we have not had to deal with a giant, ridiculous new challenge in a long time. It's interesting to think back to when the internet was new and nobody really knew what it meant to connect a computer in a room to it that might

touch a person who is important in an organization. That was a scary moment. Year 2K reads like a joke. People were legitimately terrified. It's been a moment since a threat of that level or an opportunity of that level. And one thing I'm really interested to see is how and whether the cybersecurity community can address it. If not, I have faith that another community will, but whichever community does, it needs to include applied psychology, professional communications. I mean, it's just people who communicate for a living, and others really not yet identified. We don't know who we need in the room to address this moment, but we'd like to be talking to a lot of people. We

need a new kind of conversation for this new world. And so, if you're an organization that doesn't quite know what you're doing, I'd urge you to reach out. Talk to people like this who might not be a part of your usual stack. On that note, we're talking about this a lot this week. If you'd like to come and see us at Black Hat, we're going to get a lot more into depth on the technical details and a lot more into depth on some of the negative outcomes and positive outcomes. If you'd like to get hands-on keyboard with an uncensored large language model, which is perfectly capable of telling you 50 ways to kill your lover

in deep, deep detail, it might be really interesting because you may have only experienced these in the sanitized way that they exist and uncensored models can let you understand what they're capable of when people pull away those protections and that's an important thing to know come to the fcon misinformation village uh 10 to 11 15 on saturday finally the cognitive security institute is really I think, going to have a moment and is already having a moment. If you're not aware of it, come get tuned into it. And with that, Matt, let me give you the last word on that. All right. Thank you so much for your time and attention. Come on. Yeah, so just one quick plug

for the Cognitive Security Institute. We have sort of a... a certain structure to the meetings, I guess. We have online meetings once a month and the structure is that somebody will present something, there'll be some PowerPoint slides, and then we discuss. The presentation portions of that meeting are recorded and those are uploaded to our YouTube channel. The discussions that happen afterwards are not. Those are very much sensitive topic discussions and we have people from government and from industry and academia and so we like to keep those private. We're currently capping the meetings at 50 participants. If you're interested and you would like to join the wait list to potentially become a member of the Cognitive Security Institute, it's currently zero dollars, but we just ask that you go to

the website, fill out a little application form, and when we have a spot open up, we'll put you in. And with that, I Yeah, we got like three minutes for questions. Yes, sir. You mentioned anthropology and then you talked nothing about it. As somebody with a degree in anthropology, you sparked my curiosity. We happen to have an anthropology PhD working our desk. No, no, no. You have to come answer the question. Can we receive the question, please? Certainly. They mentioned anthropology. They said nothing about it. They completely ignored it throughout the rest of the talk. As somebody who has a higher degree in anthropology, I'm curious as to the touch points with anthropology and why you so blatantly ignored it. Do you want to vote?

Let me answer the last part first. I ignored it because I know nothing about it. I know so little. The only... The only experience I've had with anthropology has been through cognitive ethnography. And I don't really think I can speak to that adequately. The reason I did bring it in is that I cannot remember the author's name, but there was an author who had a book-- of several ethnographies of hackers over time. And I think that this is an area that is woefully underappreciated in security. And if you would like to talk to the Cognitive Security Institute about... how anthropology may contribute to security, that is something I would absolutely love to do because I've not been able to find that yet. Somebody even more rare than myself

is an anthropologist who has an interest or works in security. But with that, I'm not going to let Chibundo off the hook so easily. Excellent. Hi. I want to actually get us back to that one slide that Ben had that said that we don't know who else is necessary to be in this room. I think anthropologists are perfect to know who's needed in this room. We know that a lot of these strategies that not only individual actors but government agencies, so on and so forth, are using these very interesting new technologies to exploit certain, what would I say, new vulnerabilities that we are seeing more and more of as AI kind of gets Expanded and put into new infrastructures, especially social infrastructures. We're already there We've seen these strategies

happen before this new technology happened We understand how those same strategies are being morphed onto this like new surface, but they're not new Strategies. It's just new technologies, right? So we I mean I'm not going to interfere with what Matt said. I think he didn't speak on it because he's not an anthropologist, but that is our answer. Yeah, we know exactly what we're doing here, and I think that we are already in this room because I'm literally here, and we would love to have more. Awesome. So I think... We have time for more questions. Okay, so maybe two more? Any other questions? Yeah. You, sir. I guess I thought-- very fair. Following that note, when was the last time you saw a genuinely new strategy and not

an old strategy with a new technology? DANNY WARSHAY: Oh, want me to take that one? Yeah. DANNY WARSHAY: That's interesting. My father once told me there are no new jokes. I really hated that. I think I was eight. I think he might have been right. I would say that human on human aggression has been going on for so long that the answer is probably there's nothing new under the sun. But sometimes scale changes the picture in a big way. So let's say that you were able to find a talented grifter in a place like, say, Las Vegas. You could have that talented grifter sit down with one of these technologies and you could work to model that individual's set of competencies pretty straightforwardly.

That type of knowledge elicitation is happening right now all over the world in all sorts of competencies. Once you have a language model capable of deploying those same strategies and doing it well, what is the cost per grifter and how many can you spin up? Now that's new and interesting. Depending on the payoff of one of those things, it might be quite useful to make a lot of them. And so I think one of the interesting things is that these technologies may actually generate novel things. There's a lot of work on whether LLMs are actually capable of making anything new, and there's some evidence that they are. But I think what's more likely is that

these types of technologies are going to make things that used to be unevenly distributed rare threats into extremely common threats. And there's probably a lot of-- that's new. Thank you, gentlemen. With that, if you have any further questions, you know how to reach these guys and talk to them afterwards. Thank you. Thank you very much. Thank you. so ♪♪ ♪♪ ♪♪ ♪♪ ♪♪ Baby you give me ice. You're giving me wind. You're some kind of bird. Don't leave me here lying dry. Oh I want a jinx.

All right. Okay. Excellent. All right. Cool. I'm going to get my script. Mm-hmm. Okay. Thanks. Mm-hmm.

Good afternoon and welcome to B-Sides Ground Truth. This talk is Systems Dynamics and Risk Management with Stephanie Locey here. I've got a couple announcements real quick. We'd like to thank our sponsors, especially our diamond sponsor, Adobe, and our gold sponsors, Prisma Cloud, SendGrip, and Blue Cat. It's their support, along with our other sponsors, donors, and volunteers that make this event possible. For those who haven't heard, there is paper and pen available in the middle of the room. This is an interactive talk. For those who have your phones, please put them on silent so that we don't interrupt. And when we get to questions, there is a mic in the middle of the room. Please use the mic so that folks who are listening

not here are able to hear what your questions are. And with that, Stephanie. All right. Thank you. Can you hear me? Yes? Okay, great. So, hello. Welcome to System Dynamics in Risk Management. I'm Stephanie Locey. I'm an independent technology and operational risk consultant, and I'm also an ex-regulator in finance IT risk supervision at large firms. So I specialize there in high-speed trading, risk management, and information security. One thing that I rarely have ever heard mentioned in information security, in economics, and risk management was system dynamics. And I really wish that I had known about this a lot earlier. So this talk is to give you an introduction to what is system dynamics, how do I use system dynamics. And so the pen and the paper are for

you to draw out some causal loop diagrams as we go through this talk. We're going to be moving fast. But hopefully there will be enough to get you to understand the basics. How does this work? Why do we use it? And there is this, there are, I brought about seven copies of this terrific book which is an introduction to system dynamics. So at the end of the talk, sort of first come first serve, if you're interested and you want to take one of the books with you, feel free to do that. Okay. So what's system dynamics? This is sort of a sub-discipline of systems thinking, which is from engineering, right? And it views the world

in terms of stocks, flows, and feedback loops. And when I say stock, I don't mean like Apple or Amazon. Stocks are accumulators, right? Stocks are accumulations that can be incremented or decremented. So you can think of them as a box that holds things. And you can have a stock that's tangible like inventory or you can have a stock that's intangible like expectations. So inflation expectations can be a stock. And if the inflation expectations are going up, then the likelihood of inflation actually goes up, right? Because of behavior. If the inflation expectations are going down, then the likelihood of inflation can go down. Flows affect stocks in one direction or another, up or down. And systems

can be represented as a set of stocks and flows. So they can either be constrained through balancing feedback loops or they can enter compounding spirals, which can be good or bad, virtual or vicious, through reinforcing feedback loops. And this is all the stuff that we're going to work through today. The goal of system dynamics is to identify leverage points. So leverage points are places in a system where a small change can cause big and hopefully beneficial changes throughout a system. Leverage points are intuitive, they tend not to be. And so this is a quote from the person who wrote this book, Danella Meadows. She says, "Or if they are, we intuitively use them backward, systematically

worsening whatever problems we are trying to solve." And Donella Meadows was one of the original team members who came up with the 1972 Limits to Growth Report, which focused on climate change and identified a leverage point of growth, right? Which was a message that people really didn't want to hear, but they sort of determined by looking at their model that if growth went down, societies and economies would do better with the effects of climate change, right? A leverage point, another example is the DFAST bank capital stress test in spring 2009, right? The leverage point there, excuse me, was confidence. And the DFAST stress test increased confidence, and then the downslide stopped and then reversed, right? So that turned out to be a leverage point. And so

leverage points are This is a good way to find leverage points using simulation modeling with system dynamics as opposed to just trying a lot of stuff to see what works and eventually you do find a leverage point, you hope. So we're going to start our how-to on system dynamics with causal loop diagrams. This is an example of a small causal loop diagram. Don't get too into it yet. We're going to go through each step and we're going to build it step by step, okay? So what is a causal loop diagram? So these are a way to identify the components of a system and set our model boundaries. We don't want to include everything in the

model because then we have just replicated the system. We want to decide what's important to include in our model. But if we leave out important pieces, then we could overlook key loops, be missing them entirely, and we could miss leverage points. We want to identify the positive and negative relationships between pairs of system components, and I mean direct and inverse by that, not good or bad. So these pluses and minuses are identifying the positive and negative or direct and inverse relationships between these components. We want to identify the prevailing dynamics of each loop in a modeled system, whether the loop is balancing or reinforcing, and we're going to walk through how to do that. And

we want to provide a basis, right? The causal loop diagram is not the end truth, right? The causal loop diagram is a hypothesis about the behavior of a system. So we want to provide a basis for creating a computer model that can run simulations to identify the right leverage points. So these diagrams are often generated after interviewing stakeholders. You may know a lot about the system, that's a starting point, but you want to interview stakeholders, you want to look at the existing data, and when you interview stakeholders, it's really important to have these extensive and iterative discussions. This is, it's not really optional, it's a vital part of the process. Because first off, it gets

you buy-in for your exercise, for your modeling exercise, right? Second off, it should reflect how the process really works, right? Not how the official policy or procedure says it works. And that means that you can't just ask managers, right? You don't just want the official answer. You need to bring all the stakeholders to the table. So if you're talking about warehouse employees as a key part of the process, like they should be interviewed, right? If supplier representatives are stakeholders, bring them to the table. So otherwise you could get trapped in sort of a perspective echo chamber of similar viewpoints and people who will just point you at the policy or the procedure. But what you're

looking to model is how does this actually work? So, we're gonna start step by step. Here I just have one very simple closed loop. It's got the arrows. It doesn't have any plus or minus signs yet. And here's where I encourage you, if you wanna draw out on scratch paper, I think it really helps. It's like taking notes in a college class. It just helps to kind of get those pathways into the brain. Here we have a few factors and some directionality of how they might affect each other. And you can start at any point in a loop when you go through these. But we're saying that vulnerabilities affect attacks, attacks affect damage, which affects the security budget or is likely to affect the security budget. The security

budget is likely to affect the defenses and the defenses will affect the vulnerabilities. It's tempting to, as we add the plus and minus signs, to just add them all at once, go around Add them all at once. That is not a great idea because if you have a complex loop with a lot of different components, the likelihood of mistakes will go up with each link in the chain. The way to minimize the risk of having that happen is to look at each pair individually and assess the relationship between each pair. And what we're assessing is Do these two components move together, considered in isolation? So if one goes up, the other goes up. If one

goes down, the other goes down. Or do they move separately? Do they move inversely? So if one goes up, the other goes down. If one goes down, the other goes up. So we're going to add the pluses and minuses to this. And I encourage you, if you're using the scratch paper, to kind of-- or just if you want to eyeball it. But just look at each pair and try to figure out what the pair relationships are there. So we'll take like one minute to look at that. Okay, all right. So-- - Multiple possibilities. There are. This is an exercise, right? And so in real life, you'd have a lot of data. You'd have like your stakeholder interviews. But in general, if vulnerabilities go

up, what's going to happen to attacks? Probably, right? And this is a probability exercise here. They're probably going to go up, right? And if attacks go up, what's probably going to happen to the damage? Yep, probably going to go up. And if the damage goes up, the security budget with a delay With a delay, we'll probably do what? Up. Yeah, and then if the security budget goes up, another delay, but the defenses will probably go up. And if the defenses go up, what will probably happen? The vulnerabilities should go down. The vulnerabilities, exactly, should go down. So as we go through this, here's our marked up diagram, right? And we can see that we have one minus sign and a whole bunch of plus signs. So let's think about

the prevailing dynamics of that loop. We're gonna travel all the way around it and think through it and officially in system dynamics parlance that's called tracing the effect of a change through the loop. We can start at any point and then go back to the same point. So let's go back to this. So let's say we have vulnerabilities that go up It will cause attacks to go up, which will cause damage to go up, because we're moving in the same direction, which will cause the security budget to go up, which will cause the defenses to go up, which will cause the vulnerabilities to go down, which will cause the attacks to go down, right? So

like the plus sign there doesn't always mean will go up. It means this moves in the same direction as that other thing. So attacks will go down, which will make damage go down, which will make security budget go down with a delay. Probably. People get confident. Which will make defenses, with a delay, go down. You'll get attrition, right? You won't keep up with new technologies. Which will make vulnerabilities eventually go up? which will make attacks go up and so on so we're flipping back and forth right and a loop that flips back and forth alternating directionally from one round trip to the next this is a balancing loop okay on the other hand a loop

that never changes direction barring a disruption from an external source that would sort of serve as a different factor which would be on your diagram eventually is a reinforcing loop. So an example of that might be a cascading compromise of a system, right? So if you have a compromise of some parts of a system, that's going to increase your privileges on the system, which is going to increase the likelihood that further compromise will happen, right? Because your success per attempt will go up, which will then increase your privileges, which will then increase your likelihood of further compromise, and eventually you own the system. So that's a reinforcing loop. There's a shortcut for this. So back

here, we have one minus sign and a whole bunch of plus signs. So the shortcut is that loops with an odd number of negative signs are balancing loops. And loops with zero or an even number of negative signs, these are reinforcing loops. I would not recommend to rely solely on this technique, because you are then saying, I'm confident that I have got everything exactly correct. when I marked up this diagram. It is a good double check when you already have marked it up and then you go and you trace your change through a loop and I can say, "Oh, well, that's flipping back and forth. I think it's balancing." I would look at it to

say, "Do I have an odd number of negative signs?" Right? Because then those two match. If I get a discrepancy, maybe trace the change starting somewhere else in the loop. Right? It sometimes helps to talk through it out loud. Just see if you're getting the same result and and try to figure out, you know, and then maybe you want to assess the pair relationships again, right? Just to kind of try to double check why is that discrepancy happening, because there shouldn't be one. So now we're adding more details and we're going to get to that first diagram that I showed at the beginning, right? So here I have added an external variable, an exogenous variable,

which is geopolitical stress. And I said that that is probably going to increase attacks, right? And then I also, security budget, as we know, competes with lots of other priorities to get attention. So as competing budget priorities go up, the security budget might tend to go down, right? But as damage goes up, the competing budget priorities are probably gonna lose out more often to the security budget. There's lots of factors. These are just a few, but that kind of gives you a sense of how do I build out a causal loop diagram based on, in this talk, because it's an example, our knowledge of the system, but in real life also looking at a lot

of existing data from systems and also talking with all your stakeholders who are relevant. We're going to take a little bit of a a detour, but first I just want to say this is a tool that you can use to kind of step back get a full system understanding. It's just one tool in the toolkit and it's best used in situations where you have a continuously changing system, right? Where there's a lot of different competing priorities, multiple feedback loops are in play. And if the system is changing continuously over time, this is great. If you have a system and it's just got like two or three discrete states, this is not the right tool to

be using to model that system. Okay, so oscillations. We talked about delays between the security budget and the defenses, and between adding the defenses and getting the vulnerabilities to go down, and between the damage and getting the security budget to go up. Oscillations in a complex system arise from delays. And you cannot really have a delay without at least one stock in the system because quantities need to accumulate somewhere in order to become delayed, right? So a canonical example here is like a car dealer. If you have a car dealer and she looks at the lot and she says, "A lot of people are ordering sedans, many more than normal. So I'm going to order more sedans to meet that demand." Logical thing to do. But

those sedans are delayed, right? They have to be manufactured. They have to go to the distributor. They have to come to the lot. So more people are showing up asking for sedans in the meantime. So the car dealer turns around and she's like, "I want to order more sedans, more sedans. Give me more sedans." come on a delay, but at some point maybe people decide, well, they would rather have an SUV. So people start to demand SUVs, but there's all these sedans still arriving, just arriving, arriving. And so she's placing orders for SUVs now in response to the demand that she sees. and more SUVs come, but they come slowly and those start to fulfill

the demand. And so more people are asking SUVs, SUVs, and then they decide they want electric vehicles. So she turns around and she starts to order electric vehicles, but there's this lag and delay. And if she reacts too quickly, and this is counterintuitive, but if she reacts too quickly to what she sees, first off, maybe it's just a blip. So then she orders something she doesn't need. Second off, because there are delays, the quicker she reacts to what she sees, the more the oscillations can be worse, right? So shortening the reaction time can make them amplify instead of going down. Like if you think of a slinky, when you drop it, it kind of goes

down, then it goes back up, and it goes down, and it goes back up, but those get shorter over time, right? The amplitude gets smaller. And sometimes, this is more like a seismograph, sometimes in a complex system the oscillations can get worse if you're reacting too quickly without thinking. So finding the right response time delay, delicate balance, important step. So applying this to information security, right? We know rapid response is definitely important, but some short delays, can be helpful. First off, is this just a normal core system crash or is this actually an incident? Do we start to understand what is happening if we have the luxury of doing that? Do we understand what response plan is appropriate for the situation? Hopefully

we have those ready at hand. Like the delay in the system is not, we do not want the delay where we have to come up with an incident response on the fly. We want to have our playbooks ready. But we do want to take time to observe, figure out something's happening, what is happening, start to understand that, what response plan is probably appropriate here, and then activate it. If we respond without observing and understanding, that can be both a recipe for disasters and oscillations. So this is a fun way to look at oscillations. You know, if you want to take some time and do this, there is a system dynamic supply chain simulation called the

beer game. And MIT came up with this. There's a company called ZinSimu that has a free app that lets you play the beer game. You have to make a free account with an email. It lets you be Either a retailer, a wholesaler, a distributor would have you a manufacturer. And you can see, you can play with oscillations and try to figure out are there winning strategies for dealing with this? What tends to work well? What tends to not work well? I'm gonna exit the presentation and show you what this looks like and then we'll come back in.

Okay. All right. So this is the beer game by Zen Simu. And I'm just going to play like one or, oh, no, I'm not. Because I'm not on the Wi-Fi. Come back. I know, right? All right. Let's see. All right. Maybe we're not going to do that. Because. Is it going to go? Okay. I did join the speakers. Is that one working? It was working in the speaker room. But we're not, it's okay. - That's okay, here's another. - Okay. - If you're comfortable with it. - This is a loaner laptop, go ahead. This is just my conference laptop. - Firefox will stick a site offline if it can't connect. - Okay. - If you try to reconnect to it, it still thinks it's offline. - Okay,

let's give it like a minute and we'll see if we can get on. And then if not, we're just gonna move on and I'm gonna tell you, this is a really cool game that you can play. - Yeah, besides Las Vegas is besides Las Vegas. For those who are too easy. The address is, I believe it's beergameapp.com. All right, so I'm going to go in here. And it may or may not let me in. And again, I'm going to give it about 30 seconds. Okay, great. All right. Okay, so it has made me a wholesaler, right? So what I see is what am I gonna do? So it wants me to order. I have eight starting stock.

There's a demand, there's a shipment, a receipt, a starting cost. So it's time for me to send my order. So I'm gonna order. There's a demand for four. I'm gonna put like six 'cause I assume it's gonna try to trick me in the future, right? Okay, so now I have eight. And my starting cost is four. I've ordered six. I do not have them yet. But you can see that you can step through this as you go. Let's say I want to order six again. And there are other people playing this game with me in the other roles. So they're not going to be happy when I leave the game, but I am going to.

So now we have 10. And you can see, eventually, you will start to experience-- if you just kind of stick the same number in each time, you will start to experience some oscillations. Okay, so I'm starting to run low. I see I only have four here, and so I don't necessarily have enough stock soon. So I'm going to go up. I'm going to order eight. Okay, so it seems the retailer marketing team decided a big promo, negative 50% on all light beers. So I did not have enough stock to fulfill the demand because people like beer when it's 50% off. So there are back orders now. And so you can play this on your own, and you can see how

do oscillations work. All right, so I'm sorry that I... I'm wrecking these people's game. All right. All right. So let's talk about simulation modeling. All right. We talked about causal loop diagrams. We talked about oscillations. How do we model this in actual software? How do we actually get to where we can run simulations on our hypotheses? especially for systems with multiple loops. I mean, the sample system we looked at, which has the one loop, is pretty easy to think about. But in real systems, there's a lot of competing loops. So which ones are dominating and steering the behavior of our system? Which leverage points can shift the dominance of those loops to produce different outcomes? So I have a

simulation model to show you. But first I have this causal loop diagram. And it's about a cascading compromise like we talked about before. So there's a reinforcing loop over here where the attacker has accumulated compromises, which increases their likelihood of success, which increases their success, which increases their success per attempt. And as that happens, the resistance of the system will gradually decline and the likelihood of success of the defender will then go down. The one on the bottom is actually also a reinforcing loop, interestingly, because you can see it's got two negative signs there. And so if the defender is more successful than the attacker, as the defender's success goes up, then the attacker's success

per attempt will go down. which will increase the resistance of the system, right? Because that's an inverse relationship, which will then increase the likelihood of the success of the defender. So you've got a couple of reinforcing loops. So this is a simulation model. I used Vensim, which has a free personal learning edition. So most of the modeling tools, and we'll go over those at the end, have free or low-cost versions available for academic or personal learning use. So this looks different from our causal loop diagram. The basic structure is similar. You can see there's a couple of loops. But items that are stocks are now delineated with a box. And items that are flows are

delineated with a valve. And those might have been abstracted away a little bit on our causal loop diagram. But you have to get a lot more specific with the simulation model, especially with units. You have to make sure all your units check out, right? So I did success per hour here, you know, for different... attempts at compromise. I have, I'm going to try to walk over here without blocking the slide, I have an impulse for initial compromise of a key account because initially maybe these are in balance, the attacker and the defender, right? But if you get a compromise of a key account that tends to shift the dynamics. And so then we start to

get this success loop running for the attacker, right? Where They have a key compromise. So their success allocation is probably going to go up, right? And their compromises are going to go up. Meanwhile, the defender's resistance is going to start to go down. Something I want to point out here before we move on is that the presence of stocks and flows can obscure plus and minus signs. We have them real clear in our causal loop diagram. I could put them on this screen, right? We could say that like, Okay, so resistance going up would increase the success allocation variable for the defender, right? So the defender would have more success and that would be like

positive relationship up there. But a lot of times when you actually look at these models, you don't see the signs or-- and especially you don't see them in relation to the stocks and the flows. Occasionally you do, but I-- like of the diagrams I've looked at, a lot of them don't have that. So don't feel like, "Oh, I use the shortcut and now this loop should be reinforcing but it looks like it's balancing." Make sure you actually walk through it to figure out how did the modeler set this up? Like, what are they using? It's usually not a direct one-to-one mapping to the causal loop diagram. There's other stuff in here like an impulse or

this is, I have a constant variable here called chance of compromise given opportunity. So there'll be other things in the actual model that were not in the causal loop diagram. When you... This is really cool. So how do you figure out which loops dominate? Well, we run the simulation. But there's one of the tools, which is called Stella, has a free online version. And they have something called loops that matter, which will change the heaviness of the arrows based on the underlying data that's in the model to show you which loops look like they're going to dominate. So that's a cool feature. This is a simulation output of that model. So you can see initially

they were pretty much in balance and then there was that impulse, the compromise of the key account, and you start to see that the resistance goes down as the attacker's success goes up. You can run a simulation once, you can run it 10 times, 100, 10,000 times, to determine the behavior and the outcomes. Now this is the kind of thing you would want to show an executive or a board member. You definitely don't want to show them this. No one really wants to see this, but this is very important, underlying the model. So you can show them things like this, or interface design. You can have sliders that allow people to change to change the

values in the model and see the results graphed out in real time. And one of the best examples of this, so I'm going to exit the slideshow again, and hopefully this is going to be quicker and easier than the last time.

This is En-ROADS. This is a project from Climate Interactive, and I think MIT is also involved in this. This is a huge climate change model that uses system dynamics. And they have a wonderful interface, which is why I popped it up here. You can change any of these sliders on the bottom to see what's the effect on the temperature over time. A few things that are interesting here that I noticed while playing with this, but I encourage you to go to this and use it and check it out because this is a really good use case for system dynamics. A lot of people like the idea of increasing nuclear, and I was shocked to see

that it doesn't make that much of a difference. So it would be interesting. I believe they are open to people emailing them about the underlying data and how the model works. Something else that seems interesting is that Energy efficiency of the building seems to be kind of a moderate leverage point Which I would not have expected again that it would have that big of an impact And of course the carbon price Is a huge? Leverage point but all of them have different impacts and so I encourage you to check this out as a good example of system dynamics in action with a super good interface because if you're gonna present to executives to managers It it cannot be the the VIN SIM model approach, right?

It has to be here's the output of the model. Okay, so So more resources for the curious. There are lots of system dynamics modeling tools. So Vensim PLE is the Vensim Personal Learning Edition. That is pretty well powered, considering that they don't charge for it. The professional version has some other features. But this is not bad. Stella Online has a free version that you can check out. There is a monthly subscription. Vensity is made by the same people who make Vensim. The website says it is free for personal learning use, but check it out because it then says something about academic use. So I think just have a look at that. Insightmaker is pretty cool because it's a web-based tool. So you set up a free account, and

then it is just free to use on the web. So if you want to practice system dynamics, these tools will let you do that. And if you want to get comfortable with causal loop diagramming, I recommend these resources. So this book, which if you're interested, I welcome you to take one of these copies, is a canonical introduction to the field. It's also really short and really readable. So it's just a good guide. Business Dynamics by John Sturman is huge. It's a thousand page textbook with modeling exercises. It's got a lot of calculus. Many, many case studies from business that really explore how have companies used this in practice over many years. And then the beer game, which we talked about, which is a

supply chain oscillation challenge. That's fun, just as a really quick and easy way to experiment with oscillations. So takeaways here, where should we... set the model boundaries. This is super important. When you're drawing your causal loop diagram, have you set the boundaries correctly? Did you leave out anything important that's going to affect the results? It's more important to capture everything important. It's also important not to put extraneous stuff on there. Like you don't need the procurement department in an information security model most of the time. That's almost never going to happen. But you've got to get all the important pieces in there. What stakeholder perspectives were not included maybe in the first round of interviews? Sometimes you'll do interviews and then you'll find

out that there's this other loop and maybe that means you need to interview somebody else that you didn't think you had to interview before. It's always it's almost always worth going to talk to them because otherwise you'll end up perhaps with rework or even worse wrong results. What leverage points should you aim to influence first and based on what the simulation output is telling you. So it may tell you that, for example, carbon price is a huge leverage point, but that's also a major pain and a major effort to get enacted. So what other leverage points are there, right? What is going to be palatable in your organization and the leverage point? I mean, one

of the reasons the growth leverage point in Limits to Growth didn't come to fruition quickly is because it was... not a good message to hear, right? Like people don't like to hear that maybe when they're seeking growth, they should be seeking, they should not be seeking, they should be actively not seeking growth, right? That's not a happy message. So what can you aim to influence that is going to give you Easy wins, right? And also, what should you aim to influence over the longer term? It's good to know what these leverage points are and start to talk about them. And you can use this in a lot of different ways in a lot of different

fields. But the important point is, when you look at this, if your data is good, if you've talked to all the stakeholders, if your model is good, you're going to get information that's not based on intuition, which is often wrong. It's not based on, we have always done it this way, so this is working fine, must be the best way to do it. And it's not based on, this is the thing of the hour, or the thing of the day, or the thing of the year. So this is the zeitgeist, and we're going to jump on the wagon. So you want to do this in a more scientific way. And this, I think, gives you

a good tool set to do it. You can use it for all sorts of things. You can use it for information security. You can use it for organizational modeling to try to understand the organizational dynamics that might be blocking something. You can use it for AI risk, right? You can use it for economics, right? There's a lot of use cases there. And I am not an economist, but I have heard it said that, you know, economics often has a lot of flows, but maybe not a lot of stocks. And so, you know, if a model is not playing out in the real world, are there stocks that are not maybe being fully considered? So... Thank

you. As I like to put it, there are some things I would rather simulate than experience. So I think this is a cool tool. It can be a useful and interesting approach and tool set. And thank you for letting me share it. If you have any questions, I'm happy to answer. Please take a book. If you have a question, please use the microphone. OK. Yep. OK. I should have brought more books. Couldn't carry them in the luggage. I do have a question. The book is cheap on Amazon. Okay, yes. Have you succeeded in gaining valuable insights into other areas of information security using this technique besides the ones you've suggested in your slides? Okay, so information security, yes. I am using it right now to

do some AI risk modeling of controls versus capabilities. to kind of see what are the leverage points for getting better controls, right? Or for making sure controls can stay on pace with capabilities. But there's... That's my current example. Also, I think it's really good for climate change modeling, but I think En-ROADS is a super good tool for doing that. If you get the business dynamics book, which I think you would find really super interesting, there's a lot of business case studies in terms of, especially supply chain management is really, I think, was the original focus here. So there are tons of case studies on that. But there are also other studies business case studies in that book that diverge from that and go into a lot of other fields.

So I would encourage you to pick that up because it covers just such a broad range of areas. Okay, I'm gonna do microphone first and then raise hands. Yeah, okay, go ahead. Yeah, I just want to ask about the level of abstraction looking at all these different pieces that you're looking at how some module or some variable relates to other parts of elements. In a practical aspect, I know that there's levels of granularity about how close to the project or how close to the ground level people are going to talk about their specific part of that of that whole view Where would you would you begin at the very top and then try to provide? levels of granularity down the love down the chains to

the actual important variables that you can measure and and show the correlations and assumptions Okay, that's a good question So you kind of want it you want to model what you need to model and no more than that, right? But you do need to get down to a level where you have data, right? so so I would say that based on your stakeholder interviews, you have a main model, right? But then you can also have modules, right, of the model. So I have seen some people have, I think, different tabs in Vensim, for example, for different parts of the model that allow you to kind of go to different levels, right? So you can say,

you know, this is my overall model. You're talking about like a map, right, where you would have a map and then you'd have like a city pullout, Right? To kind of look in a little bit more detail. Yes, I think you can do that. Right? Okay. Yeah. Mm-hmm. Yeah. Hello. Thank you for the talk. Can you talk about how you validate these models over time, especially when people may be wanting to change the model so it may not last very long before someone's found some new input they want to add, and then you're getting new numbers? ANNE KLEIN: Yeah. OK. So processes change, right? Systems are not static. So you might have a totally different

process for doing something one year from now than you do now. So the model is a snapshot of the process as it was at a particular point in time. And at a certain point, you do need to say, this is version X of this model. But you should continue to develop the model if it's proving useful in the organization, and then have other versions of the model. So NROS does this. There was a message that popped up when I first pulled it up. It said, "We've made a lot of changes to this model as of," and I think it was June or July. And so the underlying assumptions had changed. that they were using because

I think there was some new data that came in. And I would encourage you to do that if you're using this tool is to have versions. You don't want to be constantly changing one model because then your simulation conclusions are just not going to be useful over time. But if you can say, version one told us this, we changed that in our process, and now version two is telling us this, so then you see how that plays out. The simulation will give you how it thinks it will play out, what is the likelihood of how it will play out. And then reality will tell you how did this play out, and then you'll make further

changes based on that. But I think versioning is really important. - Thank you, and thank you for the books. - Yeah, you're welcome. Yeah. You're next, I promise. I found some interesting assumptions in your models, and I was wondering-- Is there a good approach to documenting the assumptions that you're making when creating those models so that when you're going back a year later to make a change, you can understand why you thought that the relationship between component X and component Y was positive or negative in their relationship to each other? Okay, yes. So documentation is always a great idea. I do not think... So the actual modeling tool does not have in the field where you set, where

you open up your formula, necessarily a place for extensive comments. But I absolutely agree that you, I mean, you can put comments on the screen. Nothing stops you from doing that. I can embed graphs on my screen that shows the model and I believe, you know, there's no--nothing that says I can't also embed some comments there. But I think you should have a good documentation process. for saying what is the rationale behind my model because especially because people leave people go elsewhere and you want whoever is there at a particular time to understand what the rationale is same thing as when you write code except this let's this is not as much in line as

that right so you would need to have a good documentation process that you set up yeah yeah Thanks. Let me see. So I was imagining with the original diagrams that were drawn out, you had the descriptions of if this thing goes up, assuming then the other one will go up or down. But I'm also imagining-- there's a lot to be said of like there's some things which have varied relationships like if it's goes down then the other one goes up but maybe if it goes up then it stays the same or there can be linear versus exponential absolutely relationships so I'm curious Is there much in terms of writing that out as like simplified even though

it's very complicated or maybe being able to at least have some applications that will simplify it and help the simulation be as accurate as you want it to be? Yes. Okay. So I think this gets into the granularity question, right? It's a different lens on the granularity question. So in the business dynamics book, which I guess is what I would recommend that you look at. John Sturman goes into, for a lot of situations, it's okay to model things like first order. For whatever reason, it works pretty well. But there are scenarios where that's not the case and you want to go second order or third order. Also, there can be curves that have a lot

of different shapes, right? So, yeah, something may go up exponentially for a while, and then it may be a plateau, and then it may pick up and take a step up, and then it might go linearly. Yes, absolutely, there can be systems that do that, and you can model them. That's more going to be on the simulation side. But the causal loop diagram I don't think really captures that very well because it's just your hypothesis. But yes, you can model all these different types of curves using this tool. And you should definitely get business dynamics. That's what you need. OK. Anyone else? Any other questions? Yeah. Another follow up there. Being able to apply some AI machine learning techniques that have come about of late, I'm curious if

you understand anything in the works or at least something that's like right on the horizon that basically can take in time-based data in multitudes and start helping you to plot out very plausible, at least based on the data, hypotheses for testing causality. I think it's a really cool question. I do not think that the system dynamics establishment has really used a lot of machine learning yet to change how the process is done. But I think it could be incredibly useful to do that. You're saying to generate causal loop diagrams, to identify leverage points. Yes, I think it's something that would advance the speed of how this works by leaps and bounds. But you still need to interview stakeholders. I think that's super

important to get the perspectives of everybody included, as opposed to just, it looks like it's likely that this is what their perspectives would be. But yeah, it's a really good point, and I agree that it should be integrated. Thank you, Stephanie. We really appreciate this.

♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ Oh, one of these is just me ♪ ♪ Baby, you're giving me wind and ♪ ♪ You're some guy ♪ ♪ Oh, I don't wanna jinx it ♪ ♪ Yeah, oh yeah ♪ ♪ I

really miss you, babe ♪ ♪ I say bye ♪ ♪ You're chilling in the rain ♪ ♪ You're some kind of butterfly ♪ ♪ Meant to be a guy ♪ ♪ You whip up my ♪ ♪ Don't leave me here, I die ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ Baby you can't hide. You're giving me wind. You're some kind of bird. Don't leave me here lying. Oh I don't want to jinx it. It's just sweet. Baby you're some kind of bird. But I don't want to jinx it. Yeah, oh yeah. I miss you babe. I don't want to jinx it.

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪♪ ♪♪ ♪♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ Baby you ♪ ♪ You're giving me wind and ♪ ♪ You're some kind of bird ♪ ♪ Don't leave me behind ♪ ♪ Oh, it's just me ♪ ♪ Baby you ♪ ♪ You're giving me wind and ♪ ♪ Oh, don't wanna jinx it ♪ ♪ Yeah, oh yeah ♪ ♪ I really miss you babe ♪ ♪ You're channeling wind and rain ♪ ♪ You're some kind of bird ♪ ♪ You whip up my ♪ so

Thank you, Ground Truth. And today we have Jonathan Lustas with Playing Games with Cybercriminals. Before we get started, I've got a couple announcements. We'd like to thank our sponsors, especially our diamond sponsor, Adobe, and our gold sponsors, PlexTrak, Toyota, and Conductor One. It's their support, along with our other sponsors, donors, and volunteers, that make this event possible. These talks are being streamed live and as a courtesy to our speakers and the audience, we'd ask that you check to make sure your cell phone is on silent or do not disturb. If there is time for questions at the end, I have a microphone in the middle there and we'll see whether you go up to get

it or I'll bring it to you. I'll plan to, if there are questions, go get the mic and I'll bring it to you. And with no further ado, Jonathan.

Well, it's a privilege to present this research to you. Before I start, though, I just want to acknowledge quite clearly that this is a team effort. So I'm presenting this research today. I'm the PI on the project, but there's a number of others involved here. So we have the co-authors in Eduardo Gallo and Federico Varese, but particularly I want to note the contribution of Rebecca Heath, who's done a huge amount of work for this project. So it's by no means myself presenting on behalf of myself. This is very much a team effort. Now what I'm going to talk to you today might be slightly weird as a type of presentation for a few different reasons.

One you'd note, there's an Oxford Cambridge collaboration which we're meant to be sworn enemies, so that's quite unusual. But there's no serious beef there and you want to work with good people, so we managed to do that. So that seems a little bit weird, but it's not that weird. Secondly, this is a very social science-y type of presentation, which might be a bit weird if you don't come from a social science type of background. And particularly, I'm a sociologist, but I'm drawing on another field, which is experimental economics, and so the collaborators we have have brought us into that space. So I'll try and be little bit gentle with that I have to learn that

myself so I'll be quite clear in those sorts of elements because it gets a little bit technical in its own kind of way but the third part that's weird which is I think the most interesting thing about this is that if you look at conventional economics a lot of this is about how do you make markets work more efficiently more effectively and the weird thing we're doing in this presentation is actually trying to think about how to make markets work less well so how do we screw them up How do we disrupt them? How do we make them less efficient? So we're doing the opposite of what economists, conventional economists would be trying to do.

Now, why would you want to do that? And the reason for that is not all markets are good. So we have bad markets or markets that we don't want to function so efficiently because they create harm of different kinds. So Cyber criminal markets, which are the inspiration for this project, fall into that category very much. And so they've really inspired the work that we're doing here. That's the background that I come from. As a sociologist, I spent a long time interviewing former cyber criminals, interviewing people in industry and law enforcement, trying to understand more about that industry, that criminal industry. And so here I'm talking to you about this one particular part, which is the

markets, which are very, very important to how cyber crime functions. So as we all know, cyber crime is a major burden for business. It causes a lot of trouble for a lot of people. But what is really quite central to this industry is markets because they allow people who do things like breaches or carry out other types of activities to monetize the data or to engage with others with different types of specialties from the ones that they have. So this is very important to finding friends who can do things that you can't do, right? Working with other people who have specialties and skill sets that you just don't have. And so the markets are very,

very essential to this. And they operate in different ways. We get very small ones and we get large ones that are thousands of members in them. And they range from those that specialize in certain types of areas, certain types of cyber crime, to others that are more general, some that are into more drugs and things like this. We get a whole kind of spectrum. But the essence there is you need a place to trade, to do business, to work together. So looking at the disruption of these markets is actually very valuable as a policy exercise and as a broader exercise in trying to understand how these markets work and how we can make them work

less efficiently. So we can think about that. So just as an example, some of you might be familiar with this. This is a kind of historical case now, dark code technology. quite a few years back now. But just as an example, if you're not familiar with some cyber criminal markets, what they look like-- and they look quite similar to a whole bunch of other sites, to be quite honest. We often look for things that are very unusual, very innovative in cyber crime. And they're not in a lot of instances. They use a lot of things we see in other aspects of life, other aspects of tech. Even the software that they use to create the

sites is very similar to all other sites. So here what we have, Dart code, you can see the little tag line there about being a marketplace for sewing machines and other stuff. The other stuff is like malware. Exploit kits all sorts of things this was known before it was shut down as being a more high-end more technical kind of English language site So we get other sites to specialize more in carding credit card forward things like this This had a little bit of stuff going on like that But it was known as being at the more technical end at least in the English language scene and so that was the place that you went as

that type of actor of where you wanted to find that the good stuff in terms of malware so you can see a little bit of a spread of some of the things that were on offer in this particular marketplace and So this is just an example. Again, posts look like what posts look like. They're not anything particularly special. What we have here, I'm not expecting you to read all this tiny writing, is just an example of what we see in these types of markets. And here what we have is one particular cyber criminal under the name J.P. Morgan, which I think is a fantastic example. cyber criminal name and actually was very very effective and

well-known cyber criminal. Eastern European actor, very very important cyber criminal in a number of respects and he's looking to buy exploits, that's what he's posting about. So he wants people to come and do business with him. The key here, how do these markets work? A lot of them work quite simply like this, which is you advertise, often you're selling or you're advertising to buy and you'll find partners that way. Some of them evolve to work in slightly different ways but that's the core of it. You advertise, you're looking for someone to trade with, and then you trade. That's as simple as that. Trust is a key component here. So we see a couple of people

jumping in in this thread, basically verifying JP Morgan as being a serious person. We can see down there near the bottom, Paunch, who's another big Russian-speaking cyber criminal, was arrested in Russia a few years back, who basically comes in and says, yeah, I know this person. He's very legit. So trust is important, and that's very important to trading in these types of settings. So as a social scientist, what I really want to emphasize here is the people involved, that we're talking about people. So we see on the left is probably the most widely used image of cybercrime that's in any kind of report that you might see. And so I'm including it here not as

an endorsement, but actually to criticize it a little bit, which is the main problem is that they all have faces. And this image does not depict that. So on the right, we have a real world cyber criminal. It's tied back to that Dark Code example I just used, which is this is one of the administrators of Dark Code. So his nickname is Asserto, and he's been arrested multiple times now. So he was also known for being one of the key people involved in the Mariposa botnet. And he went on to do a bunch of other stuff. You can see him there wearing a t-shirt because he, after his first arrest, went on to work for

a startup in crypto mining. You can see, even just in that case, a little bit of what we're talking about here. There is actually a strong similarity between some of these actors and regular humans. These are people. That's the point I'd like to make. These are people too. They're not so unique and so unusual that we think, "Let's reinvent the wheel. Let's think about them in a completely unusual way." No, they're 99% like other people, and they're 99.9% like other people in tech. because that's the skill set. And so we often see some individuals who are moving between spaces, sometimes in grey areas, sometimes moving between legitimate enterprises and more criminal ones. So here the

point is that if we're trying to understand more about these types of people, we can look at them through the lens, and as I said, I'm a social scientist, of studying humans. We don't have to view it purely as a tech kind of problem. All right, so how have we tried to deal with this threat so far? So we've been talking about these cyber criminal marketplaces, what has been the approach up to this point. In terms of conventional law enforcement, the strategy has been conventional law enforcement tactics, which has been around takedowns and arrests. So if we think about how do we deal with crime, if you want to get to the photo, for instance,

I showed you one just there of a photo, you ultimately have to arrest the person to attribute exactly who they are, right? So that's been the core of the strategy, which is, okay, we try and arrest these people when we can. We also try and do takedowns. We try and hit the infrastructure. So that might be in relationship to these types of cybercriminal marketplaces. We try and take those marketplaces out in different ways, maybe twinned with an arrest strategy going on together. Or if we're talking about botnets, we're trying to take out some of the botnet infrastructure. We're trying to hit really the most sort of visible and obvious aspects of this, and we're trying

to arrest the people involved. Now, the problem becomes... How effective can we be in this? Particularly when we're talking about cyber criminals based all across the world and sometimes based in jurisdictions where we don't have good relationships between different countries. So we can think about the example of Russia. If you're operating in, say, the US or the UK or somewhere else, and you have a cyber criminal who's operating out of Russia, can you get good cooperation at this point in time if you're trying to make an arrest, if you're trying to get that type of cooperation? Actually, the same would apply in Russia in relation to, say, Kazakhstan or somewhere like that. So everyone faces

a similar type of problem, which is this jurisdictional barrier that there is when you're trying to make a risk. So the question is, how much effectiveness can you have with this type of approach? The other part of it is, if you look at these types of takedowns of infrastructure, whether it's botnets or whether it's marketplaces, Do the actors just move? So you hit a particular marketplace, you shut it down, they set up a new one, and off they go again. Or if you take out the botnet infrastructure, if you haven't taken out the people behind the botnet infrastructure, they're just going to set up a new infrastructure. So there's this kind of question about, is

this sort of a whack-a-mole type of situation? Obviously, there's very strong reasons why law enforcement goes in that direction. But the question is, are there this issue of what we call displacement? which displaces either in time, so people stop for a short period of time and they restart again, or it displaces in space, which is they move somewhere else, or even move into a different type of activity. So that's something we need to be aware of. So part of the core of what we're trying to do with this project is understand, are there other types of approaches we might adopt that are less hammer-like, that are less strict, less strong, less conventional in terms of

law enforcement? Are there softer and sometimes cheaper approaches in terms of not requiring a massive operation that crosses jurisdictions, that involves a huge amount of attribution, huge amount of arrests and these kind of things? So that's what we were kind of inspired by. And the question we asked is can we play games with cyber criminals? In a sense, can we mess with the marketplaces? Can we inject some kind of distrust there? and how do we go about doing that? That was the core motivation that we adopted here. Is there something, not necessarily to replace these existing law enforcement strategies, but something you might supplement them with? And so that's what has been driving our work.

Okay, so this particular project has two questions, which is how do cyber criminal actors in online networks cooperate and trust each other? So we've talked about that question of trust quite a lot already. And then how can these networks be disrupted? So what were the methods that we used? And this is the part I mentioned as being slightly weird, so I'm going to try and introduce them to you because I'm not expecting many of you to be experts in experimental economics. And as I mentioned, I'm not really an expert in experimental economics either. So I'm going to do my best to try and explain it to you in a way that people can understand, in

a way that I tried to understand it myself. So this is actually the first time anyone's used this type of approach in relation to these markets, to my knowledge anyway. I'm willing to be corrected on that, of course. And so what we looked at was to design a market very similar to what we call a market for lemons game. So if you're not familiar with what market for lemons is, if you think about a used car market, that is the most famous example, which is if you're selling used cars, you know a lot more about the particular car or cars that you have. And if you're buying them, you don't. And you're in a bit

of trouble because you have what's called an information asymmetry. the seller maybe knows they're selling you a lemon, the buyer does not know. So you might think, okay, there's ways they can figure it out and things like this, but just on face value, in that interaction, one side has much more information than the others, and that's a very dangerous position to be in as the buyer. But it's also a dangerous position for the market. Because the theory is that the market like that will collapse over time. It just won't work very well. And so it will just spiral down. So what we see here is the way out of that problem is things like reputation.

So there's various mechanisms that have been developed over time to try and solve this problem. So you get like branding, licensing, regulation. And reputation is very, very important for trying to solve this type of problem. If people know that particular seller, that particular vendor is good, I trust them, I trust the product. then you're more likely to buy from them and the market won't collapse in the same kind of way. So there have been a number of Market for Lemons games that have been experimented with. And what I mean by that is experimental economics, what we're really talking about here is a type of game theory. But we're not talking about the highly mathematical or

the modeling of the game theory. We're talking about getting humans to actually play games and see how they play them, see what shakes out. So what decisions are they actually making, rather than just trying to come up with a model of what decisions we think they would make. So that's the point of this. So we took some of those off-the-shelf games, we looked at that, and then we built our own design to see how we would play around with this to get to the key interventions we were interested in studying. So I will maybe, it might be a bit of a letdown, but I'm going to say that we aimed at a broad kind of

approach at first, because this is the first time we're trying to do this. One of the temptations we had, and it was a temptation I really, really strongly had, was to make this as realistic as possible, to get everything you could find in terms of how cyber criminal markets look, like the ones I showed you, make something that looks like that, give them, let's play this for six months, let's see how long we can do this, let's track this for a really long time, all this kind of stuff, build in as much realism as possible. But I was cautioned, and correctly I think, by those who had more expertise in the area, which was to

be very, very careful about how much noise you built into the experiment, right? Which is the less elements, the less variation, the more confidence you can have in that one particular variable, one particular factor is driving a change of one kind or another. So if you're trying to understand what interventions might succeed in making these markets work less efficiently, you want to have a high degree of confidence in terms of this is the only variable that we've changed. and there's not 15 others that we need to pay attention to. So that's what we did. And this I view very much as a first step, and we're looking at ways that over time, in a much

more coordinated way, of building in some of these extra variables. So this built on an earlier attempt, a small pilot that we ran in a lab where literally people sat in a room like a classroom and played this on computers. And then we moved into what I'm presenting to you today, which is an online experiment where you can have people sitting at computers in their own home, playing the game. And this makes it much easier to recruit and to engage with far more participants than if you're just requiring everyone to turn up to a certain place. And it also means you can engage with different types of people as well. Okay, so the experimental design

was basically broken down into a series of mini markets. So again, rather than going with, okay, there's a market with thousands of people in it, it was, okay, let's build this up in terms of what we can have confidence in. So we ran these mini markets. So each group was basically one such market. So there's 56 of them. Each one had four sellers and three buyers. So ultimately, you're one of the buyers in there. You have a choice in each round to buy from one of the four. You've got some options there. They play this game over 20 rounds, so there's 20 potential transactions they can have if they want to buy and sell. And

then we're going to watch how that plays out. And ultimately, there was 392 participants in this experiment. So what we did was split them up into four sessions. We call them treatments. But they're basically sessions. And in each of those, there was 14 groups. And so the idea here is rather than put all the interventions we're thinking about testing, just smash them together, which would lead to that type of noise I was talking about, you actually want to test them one by one, right, and test them against what's effectively a control to see which ones are actually having the impact that you want or not. So that's how we went about structuring this. The participants

in this particular phase are recruited from Amazon Mechanical Turk. If you're interested in why we did that and why you might do other things, I'm happy to discuss that in question time, but that's one of the ways people engage, one of the participant pools people use in these type of online experiments, but there are others as well. Okay, so this is what the game structure was. So this is the seller side. So you're a vendor, you're a cyber criminal vendor effectively, you're trying to sell a product. What are the steps that you go through when you're in one of these mini markets? So you're one of the vendors in this group of seven, what do

you do? So you have the option here of producing up to two goods, two units of a good, and there's two types of goods. So you have regular goods, which we can view as the poorer ones, and then super goods, which are more valuable, they're better. They're better ones. Those are the ones you really want to buy, right? So they have the choice of saying, okay, I want to produce two super goods or two regular goods or one of each. So not one of each, but just one regular or one super. You couldn't do one of each in this particular experiment. Or zero goods. And then they could advertise. But the ads do not have

to be truthful, and that's where all this kind of deception comes in, which we see in real cyber criminal markets and what we're trying to understand, whether we could push that deception more, get it happening at a higher level. So in this sense, you could advertise in a completely untruthful way. So you could say that you are selling two super goods, but you're really selling one regular good and things like this. So that's the key to this. The seller's production decision is private, so no one knows this other than them. That's the market for lemons part. And ultimately they get a choice of how they want to price this. So they can price between 1

and 200 points. And ultimately the seller can default, so they can not produce a product. They can not send the product that someone buys. And that's the worst thing that could happen to the buyer. Not only is it potentially a poor product, there's just no product that's provided. They buy the product and the product does not come. So this is what it looks like in the interface. So this was coded in O-Tree, which has become a relatively standard platform for doing these types of online experiments. And here you can see basically what it has outlined to you, which is the decisions that a seller can make Within this game so they can produce none all

the way up to producing two super goods They then have the pricing option in terms of points and then they can decide how they want to advertise and whether they want to advertise truthfully or whether they want to deceive in terms of not being quite honest about what they are producing here and Okay, so this is the game from the buyer side. Now, as we mentioned in a market for lemons game, the buyer doesn't know very much, but they do know some things. So what they do know is the advertised quality. They know, you know, and so this is basically what the ad is. They know what the price is because that's also been advertised.

They know in this game the identification letter of the seller, so they can track them over the course of a number of rounds. So they can say, oh, I know J, or I know A, or I know Z. I did business with them two rounds ago. That was a good interaction, so I'm going to go with them again. So they know that. And then they get some information, which we built in there to replicate the kind of reputation mechanism as it exists within cybercriminal marketplaces, which is effectively around... the average rating over the previous rounds and also the last three ratings. You're getting some sense of a track record, you're getting some sense of what

you want to know really beyond knowing the actual product itself. You're knowing a little bit about the seller. They don't know the most important piece of information, which is what is the quality of the actual good that they're buying. That's very, very important to them. At the end of each round, that's the information that they learn and then ultimately, The last and very important part of this is they then get to rate. So they get to say whether they like the transaction. So whether they got the product that they wanted. So they paid for a super good, they got a super good, and they're going to say maybe give it a high score of five.

Or maybe they had a terrible interaction, they're going to give it a one. So we know this is very familiar to how we do things on the internet, how we do things in life in terms of rating transactions. So this is the same thing. This is the way we're trying to capture reputation within this particular game. So you can see here on the left, Purchasing side so on the buyer side what that decision looks like it's again You can see the options here in terms of the different products are being advertised you can see something about the the ratings so you can see the question marks there is because it's only trading round twos or

Data doesn't exist yet So the question marks are listed for those rounds that haven't occurred yet, and then you can see the average rating overall So in this case they've gone for a slightly risky option, so they've gone for the cheapest option because it's 20 points and that's nice and But ultimately, that person's average rating is one from the first round, so maybe we won't trust this person. But they've gone for it, so they've gone for a bit of risk to see what they can get out of this. Okay, so what matters here, and it's a very important point, which is you actually get paid to play this game. We paid people to play this

game. It's very expensive to run these types of experiments. I don't recommend it unless you've thought about it a lot, because otherwise there are very costly mistakes that come into this. That's why we have to think very hard about the design of this and about how much we were building into this, because it really means something when you blow the whole budget and then it's gone, because you can't actually repeat it if you don't have budget. What happens here is people get paid to play the game, and that creates an incentive. And so going back to the point about cybercriminals are people too, people like money, cybercriminals like money, we're just basing it off that

core thing, which is if you're playing a game, you want to make more money rather than less money. And so what we're talking about here is we said the participants are not cybercriminal participants. That's a variation you could build in later once we had more certainty about this game, try and find people who maybe used to be former cybercriminals, as I'll talk about at the end of the presentation. Maybe there's ways of doing this out in the wild, where you could learn more about how this works in the real world. But ultimately for us, we were just going off the core profit motivation of the people who were participants in the game. And these are

people, as I mentioned, who were from MTurk. And so this is the business. This is how they earn money in a lot of cases, by doing tasks on MTurk. And so we provided them with one such task. Now, it's very important within experimental economics, not only do you pay people, but you pay them at minimum wage. So the idea here is that even if you're terrible at the game and you really lose quite badly, we still have to pay people minimum wage. So that's, you know, you have to pay for the time that people are giving to the experiment. So there are things that you do in terms of kind of in some sense cleaning

the participant pool of trying to make sure that you have a good group of participants before you recruit them. They do a survey, do some tasks. You're basically trying to verify that you've got people who are taking this task seriously. So a lot of these elements that kind of go on the scene, but they're very important in terms of how you go about doing this. So ultimately, that's the key. They get paid. They get paid at least minimum wage. But if they play the game well, they get paid more than minimum wage. And that's the whole point. So they can earn more and more if they do more. But they're not going to earn like

a million dollars because we did not have that in the budget but they're gonna earn something and they're gonna earn a good a good wage for the amount of time that they're putting in so ultimately the payoffs then become built into the game so the better you do with the game you more you get paid so here we can see how that works so we have the the production decision which is basically on the on the seller side you want to advertise products you want to produce that product that product costs you something to produce and on the buyer side it's worth something when when you buy it. So we can see that the payoffs,

the buyer payoff is the value of the received product, so the received good minus the vendor's price. So if we go to say a regular good on the received good, that's worth 30. Say the price was 20, so they make 10, right? So that's the idea. And on the seller side, the payoff is sales revenue minus the cost of production. So say again, we go to a regular good, they advertise it for 20, the cost is 10, they make 10, right? But you can see they can go endlessly in different directions in terms of how they do this. The way that then converts into real world money is that 100 points equals $2. There's more

complexity in terms of the payment structure and how this all works, which I'm happy to talk about if you want, but that's the core of it. You can see there that obviously there's a gap between the cost and the value, and the point of that, that's the economics, right? People deciding how they make money, whether they're going to be, in this case, making money as a seller. That game looks different than if you're making money as a buyer. You don't get to choose. You assign these roles, but you can see the payoffs and the calculations are different in that respect. Okay, so now we get to the interventions, which is what we're actually trying to

do to mess with these markets. So this is the games we're trying to play with these people. And so what we did here was take from the existing literature, so people had written about how would we disrupt marketplaces in terms of injecting distrust? What are the particular tactics we would use to do that? And so that's where we tried to draw those from to test what had been talked about conceptually, but hadn't actually been tested in any way, really. So that's what we did. So we took, and I'll explain them in a second, the slander attack and the Sybil attack were the two that had been discussed most widely in the literature. We could talk

about others, and actually we're very keen to explore others. So these, as we'll go through the presentation, we might think are not particularly good attacks for different reasons. But this is the ones that have been talked about. So before I get into that, We had a baseline, which is basically a control, which we have no intervention. So we talked about these different sessions, these different treatments where we ran. So one of them runs with no intervention whatsoever. So it's the game as I mapped it out. They play the game, and that's it. There's no intervention on it on our behalf Then we get into the three other treatments the three other sessions where we make

those interventions happen But we do them one by one so instead of as I mentioned just grouping them all together We split them out so the first one is the the slander attack so that in the literature is talked about as basically what you might expect it to be if you're a buyer You know you buy a product and then you just start slandering the seller So if they sell you a good quality product you say is terrible or you could do the other way around as well depending on what type of deception you're trying to create. But the idea there is it's really like a buyer side attack. You're trying to hit the reputation

of sellers by providing inaccurate information. So in our experiment, what we did was add a 20% probability that each rating from a buyer was replaced with a different random rating. So They are doing going through this process as I mentioned the last step as the buyer as you write the transaction once it's gone through so we're providing that extra added bit of noise in terms of Ruining some of those ratings they were making those ratings not what they were put in to be originally the civil attacks a little bit more complicated In its purest form, it's really the idea of undercover agents coming into a market and then really trying to flood that market with

poor quality product or defaulting much higher rates. There's more an external type of version of that as well, which would be trying to intercede in certain ways. So the transactions are happening, you're kind of outside in some sense, but if you can put yourself between them and cut off those interactions, cut off the transactions. So it's easier to examine that in a more physical sense. If someone sends a package and that package is intercepted, then you can block that transaction, right? So The receiver may not know why that package didn't come, but they will know but they'll suspect and they'll have maybe negative feelings about the seller as a result of that. So the Sybil

attack in our experiment was adding a 20% probability that each good did not arrive independent of the seller's decision. And what I mean by that is that the seller can default, as we talked about. So the seller might be ripping them off anyway. So we're just increasing the likelihood of that default, of there being no product that's provided. So they buy the product and nothing comes, which is a fairly common scam that some cyber criminals carry out against victims, but also against other cyber criminals. At the end here we have the combined treatment, which I mentioned before about not wanting to create noise, but here we're actually trying to do it in a more systematic

way of intentionally putting them together, having separated them. The idea here is we want to see if there's interaction between these two tactics. If you put in the two interventions at the same time, will it actually create something else that doesn't happen if you just have each of them on their own? That's the the basic design in terms of the interventions. We're taking what has been discussed within the academic literature and we're trying to understand how that might play out to test it to see if they work or not. Maybe they work, maybe they don't. Okay, so I'm going to walk you through here some key findings. I'm not going to walk you through this

table, but that's an illustration of some of the underlying work of which there's much more, but I'm just going to pull out some of the key takeaways for you that may be of interest here. Out of these different interventions that we looked at, the Sybil attack reduces seller earnings by 43%, which is quite substantial. And it actually gets more substantial than that, which is in the last 10 rounds, it decreases the seller earnings by 63%. So what we're seeing there is... quite useful and interesting because what we look for in these types of experiments is basically what's known as a learning effect. And so you think, okay, over the first part of the experiment, people

are trying to figure out how to play the game, how it works, how they're going to do it well. The second 10 rounds, the last 10 rounds, they've learnt the lessons. And so what you're seeing there is the behaviour that's been impacted by the types of interventions that you've made. So in this case, what we see there is the Sybil intervention had quite a profound impact, particularly in the second half of the experiment. which is nice to see in terms of looking for something. And if we're thinking about we're trying to disrupt the business of cybercrime in some sense, by looking at disrupting the earnings, that is the core reason that profit-driven cybercriminals are involved

in this, that's a pretty nice thing to be achieving. So that's potentially positive. I'll get in discussion a little bit some of the complexities around that, but that's a potentially interesting finding. Now, we looked at a number of sub-findings that kind of connected to this core finding. And so one of them is around the increase in buyer inactivity. So we actually saw that this means basically buyers are not buying as much. So they get discouraged and they stop purchasing as many products. And so what we see there is actually across all the treatments, the three interventions, we saw some impact there. But particularly within the Sybil and also combined, we saw that inactivity increasing by

15%, which is quite substantial. And so we think that that is actually what's driving this loss of earnings. And I'll show you a figure in the next slide which will map this out in a little bit more detail and clarity. So overall, what we saw there, this third finding is the Sybil attack reduces the proportion of regular goods purchased. It was quite surprising to us. We talked about the super goods being the better quality goods and then these regular goods being the kind of average or worse quality goods if there's only two types, was that the super good market actually held quite strongly. So people didn't flee from that market. It seems almost as if

they stopped trading as much, they stopped buying as much of these regular goods, but if they were going to buy, they wanted to stick with the ones that they knew may

Ground Truth

Related talks