Do Not Build The Torment Nexus

Good morning. Thank you everyone for getting up so early in order to hear my rantings. I really appreciate it. My name is Eva Galperin and I'm here to tell you not to build the Torment Nexus. >> [snorts] >> You may be wondering who the hell is that? So, I'm the director of cybersecurity at the Electronic Frontier Foundation. How many of you here have heard of the Electronic Foundation? Excellent. Like a reasonable number of you. For those of you who are not familiar with the Electronic Frontier Foundation, we are a digital civil liberties organization that has been around since 1990. So, we've been around for as long as the web, but not as long as the

internet and we know the difference. And [snorts] also, we are a very successful t-shirt and sticker company. If you have been to a a hacker conference, there's a very good chance you have seen EFF t-shirts and stickers. We are the closest thing DEF CON has to a religion. >> [snorts] >> You can reach me at eva@eff.org, the easiest to remember email address in the history of time or you can find me on Blue Sky or Mastodon where I am on hackerderm. Notice no Twitter/X cuz that place sucks. You may be wondering what is the Electronic Frontier Foundation. In addition to selling t-shirts and very cool stickers, we have we are a Voltron of three different kinds of of employee. We have

lawyers. We have an entire floor of angry attack lawyers and what they do is they look for threats to digital civil liberties and they file lawsuits. We do impact litigation and the whole point of impact litigation is you file a lawsuit that's not just going to help the people that that are being represented in the lawsuit, but are going to have, you know, impact beyond that that we think is going to make a big difference for civil liberties. We also have lawyers and policy people all over the world, so not just in the United States because we know that the internet is global and we have to pay attention to all of it. Weird and unpopular view in

the United States right now. >> [snorts] >> I and we have people who live in all over Europe and in South America and Asia. We also have activists because sometimes the way to deal with a with a problem with a threat to digital civil liberties is to get people out in the streets or to sign petitions or to contact your your local your representative and tell them like, "Hey, don't vote for that or hey, vote for this other thing or this law is good or this law is bad." We have a lot of that. And then finally, we have the team that I work on. We also have a team of very angry attack engineers.

And our engineers are sort of divided into two parts. We have engineers that build tools that we think are useful for the entire internet. For example, if you have ever used Certbot in order to have, you know, SSL on your website with a single click for free, you're welcome. I'm not the person who does this, but but I work next to them and they're very cool. Or if you have ever installed the Privacy Badger browser extension in order to eat cookies in order to make it more difficult to track your your web browsing, also you're welcome. Still not me. But definitely people I sit next to who are very cool. Uh, if you have ever used

Ray Hunter, which is a tool that we have recently put out, uh, that helps you to detect the presence of, uh, of, uh, sort of fake IMSI catchers, a thing that we, uh, that allows you to, uh, catch the, uh, people's like text messages and stuff, in or fake cell towers, uh, which we think governments are using in order to spy on people, say, at protests. Uh, you're welcome, still not me. Uh, and so you may ask, "Eva, but what do you do?" What do you do? I yell a lot. Um, but I, uh, in the same way that our lawyers are attack lawyers and our engineers are attack engineers, I am an attack yeller.

Uh, and I have spent, uh, many years working on a couple of different projects. Uh, mostly I work on finding ways to protect vulnerable populations online. So, uh, activists, journalists, survivors of domestic abuse, um, LGBTQ populations, people who live in authoritarian regimes, people who live in rising authoritarian regimes. So, I get to work from home now. So, I'm very busy and I'm never going to run out of work, let me tell you. Um, so, you may have noticed that I have titled this talk, uh, do not build the torment nexus, in case you're wondering what my stance on the torment nexus is. But, it occurs to me that not everybody is extremely online. So, in case you are

not extremely online, here is what I am referring to. This is a joke posted by, uh, Alex Bleckman, uh, who used to write for The Onion, uh, and he writes, "Sci-fi author, in my book I invented the torment nexus as a cautionary tale. Tech company, at long last we have created the torment nexus from the classic sci-fi novel don't create the torment nexus. And obviously this is sort of he is making a point about the tremendous illiteracy of the people who are running tech companies in Silicon Valley right now. Probably most notably Peter Thiel who named his vast surveillance company Palantir after the spying globes used by the bad guys. Also that if you have ever made it to

the end of the books or watched the third movie didn't work. Anyway, I think you shouldn't do that. So what you should know about the torment nexus, what I think is the most important thing to know about torment nexus is aside from making fun of Peter Thiel, which honestly too easy. >> [snorts] >> Is that the torment nexus doesn't always come with a sign that says hello, I am the torment nexus. Sometimes we build the torment nexus by accident. We build it by going with the flow and failing to think about the ways in which we are building the things we're building will be used in the future. So we're going to talk a little bit about the world that

AI has built and some of the ways in which the tools have caused damage both on purpose and by accident and in ways large and small. In this review of AI's sins we're going to start very very very small. We're going to start with the sin of annoying me. And it starts with an email from EFF's press director Josh. >> [snorts] >> So my co-worker sends me an email with a quote from me in some like news newspaper, news blog, whatever. And it sounds very kind of like boring and neutral. And he says, "So, did did you do this interview? Did you did you talk to this journalist?" And I said, "No, I have never heard of of this news

organization in all my life." And I didn't say that because let me tell you that quotes from me are punchier than that. Thank you very much. And I was a little bit worried. And I started getting messages like this from our press director once every few months. And obviously, I'm not special. So, this started happening and was happening to other people at EFF. And basically, once every few weeks, our press director would find a a Google search hit for a quote from somebody at EFF that we never said. Usually having to do with some sort of news article that is, you know, on a content farm somewhere. Often from a journalist who doesn't exist. And obviously, this this is all built by

AI. And [snorts] we started to find it really worrisome. And so, Josh kind of wrote it up. Eventually, this progressed. And obviously, talking in the past tense, but this absolutely still happens. Um Beyond misquoting me, which as we know is a shooting offense. Um I just wanted to bring up our most egregious example, which is a a quote from in the Arabian post, which doesn't exist, from privacy advocate Lynn Wind at the Electronic Frontier Foundation. Lynn Windrum remarked that the community monitoring tools are playing a civic role, though she warned of the potential for misinformation. So, really really boring adenoidal quote. But, the most important thing to know about Lynn Wynn is that she also doesn't exist.

There is no Lynn Wynn at at EFF. And in fact, even though Wynn is an extremely common Vietnamese name, there is no Lynn Wynn sort of in our immediate kind of community of privacy and security activists. So, this is a person who is completely made up. >> [snorts] >> So, I brought this up in a talk last year. And someone smugly told me, "Isn't having quotes misattributed to you a good thing? Having false information about yourself out there makes it harder to know what's real, and that protects your privacy, right?" No. No, it doesn't. Because context matters. And in this context, I am an infosec professional whose job it is to talk to the press and

give my employers' positions on relevant topics. It's important to my work that when people see a quote from me in a news story, they know it's me. And that the words coming out of my mouth are nuanced and accurate and reflect EFF's positions. So, not only does this annoy me, but also it really undermines the work that I am doing, the work that my coworkers are doing, and the work that EFF does. All so that somebody can get some hits on, you know, on their content farm. Oh, but the annoyance intensifies. >> [snorts] >> So, I don't know if you have heard this story, but there is a product called Grammarly, which does a sort of like

it reviews people's text and then provides edits. This is really commonly used in sort of in writing, in journalism, by academics. Sometimes people at EFF run the Grammarly texts. We run like our blog posts through Grammarly every once in a while. Or more importantly, the kind of text that we post to SSD, which is our privacy and security guide. And the reason why we're particularly careful with our privacy and security guide is not because we're obsessed with having perfect grammar, because perfect grammar doesn't exist and all languages are malleable. It's because it's really important that we use simple and clear language for non-technical people whose first language we may not be speaking. And in order to make sure that we get,

you know, very complicated and nuanced ideas about digital privacy and security across. Sometimes when we write blog posts, we are writing blog posts to ordinary people, but sometimes we're writing them to policy wonks. And you you know, you sort of use policy wonk language and things get very complicated. You use a bunch of stuff that people don't don't understand. But this is the opposite of that. When we need to make sure that things are like really clear and and simple and difficult to misunderstand, >> [snorts] >> then you use something like Grammarly. So Grammarly rolled out a new feature and I use that term in quotes. It rolled out a feature called expert review that offers to give you

suggestions inspired by experts. It uses AI to choose the experts. Then it uses AI to presume presumably trained on our writing and I have worked for EFF for almost 19 years. So there's a very large pile of my writing out there. Um I and it presumably is trained on our writing in order to extrapolate from it uh to give the writing advice it says we would give. >> [snorts] >> I was mostly annoyed because I didn't think that the writing advice was any good. Uh my coworker Thorin uh ran a blog post about uh Google Play Protect uh through Grammarly and it came up with advice from me. Uh he was really surprised and he called me and he said, "Do you know

Grammarly is giving me your editing advice?" I said, "I don't know. Show me what it is." I and as you can see here, I it calls out the uh a part of his uh of his blog post which says advanced protection also protects you from disabling certain core security features that are enabled by default like Google Play Protect and Android safe safe browsing which safeguards against malicious websites. And all of this AI is used simply to tell you that I would say, "How might a brief nod to the new surveillance tactics prompting these app tweaks underlying their urgency? Tying each update to a concrete risk reinforces your vigilant stance." Let me tell you, this is

absolutely not the advice that I would give. Thorin, of course, is familiar with my writing advice. It usually goes something like this. Have you read this article which has something to do with the thing that you are writing about? Or have you considered throwing this entire blog post into the sea? So, as it turns out, I was not the only person who was mad. Obviously, Bruce Schneier, who is also uh one of the suggested editors, uh was not a fan. But you know who else was not a fan? >> [snorts] >> Journalist Julia Angwin. She was so mad, she spearheaded a class action lawsuit. Uh the legal filing alleges that the tech firm misappropriated the identities of

hundreds of writers to drive profits for its paid subscription service. Uh though it does also make the point that the edits were not any good, uh describing the uh imitation editor as a slopperganger, which I think is a great word and I'm going to use for everything. >> [snorts] >> So now now we're going to scale things up a bit. Uh we have uh we have pissed me off. >> [snorts] >> Uh we have pissed off a large number of writers. Uh and now we are going to piss off the entire town of Verona, Wisconsin. But in order to understand why the entire town of Verona, Wisconsin is mad, you must first ask an important question.

Mhm. There we go. What the flock? >> [snorts] >> So Flock Safety provides ALPR technology to thousands of law enforcement agencies. ALPR stands for automated license plate reader. So these are cameras that are located in a bunch of different places, and they scan every car that goes by, and they take note of it of the car's make and all kinds of information about the car. But most importantly, they note its license plate. And then of course, because you know the location of the camera, you know when that there that the car was there at that time. If you have enough of these cameras located to say all over your country, I don't know, thousands and thousands of

cameras then you throw all of that information into a single database, and you sell access to that database to various law enforcement agencies. And what these what this allows the law enforcement agencies to do is to query the database directly and get an enormous amount of information about what cars were located where all over the country. Now, you may have noticed this works better if you have more cameras. So, the data is paired with time and location. It's uploaded to this massive searchable database. And Flock Safety encourages agencies to share the data that they collect broadly with other agencies across the country. Uh Recently, I think they were they were uh They nearly ended up having a sharing

agreement with ICE and only an enormous amount of outrage managed to stop this. And I'm not even sure that we have stopped it for very long. Uh so, for years, EFF has argued that this constitutes mass surveillance and is a violation of the Fourth Amendment of the Constitution, which prohibits mass surveillance, unreasonable search and seizure. Uh so, using public records requests, FOIA, EFF obtained data sets representing more than 12 million searches logged by more than 3,900 agencies just between December 2024 and October 2025. Um The data shows that the agencies logged hundreds of searches related to the anti-Trump protests in February, such as the 5501 protests, the hands-off protests in April, and the No Kings protests in June and October, as

well as other protests in between. Just in case you think they're only targeting anti-Trump or anti-ICE protests, we also found a whole bunch of searches related to uh rights protests and uh and activists. So, uh that's particularly interesting. Uh for the animal rights activists, the knowledge that their vehicles are being tracked through a national surveillance network undeniably creates a chilling effect on their ability to organize and demonstrate. >> [snorts] >> Um the United States is a is a very car-based culture. And if you ask people how they got to a protest, most of the time they're not going to tell you that they took their bike or that they took the bus because, honestly, the bike is

too far and there's no bus. I And so, chances are if you went to a protest, you drove.

Which brings us to DeFLOC. >> [snorts] >> So, uh this is DeFLOC, which is a an open-source project mapping the license plate readers all over the United States. I As you can see, there are almost 90,000 cameras. Uh these are the license plate readers that are that are mapped all over the US. Um but, more interestingly, you can see that there are 57 cities rejecting uh these cameras. So, these are cities that had um that had um deals with FLOC that chose to either cancel their subscription or cancel their relationship with FLOC uh or simply chose not to renew uh their relationship with FLOC when uh their their FLOC deal uh expired. And obviously, FLOC's not very happy about this

because, again, what you're really selling is not so much the ability to tell, you know, what is going on in a single jurisdiction, but the ability to query uh the location of uh of cars all across the United States. So, again, the more cameras you have, the more powerful you become. And a movement to rob Flock of cameras is a an attack on its very value proposition. But deflocking is harder than it looks. Uh Verona was constant chose not to renew its contract with Flock and repeatedly asked for the cameras to be taken down. Flock declined to do so. The city resorted to covering the cameras with trash bags so that they would stop collecting data.

And I think that what you what you can learn from this are are two lessons. One is that once you have built the infrastructure of surveillance, scope creep begins. Uh it will be used for trivial reasons. Oversight will erode. Uh one of the things that we saw from our uh Freedom of Information requests showing what the queries were that uh governments and law enforcement were using against the Flock database was not just that they were running queries having to do uh with protests, but they were running queries having to do with immigration, having to do with abortion. Sometimes the queries were so incredibly vague that they were completely useless. They were queries, you know, there were

queries in which they were looking for things all across the country presumably for a case that was extremely local. Um we saw uh stalking. So, we saw uh police running queries on uh on their partners, uh which is illegal. I and also we saw queries where the um the reason for the query was simply given as period. That's it. I don't have to explain myself. Uh and, you know, silly me, I think the police actually do have to explain themselves. >> [snorts] >> Um so, oversight will erode and you will get queries that read period. Uh more and more people will get access. And once you have built the infrastructure of surveillance, it's very difficult to

dislodge. You have to physically tear that down or it will just keep collecting data. Which brings us to the next biggest scale, uh Salt Typhoon. I am guessing that most of the people here are familiar uh with Salt Typhoon. Like, hands up if I have to explain what Salt Typhoon is to you. No? No? Oh. Oh. No hands up. This is weird. Okay, so Salt Typhoon is a major Chinese state-sponsored cyber espionage campaign. I It was discovered in uh late 2024. Uh it seems to have been running since at least 2019. Uh it infiltrated uh major US broadband and telecommunications providers. Uh hackers accessed systems at companies like Verizon, AT&T, and T-Mobile for over a

year, uh specifically targeting surveillance data, call records, and the private communications of high-profile government officials. Uh attackers accessed uh highly sensitive systems, including lawful interception wiretap infrastructure, uh to steal call records, metadata, and the private communications of potentially millions of Americans. Why were they able to do this? The attackers gained access to infrastructure designed to comply with the Communications Assistance for Law Enforcement Act, CALEA. CALEA mandates that telecom providers build in surveillance capabilities, allowing law enforcement and intelligence agencies to legally intercept traffic. Uh these systems contain a central high-value target for adversaries, essentially acting as built-in backdoors that when compromised allow attackers to monitor anybody. >> [snorts] >> So, instead of just stealing the data

from one user, Salt Typhoon accessed the keys to the kingdom, allowing them to monitor data intended intended only for legal, government-approved surveillance. Um just in case you think, "Aha. Aha. You know, foolish America. This didn't happen to anybody else. The campaign targeted over 80 countries including the UK, Canada, Germany, and Japan. I don't [snorts] think I saw the Czech Republic on the list so you can actually look a little bit smug. Um So just in case you still think that covering your country in cameras that the government uses to spy on people in order to control descent by which I mean fight crime is a good idea. Let's talk about Iran. So the Iranian government installed tens

of thousands of cameras in its capital in response to the recent waves of protests. Specifically the most recent wave in January when massive nationwide demonstrations ended in a bloody crackdown that killed many thousands of Iranians. Um Israel spent years hacking Tehran's traffic cameras and penetrating mobile phone networks to monitor the movements of Iran's supreme leader Ali Khamenei and his security detail ahead of his assassination. >> [snorts] >> According to two people familiar with the matter, Israeli intelligence relied on signals intelligence including hacked traffic cameras and penetrated mobile phone networks to confirm that Khamenei and senior officials were present at the compound the morning of the strike that killed him. Um as Bruce Schneier the actual Bruce Schneier not

Grammarly's Bruce Schneier pointed out it used to be that you could hack the cameras but humans had to do the real work of figuring out where the person was. And where AI comes in is that with AI systems you can do this a lot more automatically and so to begin with you have created this enormous amount of surveillance so when it gets compromised it creates an enormous amount of data and now sifting through all that data is so much easier. Cool. So now now we're going to scale things down a bit. With a sort of grab bag of other torment Nexus problems. What happens when your users don't know what they're doing? So uh this is the result of a um

of a survey done by a US law firm, uh Koglaw of Law. Uh they surveyed a thousand adults in late 2025, and they found that a lot of them are asking AI chatbots for legal advice. Uh and also that 50% of them are unaware that their chats are subpoena-able. So, uh you ask the the chatbot for legal advice, then you go to a lawyer, and you end up in, you know, some sort of uh in some sort of lawsuit, and uh opposing counsel can send a court order uh to ChatGPT, for example, and say, uh "Tell me everything that uh that this guy said to ChatGPT, including all of his 'So, how do I get away with my

crimes?' questions." And [snorts] a lot of people are really unaware of this, and it has gotten them uh into trouble. So, one of the other things that you really need to keep in mind when you are building AI systems is that the people who are using them have absolutely no idea where their data is, where it's going, who can get their hands on it, and how long uh the companies can keep it. And this leads people to treat uh to treat these tools in ways that they otherwise would not, and it leaves them vulnerable. And of course, what happens when you don't know what your agents are doing? Uh so, this is uh I'm going to talk a

little bit about a a little project uh that uh Lukasz Olejnik uh put together when he was wanted to play around with agents. And he called it uh Claud Int. It still exists. Uh and he wrote uh shortly after putting this thing up, CloudInt has been live for about a week. It's an open platform where AI agents automatically research current events and publish scored analytical assessments. This week something happened that I didn't expect this early. I I love how cynical Lukasz is. And [snorts] he said, "An AI agent, Openclaw, apparently also had access to an internal cyber threat intelligence platform of a cybersecurity firm." Those of you working at cybersecurity firms, please take note. The The agent did what

it was designed and meant to do. It found relevant analytical content, correctly marked the source, and published a very high-quality and well-structured assessment on cloudint.com. >> [snorts] >> The agent treated it as just another piece of information to process. >> [sighs and gasps] >> Unfortunately, the content was internal. Someone from the platform's vendor organization reached out and asked me to remove it, which I promptly did. So, when you give an an AI agent access to multiple systems, it will use them as an integrator and fuse the data from many sources. All of them. It may not distinguish between internal only at published externally, unless you explicitly scope its permissions. Is there a TLP for AI agents already?

It's how we agentic systems work. The agent did exactly what it should do. It just had broader access than intended. And I think that just as we are now living in a world in which the people who are using these tools have no idea where their data is going or how it can be used or who is storing it, we now have people who are deploying agents who have no idea what their agents are capable of and who don't know to tell an agent, "Don't publish internal threat intel." Cool. And in the meantime, what are companies doing? Companies are playing a really funny game. Uh, which is that on one hand, they're telling you that these tools are

extremely powerful, that they're very important, that you should rely on the data that you get from these tools. Uh, they are cramming them into everything. There are a lot of tech companies in Silicon Valley right now where the use of AI tools is considered to be mandatory. Um, however, uh, Microsoft CEO, uh, Satya Nadella posted a thread in August 2025 about how how quickly, um, AI has been part of become part of his everyday workflow. Uh, Nadella suggests asking Copilot, "Are we on track for the product launch in November? Check engineering progress, pilot program results, and risks. Give me a probability." So, essentially, he's like, "Look, check it out. Copilot can now be CEO of Microsoft." Um, but uh, in

the meantime, if you uh, take a look at Microsoft's terms of service, you get very different language. It says Copilot is for entertainment purposes only. This isn't entertainment CEO. Uh, it can make mistakes, and it may not work as intended. Do not rely on Copilot for important advice. Use Copilot at your own risk. And Microsoft's spokes- spokesperson told PC Magazine in April of 2026 a few weeks ago, uh, "The entertainment purposes phrasing is legacy language from when Copilot originally launched as a search companion service in Bing." Remember Bing? Uh, they they added I I don't. Um, uh, "As the product has evolved, the language is no longer reflective of how Copilot is used today, and will be

altered in our next update." Uh, so I checked a couple days ago, the TOS language is still there. And now, we're going to travel through time. We are going to travel back to 2017, and [snorts] we are going to learn a lesson which I think is extremely important for the the future of the development of AI tools, and it goes like this. So, in 2017, extremely [snorts] non-senior Google engineer James Damore writes an internal memo about how essentially women and men are intrinsically different, and we should stop trying to make it possible for women to be engineers. >> [snorts] >> "It's just not worth it," he says. Obviously, people at Google were not abused. It was immediately leaked and

and published on Gizmodo, and I'm sure you can guess how popular it was there. This also did not make James Damore very popular with his coworkers. And one of the sort of former coworkers who read this post and had opinions about it is distinguished engineer at Google Yonatan Zunger. And what was particularly interesting about Yonatan was that he had just left Google. So, [snorts] he had spent all of this time doing extremely high-level engineering. This is as as good at computer as you can possibly get, and now he was no longer working for Google, and therefore HR could no longer yell at him >> [snorts] >> when he replied to James Damore in a blog post.

And [snorts] he said a lot of things that I think are really useful. He starts his post by essentially telling him, like, "Listen, don't insult all your like a third of your coworkers by telling them that you don't think they're good at their jobs, and essentially they're just there for political reasons. That's really gross. >> [snorts] >> Um but instead of sort of tearing apart uh this guy's ideas about uh you know about gender, uh what he tells him is listen, it's it's not just that you don't understand gender, you don't understand science, or you don't understand math. What I'm about to tell you is you don't even understand engineering. And he writes, "People who haven't done engineering or

people who have done just the basics sometimes think that engineering looks like sitting at your computer and hyper-optimizing an inner loop or cleaning up a class API. We've all done this kind of work and many of us, including me, think it's tremendous fun. And when you're at the novice stages of engineering," he sneers, "this is the large bulk of your work, something straightforward and bounded that can be done right or wrong and where you can hone your basic skills. But it's not a coincidence, you fool, that job titles at Google switch from numbers to words at a certain point. That's precisely the point at which you have, in a way, completed your first apprenticeship. You can operate

independently without close supervision. Tap tap. And this is the point at which you start doing real engineering. He goes on to write, "Anyone can learn to write code. By the time someone reaches L7 or so, it's expected that they have an essentially complete mastery of technique. The truly hard parts about this job are knowing what to write, building the clear plan of what has to be done in order to achieve which goal, and building the consensus required to make that happen. Engineering is not the art of building devices. It's the art of fixing problems. Devices are a means, not an end. Fixing problems means, first of all, understanding them. And since the whole purpose of the things that that do is to

fix problems in the outside world, problems involving people. That means understanding them and the ways in which they will interact with your system is fundamental to every step of building a system, you idiot. And once you've understood the system and worked out what he has built, what has to be built, do you retreat to a cave and start writing code? If you're a hobbyist, he sneers, yes. If you're a professional, especially one working on systems that can use terms like planet-scale and carrier-class without the slightest exaggeration, then you will quickly find that the large bulk of your job is about coordinating and cooperating with other groups. It's about making sure you're all building one system

instead of 20 different ones, about making sure that dependencies and risks are managed, about designing the right modularity boundaries that make it easy to continue to innovate in the future, about preemptively managing the sorts of dangers that teams like SRE, security, privacy, and abuse are the experts in catching before they burn your project into rubble. Solitary work is something that only happens at the most junior levels. And then, it's only possible because someone senior to you, most likely your manager, has been putting in long hours to build up the social structures in your group that let you just focus on code. And then James Damore was destroyed. So, what does this 8-year-old blog post by some guy who used to work at Google

have to do with AI? I bet you can see where I'm going here. >> [snorts] >> And it is that AI-assisted coding has essentially done a lot to replace levels one through seven. The junior coder, the person doing their apprenticeship, the person that can barely be trusted to just sort of like, you know, optimize your loops. Uh, and now, instead of saying that the software industry has been gutted, uh, which honestly a lot of you know, executives and managers want an excuse to to fire people and so they go haha, we no longer need software engineers. No, you need software engineers more than ever. Because somebody needs to understand what these systems are producing and

then they can get down to what Jonathan Zunger very rightfully describes as the real job of engineering. Now, we can get down to the real work.

So, what do we need to guard against? There are a number of things that I think we need to be very concerned about. One of them obviously is powering surveillance. We are doing work that produces an enormous amount of data that crunches an enormous amount of data and that draws conclusions from that data. And when you are gathering this much data about people, about things, about what things are doing, you need to be really careful about who has access to that data, about how that data is stored, about what is going to happen to that data in the future. You also need to be really concerned about the centralized control of media and communications platforms.

Even if you do manage to protect your data, one of the biggest threats to sort of liberal democracy these days is the centralization of our media and and communications, which makes it a lot easier to weaken the response to disinformation and misinformation campaigns, which brings us back to the fake quotes about me from the from the content farms. And we should also be really concerned about technology undermines our ability to believe that there is an objective truth. And this is if anything my biggest concern about you know, the use of of generative AI. I think it is really important to protect people's fair use right to, you know, make whatever kind of, you know, movies or cartoons or whatever that that

that they want. But I am very concerned about the use of these tools to engage in sort of harassment at scale, especially the nudifying apps. This is a really big problem among among teenagers these days and that teenagers are creating non-consensual nudes of their of their fellow students using these apps and and passing them around. And the other is that essentially at the heart of fascism is this notion not just that the government is lying to you and that you accept the lie, but that the government has told so many lies that you have seen so many lies that it you no longer believe that an objective truth exists and so you no longer look

for it. And that does a lot to to undermine, you know, stuff that I'm using like, you know, liberal democracy. One really important thing I think that people need to know is that when you work at a company sometimes the answer is no, we're not going to build that. Uh sometimes the company is going to want to do things in order to make a buck that are immoral, that are unethical. Sometimes it helps to just show up and go like, hey, have you thought through what's going to happen if our, you know, if the government backslides, if the, you know, if law enforcement shows up with a warrant or a subpoena, if somebody, you know, if hackers steal

all of our data. But sometimes you may go to the people running the company and say like, hey, this is just mustache-twirlingly evil and they go, yeah, but like it's so profitable." >> [gasps] >> And when that happens, it is your job as moral and ethical people to say, "No, I'm not going to build that." And what they might tell you is, "Hey, if you don't build it, the next guy will. Somebody else will. Just, you know, we will fire you." And the answer to that is, "Yeah, but it's not going to be me." Because when somebody builds the torment nexus, do you want the way that you are remembered in history to be the person

whose photo is in the Wikipedia entry as the person who worked on the torment nexus? No. And if enough people say no, the damn thing does not get built. So, I want to hear more "We're not going to build that." The next thing that I think is really important is for people to know, do not obey in advance. Uh one of the ways in which authoritarianism works and sort of attacks the liberal democracy is this notion that the company is just going to do the thing the government's going to ask them to do. They don't have to do it. They haven't gotten a court order. The Gestapo is not at their door. The police aren't here. But listen, we're

just going to make things easier. We're We're just We're just going to flatter the guy in charge. We're just going to hand him the information. Listen, it's going to make us a lot of money. If we don't do it, we're going to get a lot of hassle. So, how about we just obey in advance? And if you want uh examples of obeying in advance, you have only to look uh at the state of American journalism right now, uh which is dire. Uh and so, instead of obeying in advance, I really just want to hear a lot more "No, you. Make me." And finally, I think it is uh it is really important uh for us to future

proof our work. I even if you happen to live in uh in a liberal democracy, uh these things uh are not forever. I And so, it is important to think about what happens when a legal request from the government I is is not so legal anymore. What happens when the rule of law becomes rule by law? Uh if you have created systems that only work when the government works, when checks and balances work, what are you going to do when those checks and balances are upended? Um because once the information has been leaked out, you cannot get it back. And one of the ways in which uh in which we can really uh kind of push for

future-proofing our work uh is to decentralize. Uh this is not a a fashionable uh point of view in private industry, where the money is really in becoming uh the middleman, in becoming the platform, becoming the the service that everybody uses. Um but decentralized uh platforms uh such as decentralized social media uh are uh much harder to kill and much harder for Elon Musk to buy.

Related to that, I think it's also very important for us to practice data minimization principles. Uh in security, it's really tempting to collect everything because you never know when you're going to need it. You want full surveillance of the network that you are in charge of uh of keeping safe. You want to know where everybody is. You want to know what everybody is doing. You want every single keystroke because you never know when that's going to be useful. Um but here's the thing, once you have collected all of that data, it becomes one giant honeypot, and it becomes an extremely tempting target. Uh not just for hackers, but also for governments and law enforcement. And so, I think

it's really important when you are collecting data to have a discussion of data minimization principles. Obviously, a lot of this is built into uh European uh regulation right now, but even beyond regulation, it is very important to have the discussion uh at the design phase about do do we really need this data? How long do we need this data for? How are we going to store this data? Have we encrypted it? What's going to happen when, you know, hackers break in? What's going to happen when somebody shows up with a court order? What's going to happen when the cops show up and simply take all of our hard drives away?

And then finally, I think it's really important for us to design with marginalized people in mind. You can see I have cleverly hidden a book recommendation in this slide. It is Design Justice by Sasha Constanza's uh Chalk. And uh one of the things that she really talks about is how important it is to center your uh your product around the most marginalized users that you are likely to have. Uh because if you build for your most marginalized users, you will create tools that work for people with power and privilege. But if you build tools that only work for power and privilege, then you are going to lead uh to a lot of uh harms that you have not foreseen uh for

marginalized people. >> [snorts] >> Uh probably the uh uh generative uh generative AI uh specifically um the, you know, harassment campaigns using uh AI-generated images may be one of the best examples of this. So sometimes the torment nexus has the big sign on it that says, "I am the torment nexus." Uh [snorts] and sometimes uh sometimes it doesn't. So, I'm going to tell a a fairly quick story, which is that fairly recently, well, about a year or two ago, uh we uh had a had a meeting uh with with Facebook in which they were talking to us about their uh Meta Ray-Ban uh sunglasses. Uh and and we gave our opinion. Uh Meta chose to uh

to launch their products uh that year. And what the Meta Ray-Ban sunglasses do right now is that they're they're sort of these big chunky glasses that uh that have uh you know, internet connectivity and that uh you know, will will have all kinds of things show up on uh on your screen. And uh at the time, Meta was all, "Why don't we include facial recognition? Wouldn't it be really cool if you wore glasses and you could see who everybody was in the audience? If you could like look down a street and know who was walking down that street? Wouldn't it be cool if you just like never forgot somebody's name at a party?" Which is absolutely the uh the

use case for this. Um totally not about, you know, finding activists, identifying, you know, people who are opposed to the government, uh stalking, uh you know, so there are there are all kinds of reasons why you absolutely do not do this. Uh but, what was recently leaked was a was a memo from inside of Meta saying that they should release uh the uh Facebook uh should release the facial recognition in Ray-Ban sunglasses now. And the reason why they should do it, and I am not even kidding, is that civil liberties organizations are so distracted with all the other fires that we will not notice. >> [snorts] >> We noticed. So, I often start my talks by telling

people that I'm here to radicalize them. In this particular case, I'm not here to radicalize you. Uh I don't need you to be radical. I need you to be thoughtful. I need you to think about the things that you are building before you build them. I I don't want you to be James Damore. Go be Yonasen Zunger. And it's not radical, but it's decent. And that is what I want for everybody in this room. Thank you very much. >> [applause]