Jordan Sessler and Anthony Hendricks - Fallout from Florida (Oldsmar water plant hack outcomes)

um i will go ahead and kick it off i think my name's jordan um this is my colleague anthony we're both attorneys and we both do some cyber security practice in fact i'll brag on him for a second he was really one of the first lawyers in the middle of the country um that took any interest in cyber security um and so he's a big reason why i'm here and why some of these issues are being paid attention to our talk today uh sort of starts with a heck of a water plant in florida and the scada system there and then tries to document the vulnerabilities that we have really across our critical infrastructure the lack of regulation for that which in

large part sort of arises on the fact that there's very little funding for our critical infrastructure and then where things are going today and so as as people who are from the middle of the country and from the south uh you know we've seen and worked with many of the clients operating these sort of small town water plants and understand how tiresome the situation can be um so we'll skip our bios because it doesn't really matter i i will jump in here with just one thing while we're both attorneys uh we are not your return so uh this talk should not be considered legal advice instead just think about this as a conversation between friends but if

you need a lawyer uh we we know some people that can help you uh and so please enjoy the presentation and because it's the end of the day um we'll sort of uh walk you through it now so if you end up having to leave uh if we ever run into you you can pretend that you saw the whole thing um so our agenda is pretty simple we're going to start by talking about what happened in oldsmar and explaining what scada is we're going to move on to some of the vulnerabilities we're going to talk about really just the lack of regulation especially in the government sphere and how our current system of what's called sector specific

agencies or sector specific regulation the faa regulating everything about planes the epa regulating everything about water the doe regulating everything about power plants really doesn't help us here especially because we now have saiza and size as the capabilities to regulate really all things cyber security the fact that we're leaving other agencies to regulate that is is hurting our cause um and and then we'll sort of look to the future a little bit um and so with that in mind i think we'll start by just explaining for anyone on here who doesn't know what scada is what scada is stands for supervisory control and data acquisition it's a type of ics or industrial control system it's essentially a network either local

or non-local which is going to connect monitoring equipment control equipment and usually a data historian or some sort of archival equipment all on the same network generally produce it to one or more interface stations where you'll be able to view everything about what is going on with that equipment or with those controls as well as outsource um really any amount of data related to to to that network and so this is pretty commonly used obviously in the industrial setting if you have an automated plant or an oil rig or a wind array but it's really used throughout our entire infrastructure system in the united states so our you know power plants water plants water treatment plants

nuclear plants down the list it goes and as a result of that it really comes in all shapes and sizes you might have a very simple system which is sort of what's on the right of your screen there maybe at one plant just one computer at one plant or you might have a system which covers multiple plants so you know i've seen clients that have a small single plant 200 000 scada system there's also multiple plants with sort of dozens of integrated networks that could run 400 million dollars and so that really changes the scale of what it's like to ensure cyber security on the scada system and and you know how to work within that

frame this sort of documents what a more technically advanced scada system might look like with multiple plants and it's going to have redundancy built in it also has firewalls decent segregation and layering so i won't spend too much time on it now but this is sort of an example of how you would want to build a scada system obviously because there's so many different types of scada systems and because we're nerdy lawyers who would love to tell you every regulation which applies to every different type of system we just can't do that here so we're limiting the scope of our presentation a little bit and sort of using scada as an allegory for ics using critical infrastructure as an

allegory for all sorts of industrial systems and then you know for our purposes water plants in particular as an example of that critical infrastructure both because oldsmar was recently hacked and someone tried to poison the water at a water plant but also because our water plants are uniquely vulnerable and really under-regulated and so there's this i think really tragic potential that what happened at oldsmar could happen at 50 60 70 000 plants which in reality have worse cyber security than oldsmar and happened at the same time and as we saw in texas earlier this year even a limited disruption of our critical infrastructure can have catastrophic human effects and so one of the concerns that many

people in the industry have is that there could be a widespread coordinated disruption of our critical infrastructure because of our scada vulnerabilities and that could be sort of a tragic outcome and it's something that is important to discuss with people who are on the skull and i think important for all of us to discuss with people who don't know about cyber security so with that introduction i will turn it over to anthony to do the fun part which is to talk about the oldsmar water plant hack yeah can you get just the next slide please george thank you so much uh so you've probably seen the headlines you've probably seen the new stories uh about how criminal hackers tried to

poison uh a community and so it's pretty scary but what what actually happened so we have criminal hackers who remotely access the scatter system at the old mar water treatment plant and so while the hackers were only in the system for three to five minutes according to reports they were able to take control of the system and so what did they do when they were inside well the first thing they did was they tried to turn off monitor so they could not be detected and then the hackers turned their attention to using uh controls to add a lethal dose of sodium hydroxide into the water system and so we're very very fortunate that the employee working there that day

uh he did the right thing he's noticed something that was odd he spoke up and uh they were able to detect it in enough time and manually override the system let's go to the next slide please thank you so much so the next question that everyone started talking about is how did this happen how did this happen and so there were a lot of conversations about what were the potential causes of it according to florida investigators uh they pinpointed a few things that may or may not have actually caused it and so one of those issues is the use of the password what type of password do they use at the facility and then they pointed out that

the facility didn't use a two-factor authentication uh in in the system and then it recycled passwords and shared passwords and so those are the specific things for for that incident but it points to some other issues with just scada and water treatment plants in general that some of them use older computer systems that lack allowing for automatic updates and some of you may be scratching your head saying why would they still have this type of thing but i always like to use the example of our internet iot internet connected devices so your smart tv uh that allows you to uh connect to the internet the tv is probably gonna work for longer uh than the company provides updates and

so then you have that choice of do i throw away a tv that works that's still able to do all of these things even though it's not getting updates anymore and so that's kind of uh the decision that these water treatment plants have to deal with and so one of the key things is that everyone used the same password at the facility and that they didn't have a dual factor authentication so as a lawyer i just want to just take a brief moment and tell everyone watching that if you don't use two-factor authentication in your personal life you probably should because it's a wonderful safeguard that just in case someone gets access to your password

uh they don't automatically have access to your account uh because they need two forms to verify identity for a login credential now let's get to that next slide

so the system uh jordan i think this is uh yeah i'm happy to talk on this i mean as anthony said right one of the major issues that you run into at scada plants or schedule systems at plants throughout the country is that they're outdated most of them were designed when there was no hookup to the internet there was no broader network interface and now they've somehow been integrated into that and people simply just do not know that that capability exists and no one is working to secure it another major factor here is that there are just so many individual water plants or power plants or wastewater treatment plants in the u.s there's about 170 000 water supplies in

the united states and about 80 percent of our water comes from 148 000 different public water systems in the u.s now within each of those different supplies or systems you might have multiple plants multiple scada networks each of those scada networks might have been built at a different time they might use different hardware and in a lot of these instances these are water plants serving less than 5 000 people and they just simply do not have the tax revenue to update their raw infrastructure let alone their cyber infrastructure so anthony said they're using things that work manually but may not work in our modern sort of life that we live um and so you're dealing not only with an unsecured

system as a whole or a system that people do not know is hooked up to the internet you're dealing with under secured component parts you're dealing with integration that we just wouldn't tolerate today single factor authentication no factor authentication a lack of layering two-way information exchange and a lack of segregation and then in a lot of instances for some of those same just cost factors there isn't any manual monitoring there's no internet monitoring that's outsourced there's no service company that's looking after this full time or 24 7. and most of these plants are only staffed eight to five which means that no one is at that plant to notice something going wrong after hours so in oldsmar as anthony

said there was an employee on site quite literally noticed his mouse moving thought it was weird saw that again a few seconds later thought it was weird checked everything and manually overrode the system but if that had happened at night there's a good chance there would have been no one on site to detect that and some of these plants will tell you that they're not producing water at night so that's not a big issue and that's true and they certainly just there isn't public funding to staff at night but if someone got into the system overnight and overrode the monitoring so all of the tests looked like the water was safe when they arrived at 8 am the next morning

you could have hundreds or thousands of gallons of water in people's taps people drinking that we simply do not know whether it's safe or not so i mean a great example of this that had very little to do with scada was flint um that's sort of an example of how we just have a very federalized water system in the united states and the same applies to waste water or to power plants or to a whole host of different infrastructure items but you know the water you're drinking in flint may be different than the water you're drinking in new york and similarly the security of a scada system in miami is going to be very different than the

scada system in oldsmar and we've seen that firsthand so before you move slide i just have one point uh that i wanted to just add before we move on a lot of times there was a lot of commentary uh after this in cyber incident uh blaming these water systems and we highlighted some of these issues with these older systems about some people not realizing that they're linked to the internet or that they lack passwords and so people were you know using this as an opportunity to kind of uh punch down on the people who operate those systems but you can see these types of things happening in all uh avenues and so if we return to

our example uh that i used earlier about internet connected devices uh there have been a number of instances where uh people have bought iot devices that have certain features or additional things that we just don't know about for instance the ftc went after a company that had a monitor that was just listed as a video monitor but it actually had a microphone that people just didn't know so they thought they were only collecting video when they were actually getting audio a lot of times these iot devices have default passwords that we just don't realize that there's an actual password for this and so these things are happening all across the world and so while we

need to address these for our water system i don't think this is an opportunity to punch down the people who actually work and operate in these spaces yeah that's a very good point and um i mean and if i came off like that i should apologize because i mean honestly one of the purposes of us speaking about this issue is to bring light to the fact that there just are not resources in many of these water systems to detect this to understand the nuances that anthony anthony's talking about to understand that their scada system needs updates and there's certainly not resources available to fix those updates if they're needed and this is as of so many things i think the united

states today it's a system failure it's failing to invest in these small communities or in some of these hard to swallow costs and then getting potentially risky outcomes as a result and so you know again if you're to take an example of a small town in arkansas where i used to live you have five thousand taxpayers you're shutting down your schools you're shutting down your library you you can barely afford to keep your pipes running or street signs on the side of your street because you have a shrinking tax base it's very difficult to be told that you have to spend six hundred thousand dollars to bring in consultants or experts from out of state to fix your skate assistant

at a water plant that is still delivering you good water and whereas we'll get to in a second there's no regulations that say you have to do that because the federal government knows you can't do it and the federal government is unwilling to make you do it or fund it as a result um and so i guess briefly before we move on it's worth i mean i think everyone on here is going to be mostly familiar with some famous skate attacks that we've been through but it's worth running through some of them because it's not just old tomorrow with drain cleaner in the water um it's also the target data breach from 2013 that was originally a scada security incident

so the hackers got into the hvac scada system for target and then because they had two-way communication we were able to go back into target's broader network and take their information there was a dam in new york that i think was iranian hackers got into in 2013. nothing bad happened but they were clearly in control of the dam and that's what started to raise alarms the russians hacked about a dozen or two dozen power plants from 2014 to 2017 this prompted a pretty robust response from the doe um and that's really actually what's been driving some of our cyber security reform at least from a governmental perspective up until oldsmar um but we don't know that much about

that just because most of that was classified there's about you know of all these different you know critical infrastructure scada systems there's about 400 unique vulnerabilities that are reported every year by scada systems from across the country that number has actually been going down a little bit over time that's not it's not entirely clear whether that's because there are fewer vulnerabilities or whether that's because all of the rich scada systems have addressed them and the poorer scada systems at the poorer water plants and smaller towns simply do not have the resources to assess vulnerabilities um there's also been a few famous ones across the world and i think i'll let anthony talk about that sure uh i think everybody is kind of

familiar or at least heard about sus next which is the oldest and probably the most well-known scatter uh cyber security issue and so uh it's probably the worst kept secret is about uh who who was responsible for this attack uh you'll notice a lot of people are pointing at the united states and israel uh for uh cessneck uh dealing with uh iran's attempt to enrich uranium for uh nuclear weapons then we have the black energy in 2015 in ukraine and so this was the first successful power grid attack and so for that one it was interesting because it started with just an email it was a spear phishing uh email attack that allowed them to get access into the

system and then into the scada system and then causing issues with the electrical grid and then we have a probably the most recent one happened maybe last month or earlier this month uh in iran uh after announcements regarding enriching uranium uh there was a attack on the system that a lot of people are pointing to israel again uh for being involved and causing that to happen and so they got involved they got access into the scatter system and were able to cause the center fuses to start uh speeding up and causing damage

now we're going to talk about regulation which is really what gets us excited as lawyers but probably makes everyone else a little regretful of being on the line right now you know the long and short of it is that especially in the governmental sphere there are almost no binding regulations on what your scada system should look like and on what cyber security or data security for that skate assistant should look like and i think it's worth zooming out in the critical infrastructure context and realizing just how crazy that is because everything else is regulated i mean i i've worked with clients where they have a 12-foot room instead of a 10-foot room they get inspected by the state or by

the epa or by the doe and they get a notice of violation osha comes and inspects every working space to make sure that everything is safe down to the amount of chemical the type of chemical the ventilation the alarm system the redundancy alarm system to keep employees safe and yet here we are with cyber security when we know that there are vulnerabilities and it's almost completely unregulated a lot of this ties to the fact that the regulations remain sector specific which in a lot of ways makes sense we want the faa regulating plants we want the epa regulating water we want the doe regulating nuclear plants and the dewey does a pretty good job of

that but some of it doesn't because the epa is probably never going to know as much about cyber security as saisa in the same way anthony and i are lawyers we're never gonna know as much about infosec as people who truly work in the field directly um and and so it's it's just a little bit confusing that the sector-specific regime has stayed in place and the obama administration in 2016 actually issued an executive order which is what sort of inspired saiza even though siza didn't come into effect until 2018 as an independent agency which ordered a coordinated response especially for critical infrastructure and this was a pretty famous executive order and a lot came out of it

but ultimately the government backtracked on creating a single cyber security agency to regulate cyber security and it said that's going to stay with the particular agency that regulates so in the water context water plants remain almost exclusively regulated by the epa this includes cyber security the lone exception the only thing that really directly applies to scada cyber security for water plants just since we're taking water plants as our example is the awia which is passed in 2018 and that just basically required every water plant that served more than 3 300 people which is most but not all i mean some water plants have no obligations at all in terms of cyber security to have a risk assessment and put in

place a emergency response plan which has sort of been the foundational requirement that we've seen throughout the us government and industry but there was no requirement those response plans were actually submitted to the epa or reviewed by the epa or approved by the epa there's no requirement for what they should contain and the epa sort of i think because this is outside their field um only offered about four pages of guidance and one self-assessment tool up until relatively recently and again you're looking at a 400 million dollar scada system spanning 50 miles versus a 200 000 system spanning a half mile you're just going to have different needs and different security issues um and so we feel like that is somewhat

insufficient do you want to jump in yeah yeah before you before you move along i i did was want to talk about the kind of attempted guidance that was provided about apa because they did they did something and so if you're starting from nothing uh it does help you a little bit they provided a template uh with a instructions that kind of walks you through it but even when it comes to the cyber security aspect of it it's very small there's only one section on cyber security intrusions and so while it's good if you have nothing in place that it kind of walks you through creating something uh it's it's very small right and i

think that speaks to i think sort of an important issue in this talk which is that most people on this line who work in infosec if you have anything to do with scada you're probably starting from a much higher level of security than most water plants in the united states so there is sort of a normative employment uh importance of moving from nothing to something but at some point we need to get everyone to the same level and then go up from there i think what the epa was in many ways trying to do or awia was trying to do was to get everyone to at least comply with the nist framework but that just wasn't carried out in

practice and it could be that they just knew that water plants were not capable of doing that and they wanted to nudge people in the right direction um now of course if you're watching this and you work in scada you go what are they talking about there's no regulations we do all sorts of inspections we comply with plenty of standards in private industry at least most people are trying to be as secure as possible and yes that's true in a lot of private industry it's not always true in government or government adjacent areas and most of that is not binding so those are standards right from nist your cyber security framework which is excellent there's also all of the

different ansi standards which you would work with from an engineering perspective um those are separate by field and need the operative one for water plants is the ansi g430 which is joint with awa which is the american water works association which at least up until recently was really you would almost say the pseudo-governmental actor in this field um partnering with utilities and water plants to help assess cyber security implement both the nist and the ansi guidance and so again if you're an industry you're probably trying to comply with that most people should be trying to comply with that but if you're working in critical infrastructure on the government end and you're underfunded and you're not required

to comply with that it's a little bit of a different situation um again a few other standards i think most people are going to be familiar with we had the nppd up until a few years ago that's not part of size and then of course i see a cert because i think most people on the scholar are going to be somewhat familiar with and has been a great resource for compliance and that brings us to saiza yeah so in 2018 we had the cyber uh security and infrastructure security act uh which created the agency and so it's a dedicated agency to cyber security issues and so uh it's a one-stop shop i really like this idea uh we're gonna

talk about some small criticisms of it uh you know because these are our experts um we decided to create an agency that has cyber security experts we should be using them so they're described as leaders coordinators and partners but not actual regulators and so they don't have uh uh enough so let's just yeah absolutely and so as i mentioned no regulatory or rulemaking authority and so instead they're they're able to advise they're able to give guidance uh but they can't make requirements on what type of security you need from scott well if you listen to them you're going to be in a really really good position but they they aren't able to require you to do those things and so they're not

going to make the epa and they can't make the epa adopt certain standards or do additional things and so they don't have any enforcement authority an enforcement authority is very important because you can get people to move you can get people to do things if you have this enforcement authority so instead they do a lot of other things that are helpful uh but probably not the best use of their skills and so they assist uh in helping the fbi and the doj and enforcement actions but they're not doing the enforcement themselves and so they're limited to investigation uh of issues but they're not about uh preventing issues and so while that's important and it's great to

have them in place to do those things we'd rather have them at the front end protecting us and i think that's a really great lead to sort of the state of compliance for scada cybersecurity in the united states today at least in the critical infrastructure or water plant context which is just without any front-end requirement on what a scada system needs to look like or what improvements need to be made and without any front-end investigations to make sure that folks are complying there's predictably little compliance um you know we're just more exposed than we should be in this field and against that that's not a criticism we should be investing in helping local plants improve their

cyber security we should be allowing saisa to propose two pages of regulations for epa to enforce we should be moving in that direction um and a lot of this i think as we mentioned earlier like leads to stratification if you're if you have a giant nice scada system in dallas you're going to have more funds to update your cyber security to update your scada every five years instead of letting it run for 35 years and you're gonna end up with better results but if you're not in a more well-off area or a better led area you could really be falling behind and you just simply don't have the resources to catch up um i guess sort of along that point and and

as we talk about the regulatory and enforcement limitations of saisa back-end enforcement is just never as effective as front-end prevention and so as great as an ics or saisa or doj or fbi investigation is on the back end it just is not as effective as building things right on the front end and we're also then losing sizes expertise and helping make sure that plants look good on the front end and this sort of leads you to oldsmar a functioning water plant that had a scada system actually had enough a secure scada system but it was single authenticated they had a shared password no one imagined that it could be accessed from anywhere in the world

and no one imagined that someone could go in and you can turn off monitoring and i think this is why you see a lot of coverage in the press today about size of being underfunded outmatched and exhausted that's a political headline from last month is you have a whole bunch of very smart people who want to help who just are not currently given the regulatory authority to achieve what they want to achieve and it frustrates other agencies too um you know again epa does an amazing job of regulating water plants and water safety and water as a whole and i think it's frustrating for them that they can't get cyber security regulations in place to keep that water safe

and so sort of revising our regulatory framework can go a long way to making sure that we're all on the same page and that honestly everyone's happy instead of exhausting so before you moved on that slide i just wanted to add one thing uh you know why prosecuting cyber crimes is important uh it's very difficult like it is a difficult task because uh for your local prosecutor uh they're used to dealing with crimes that happen uh right there in their neighborhood and so the end their criminal is right around the block or within their district uh for these cyber crimes your criminal could be anywhere and so it is very difficult to prosecute cyber crimes

and so because of that we don't just want to worry about back enforcement we want to prevent this from happening so that we have less opportunities for the doj to attempt to try to prosecute cyber criminals that may be in a different country and so you've probably gotten the sense from us that we're not a huge fan of the current regulatory scheme for scada security in general um but the good news is that we think oldsmar has really changed the game what we've seen in the press what we've seen from congress we've seen from state legislatures in recent months i think shows sort of the grounds of support for reform that we've seen in other data security

sectors so it you know the law always lags technology you know the ccpa for you know privacy probably should have been passed 15 years ago it finally got up to where technology is today scada security probably in place 30 years ago and it seems like oldsmar may finally be pushing us in the right direction and so we're pretty excited about what we've seen just a general increased awareness the fact that everyone on this call was probably aware of oldsmar is a good thing the fact that the press is engaging with how our water plants are exposed is a good thing and we've even seen the head of the intelligence committee in the senate send in this case now i think multiple

letters to doj epa says after oldsmar questioning what happened and you could tell from the congressional perspective they thought this problem was solved they thought they'd funded agencies they thought they passed the awia they thought the doj was on this and they thought that saiza was on this and they were genuinely i think confused what was going on and we've seen from congress a lot of people are not tech savvy and so the fact that they realized that there was no binding requirement for the epa to review a cyber response plan that there wasn't says authority to regulate this that there wasn't any hard and fast requirement it sounds basic but learning about cyber security

and learning about scada and learning about the limits of the law is something that we need our legislators and our congress to do and so we think it's a really good thing that that's been happening and it's actually led to some action yeah uh so one thing is the increased funding and so now you're seeing dollars uh moving towards uh agencies uh modernization of agencies and technology upgrades but what i think is really important and probably isn't covered as much is that this change of thinking regarding infrastructure uh president biden has an infrastructure plan that thinks about infrastructure not just as your roads and your bridges but also the technology that we all rely on

uh related to those things and so that switching of the mindset to think of infrastructure to include both those physical things and those technology things is really important and so it's not just the money money is important but it's also a changing of the mindset in our approach of dealing with our infrastructures so there are all types of other bills uh that are going out and i know jordan wanted to highlight uh usda uh and so they're soliciting funding requests for uh water plants uh based on the available funding and so that's something that's pointing in the right direction uh could you just talk about that request yeah yeah i'm happy to i mean it's more indicative of the changes that

we're seeing than i think it is in itself truly significant but for you know we just have not had the investment in improving infrastructure especially at for example small water plants in a long time and the usda and there's other authorities as well have funded these sort of investments at least through low-interest loans to municipalities for a long time in the past 20 years those loans have been virtually non-existent and they haven't really extended to cyber security on march 15th the usda reopened rfps and is actually requested then for cyber security investments at water plants and i think that there's parallel rfps for grant applications or loan applications for other types of critical infrastructure and this to us shows that 1.65 billion

that went to cyber security causes in the american rescue plan is actually being spent on that and i think it shows an optimism on the part of the government on the agencies that more money is coming down the line and so even where you might have a biden infrastructure plan that doesn't say it's addressing cyber security explicitly it clearly is agency wise and that is an incredibly good thing it's also something that if you work in any sort of private public partnership this is an opportunity for free improvements or drastically reduced cost of improvements so you know we know plenty of folks who work in those private public partnerships this is an opportunity to go to the

public end of that partnership and say let's go apply for those usda loans before they're gone let's get our foot in the door now let's get some improvements in place because that's just whenever there's free money that's something that we should be able to look out for and we should try to find and and i think critically one of the reasons why there has not been regulation reinforcement is because the government knew that water plants in rural areas were too poor to comply and so they didn't want to be in a position where everyone was non-compliant funding for upgrades is a clear step if you're reading the tea leaves toward regulation coming down the line

because once every water plant has an updated scada system it will be much easier to tell every water plant make sure you're sure you have dual authentication make sure you have basic layering make sure you have one-way exchanges sort of those there's very fundamental things which we can't ask now because it could cost a water plant in a small town six hundred thousand dollars to fix which they don't have once we've paid for those improvements it becomes very easy to ask small town water plants to do that and so we think this funding is a clear indication of where the law is moving and where the agencies are moving because the agencies can move in this

direction even without an act of congress i think sort of you can handle this i'm similar in the size of context um yeah yeah no we talked about some criticism besides uh the simple fact that we had our cyber security experts uh with their hands tied yeah and so uh you know with this increased funding uh we're allowed we're allowing these cyber security experts to be a little bit more hands-on to be able to do more directly related to cyber security and for scada and for protecting us and congress is also trying to do their part they're trying to uh create some bills that would actually increase the role for these agencies to address credible infrastructure

and so uh that that's something that i think we're all going to be following uh and so this is already being done from a national security perspective uh with the department of energy as a priority for several years and so when we talk about uh the scada issues and this water plant we're really talking about not only a cyber security issue but a national security issue a public safety issue and also environmental issue because uh because of our water and i think to anthony's point not a criticism of size a criticism of how sizes hands are tied and how we empower sizes to act more like a normal agency we could solve many of our cyber

security issues at least in terms of critical infrastructure within a matter of months or years and so that brings us to looking forward and i think this ultimately becomes more open-ended again with the increased funding with the bills on the floor with what agencies are already starting to do and the interagency cooperation regulation of cyber security for scada systems at least within critical infrastructure is imminent probably within two years you're going to start seeing regulations be proposed and finalized and realistically within the next five years there's going to be increased regulation for private industry as well and again private industry may already be more compliant that may not be as big of a deal but it's still going to be i think

significant that this is going to be regulated as any other national security issue would or any other workplace issue would which we're excited about even if it's long overdue basic compliance on that is going to be to make sure that you're at least complying with either your corporate standards which may be more exacting or the voluntary standards that exist from nist ansi some of the other resources out there again if you're rebuilding a scada plant think about cyber security when you're doing it that may mean updating a part to make sure that components are secure it may mean integrating things replacing certain lines or networks may mean thinking about how you're going to restrict or expand access

and security that you're going to use when you do that another factor here which sort of comes full circle to the beginning is think about manual monitoring you may be able to skip some compliance issues especially if you are at a less well-off plant or a plant that hasn't gotten funding yet by just having manual monitoring in place so that you have a safeguard and ensuring that you have a sort of manual override so that if you notice an issue you can override the scada system or another option would be to do sort of one-way historian outsourcing to see if you can find a partner to reduce cost um so yeah let's talk about some other

things that you can do to kind of just get ready and so one of the first things and probably the most important things is to create a response plan and so when i talk to clients about creating response plans i talk to it i talk to them and refer to it as kind of a flavor and so growing up when i played football the coach made us run laps if we didn't study our playbook and so for clients uh once you have a response plan in place you need to study you need to train your personnel and your employees to know what their responsibilities are because the worst thing that could happen is that you put all this time and effort

into creating a plan and then you don't use it because you can get it to all types of trouble by having a plan that you didn't follow and i think that just one of my favorite quotes from the talks earlier today was that you know if you were in sports you wouldn't just practice once a year um i think that's the same thing here right like make sure you have a plan make sure you use it but it's not enough to put that in plan put that plan in place once report it to the epa and move on you need to make sure you put it in place you're practicing it and you're bringing in some of these

non-governmental organizations or federal agencies to test it realistically as frequently as possible and if you negotiate that and you act in good faith you can usually ensure that if violations are found you're not facing undue liability um so it can be all help no hurt and i think that's just a great way to sort of look at it especially in the sports metaphor oh yes yes yes and so new regulations are going to come and so you need to prepare for these regulations and so one of the easiest starting points is to look at the standards that are out there because i think that'll be a starting point for agencies a starting point for congress looking at the standards that

are there so those are things that you should be striving to meet and to apply uh you should also kind of monitor what's going on uh when these standards are coming you can do that internally or you can hire outside counsel to help you with that process and then finally you need to prepare for enforcement because if there are rules that are happening that are going to be in place then that means enforcement is going to happen and so you should keep good records about all of your systems uh you should uh find a lawyer uh who can help you uh that you have that in place so that you're not scrambling after an incident

uh you should think about ways to mitigate your risk either through insurance or through uh some other type of process yeah and i think on the security effort or the keeping a record of your security efforts especially if there are multiple actors involved it can be pretty important to have those records so if you're a private operator operating a power plant that was built through a you know an rea grant 60 years ago and is still owned by the town that you're operating the power plant for you need to come in and document the condition of the scada system there when you arrive you need to document improvements you're making you need to document any tests you're running on it

and if you're not allowed to make improvements because it's owned by the nearby town you need to send requests to the town to upgrade it you need to show the town why there's a need and you need to document when the town is not upgrading their scada system per your request so that if there's any dispute as to liability later it's very clear that you as a private operator did everything you could do and you're not at fault that's sort of one of the lawyerly things that's just important is when you have that record it's easier to litigate these issues later in terms of other things you can be doing right now take advantage of your resources and

again most people on this call are going to be working in private industry they have in company resources they have auditors they like to use consultants that they like to use those are great resources but if you do not have those resources there are free resources available that a lot of people don't know about or they don't know don't use or they're afraid to use um if if you're in a public private partnership or you're a public water operator go get usda funds right now to rebuild your scada system the government may pay for more than your scada system they may pay for a whole new plant or a whole new water tower or other aspects that can be

great and can allow your skater system to run for the next 70 years with minimal intervention or upkeep this photo on the side is kind of weird right but you basically see critical infrastructure that was built out into a lake or out into an ocean grants exist for that and there's money out there that you can take advantage of which i think is important you can also work with the ngo actors like awwa to comply with existing standards partner collaborate you're getting someone else to basically do part of your work part of your internal auditing and external consulting for you free of charge because it's serving the public interest which whether you're private or public

operator when you're running critical infrastructure that is in the public interest which is why some of these resources are available to both types of actors we it doesn't matter who's providing water to the town i'm living in i want clean water and presumably so does the us government that's why we're so excited about the regulatory reform and the increase in funding and it's one of the reasons why we want folks to take advantage of these resources and even seize it has you know they cooperate they partner right now that's really all they're allowed to do because of their regulatory limitations but you can call size it you can get a size of audit you can work with saisa

and i would really advise you to do so now because not only you're getting government subsidized benefits but there's no regulations in place right now so if you work with one of these organizations or you work with siza and they identify features that you need to improve recyclable cybersecurity shortcomings right now that's a success story for them they help someone improve their cyber security two years from now if you have those shortcomings and it's in violation of a regulation you could be liable for damages you could be liable for other bad outcomes that you don't want so there is actually a window right now i think to take advantage of resources and to improve well everyone's happy to

see improvement and before anyone blames you for being behind so especially if you're a publicly owned utility or a small town i think now is the time to take action and again some of these same things are going to basically be available to private operators it can be a little bit more complicated you may sometimes need a private public partnership but there just are resources it doesn't mean that they're better than private audits or consultants or or if you're at a larger company and company audit oftentimes they're not but they are a second look which is important and they are affordable if you can't afford those internal resources so one important thing to remember is

that doing the right thing having good habits they they pay dividends they they help out so much and so the approach that everyone should take is to be mindful of cyber security issues uh when maintaining your system so if you have an old system you should see if it needs to be upgraded uh oftentimes a system may still work physically uh even though the technology is outdated or you're no longer being able to get updates and so that may be a reason for you to upgrade even if it's physically still able to complete the task you need to train your staff on handling security incidents and identifying the right approach as we talked about that old smart

incident uh the employee was the saving grace he saved the day and that's because he was trained that's because he was confident enough to speak up and that's something that's super important and then we talked about a plan earlier but you should have a plan you should study a plan it should be in place now if you have a newer system if you're fortunate enough to have a newer system then you need to listen to the technology team the people who are are going to add upgrades who are going to patch your system and you should follow their advice uh you should secure your components and make sure that everything that's integrated and connected is also secure

and to beat that horse on that i think make a record of all of those things one of you know our argue is from the legal view the more you can document doing all of these good habits the more if you ever get into trouble you have that as a defense which is important for risk management and limiting liability and i think that pretty much wraps it for us i mean i think our takeaways are that we got very lucky in oldsmar which is atypical to get lucky but sadly oldsmar is typical of how exposed our scada system is and scada attacks are increasing um i think that as we get into the age of

cyber cyber warfare between the us and adversaries we should be very concerned about this and so i'm glad that changes seem to finally be coming about um we're always happy to talk more about what those changes are going to look like a lot of it is tea leaf reading right now but if we can simply get everyone up to a baseline level of security it would minimize the fear i think that 50 000 water plants are going to go down at the same time and 50 000 wastewater treatment plants are going to put out sewage sludge into your drinking water through a cyber attack and those are things that truly do keep us up at night

and i think keep the professionals at siza up overnight as well um so again identify your resources comply with all requirements that you think are necessary or to currently exist and work to get it ahead of the game um i think that's pretty much it for us yeah thank you so much yes so if at all possible if we could get you to hang on and maybe answer a few questions yeah sounds great all right and it's uh it's what almost five o'clock for for you all right five o'clock so we're sorry to keep you all the way to six i was trying to talk fast to get everyone off the line but then i just talked

more so no no it's fine i mean that that's the expectation you know we we go to six o'clock um with this conference um [Music] yeah so so you know it's fascinating first of all it's great to have lawyers preventing at b-sides knoxville that's a first for us um you know we we've always been a bit different from the other b-sides there's uh over 130 b-sides uh cyber security conferences uh worldwide it's intended to be kind of a like a regional community type thing um but where we started out one of the companies that helped build besides knoxville actually had a research center in israel in in tel aviv you know so we've always been kind of global we've always

kind of attracted um uh talks from from all over the place uh so so it uh it suits us it fits our our model um but yeah so um yeah very sobering topic as as somebody in in the discord uh put it um and i wonder you know talking about 50 000 water treatment facilities you know i i remember looking at voting security and thinking you know oh my god how can we get this right with 3 000 counties over 3 000 counties in the u.s all doing their own thing you know buying voting machines off of ebay and stuff like that and and now you're telling me that the number of water treatment plants

like you know subdivide that 3 000 you know by a few orders of magnitude more and hundreds of thousands of of these facilities but i wonder if that's not a saving grace you know the fact that you can't hack any one point and and get kind of wide wide scale destruction or uh or anything like that um or or is that kind of a like a a false view of it you know maybe you know 60 of them all use the same software you know and if that same software were attacked you know it would affect multiple of them but i i i do wonder if what's your thoughts on that is is that a saving grace that it is so

fragmented or i think or not i mean so so it there it is remarkably fragmented there are several software companies that in terms of modern scada systems are used everywhere and that's similar to what we're seeing with modern systems everywhere right you think with solarwinds hack um but in terms of these plants that are truly starting from nothing security wise they are all different i mean this is every shape and size so you would to get all of them at once would really require a nation actor or state actor coordinating attack with thousands of actors at the same time which is we know possible but i think to your point much more difficult a lone wolf

could not bring down thousands at the same time it probably would be limited to a handful or using not just the same software but the same hook up the same security the same component parts because on some of these you will have a component part that will trip um if you try to access it remotely because it's so old it just fails in the moment so i think you're right that it's a saving grace but i i would say that we're starting from a much lower level of security probably even than voting machines yeah so i think uh here regionally in east tennessee i'm not i'm not aware of any full-time cyber security folks at water treatment facilities i know our

electrical utilities do have some full-time security folks um so we're uh tennessee valley valley authority is big here you know uh kub here in knoxville uh you know you know i know they they've had uh dedicated security folks for a while but the water treatment plants yeah i don't know i don't know that uh you even hit any of that audience here but we will get this up on youtube you know and hopefully we can get the word out there so that that was kind of one of my questions is uh you do have the ear of the cyber security community so is there anything that we could be doing you know to kind of help get the

word out you know like it's it's shocking to me that one of the things you're focusing on is uh hey you know the the usda has has uh grants you know that people are not using like like the money's out there and people are some of these uh uh utilities are maybe not even aware of it you know so that that's a bit shocking to me that um you know it sounds a little bit like uh you're trying to get kids to go to college and hey there's pell grants like you know like most of these grants end up these college grants end up going unused and uh you know so in my mind it's like talking

to college kids here like go visit your local water treatment plant tell them to you know check out these grants so do you know the what is the name of the grant um that you know where should they be going i guess first of all and then second question the cyber security community what can we do to maybe help them out so i'll start with your second question about what the cyber security cyber security community should be doing uh the first thing is everyone hears the scary stories uh but i think it's good to hear uh that you know while this is a big task there are people who are going to be able to help you and then

that there are small things that you can do cyber security related just small things that can start helping along the process because if you just hear a presentation and talk about all doom and gloom you talk about how we need to spend so much money we need to update and upgrade everything people kind of turn it off but if you tell them hey uh i know we need to upgrade eventually but hey this password let's talk about password hygiene um let's do these small things uh let's talk about some training for employees about who they need to call or report to uh when these types of things happen uh and so these are small things

that can be done that are way cheaper than updating the whole system that people should know about that are that are available and so people should understand that there are small cyber solutions and big ones and all of them are important and if all you can do is take a small step right now take the small step i that was incredibly eloquent i think is is the right approach i mean i i think that's that's a room for people on this call not only to be ambassadors but also sometimes it's a business opportunity or pro bono opportunity to just reach out and talk about the issues so if you've seen this you might have a water plan especially

in a larger area that says yes we consult with someone we've done everything we need to do we're in great shape you might have another which i've worked with some of these smaller town water plants they struggle to have an operator who who is at the site able to know what's going on in the water they certainly don't have the resources to have a cyber security technician there but if you went to the management and said hey there are resources this is what that looks like i think there's an interest i mean as much as we did talk about the doom and gloom everyone here especially in the government infrastructure wants to do the right thing we all have an interest

in good water we all have an interest in our infrastructure working everyone who i've ever met both regulator and operator and and everyone in between i think has good intentions and so i think that's important to remember and i think that's our window to all work together and and solve this in terms of the specifics i think that's a rural development grant i will post a link to that particular rfp on schedule but there are dozens of other similar rfps i can try to run them down or if anyone wants to reach out i can try to search for them not all of them are online yet because many have been taken offline for a while

but in general there's usually development grants or loans and oftentimes they take the form of loans on a reduced interest or forgivable loan to develop some of this and i've seen that work in a public private partnership where private private company decides to work or reduces the rate to work with a public actor they work on submitting figuring out needs submitting a loan together getting that loan and then carrying out the work on the back end as a contractor for the public entity so so can i ask a quick stupid question here um these these utilities are are these non-profits are they private businesses are some of them public is is there a mix of all of the above

because i can't imagine that most of them are profit focused or very profitable so yes they are all of the above so some a lot of utilities are still locally owned so if i have the city of xyz city of xyz will own their water plant and they will try to operate their water plant and hire people to run their water plant and that's essentially a subset of the city it's also pretty common to see it done as some sort of a public trust or a utilities association where it's technically separate from the city but it is wholly funded by the city and cities do that to get a tax assessment to pay for it

so you'll pay so it's a mix of everything it's a mess yeah yeah and then you'll have private operators who are hired to run that a lot of cities will run out of money to pay for their own water treatment so go out and hire a private operator some are purely private and just simply supply and then some are somewhere in between so tva is a great example of a semi-governmental non-local organization and that's one of the reasons why it's it's one of the gold standards in the industry i mean oh i didn't realize it was well yeah i'm a tva i i can't they're not my client if they were i couldn't talk about them i i don't know

the details but tva is well regarded and i think part of that is because of its size and and it's sort of semi-governmental but non-local role it allows it to have more resources and use them to create uniform outcomes over an area as opposed to wildly various outcomes okay yeah we we probably we very likely have tva folks watching this now yeah all right excellent um let me see if we have any other any other questions here i think the main thing was uh was asking about the you know how to find those resources you know usda you know the the low-interest loans um you know some of that as we maybe think about reaching out and

helping here because i i imagine out of those 150 000 plus um you know plants that you mentioned um probably very few of them have any kind of dedicated cyber security folks so it might be might be a lot of pro bono necessary here you know maybe maybe this is something we we create a push for in our industry you know for us to to step in contact them help out where we can i know we do have some people in our industry who um will take a look at what's available on the public internet and it's almost a trope of the cyber security community that hey you can go and show dan uh you

know which is a resource that scans the entire internet to see what's exposed and vulnerable and uh you know like if you have any idea what you're doing you'll find a water treatment plant in three minutes or less or something like that there are screenshots everywhere because you know you see this this old software you know from like the mid 90s running on windows 98 something like that and it's got the little diagram of the uh you know the for the chemical release and and the uh you know i i don't know the right terminology for what we're looking at there but yeah you see a lot of breweries and water treatment plants on on just oh wide open to the internet

like click here to release chemicals so yeah yeah if we if we can help in any way um you know that that might be might be a cause for us to latch on to as a conference who knows sounds good dust this was so fun talking to you about these thank you so much for uh allowing us to speak yeah yeah no it it definitely seemed super interesting to us and i i'm glad that we we chose your talk we do blind reviews and uh we had never received a submission like this so we were we were immediately interested uh you know to see what it would be like and i i don't regret it

thanks for having us thank you all right really appreciate it and uh you guys are the last talk so i'm just gonna wrap up the conference a bit here uh you can stick around if you if you want to for that uh there's there's not gonna be much from here on out but um definitely uh i appreciate y'all giving that given that talk and and uh that information um yeah if you want to email over any of that you know we'll get it to the people asking for it um you know how however you want to send that that's you know that that information over on on the grants and the the loans sounds good we'll hop on