← All talks

IATC - Water, Water Everywhere: The Krakens, Kelpies, and Mermaids in today’s Water Sector

BSides Las Vegas59:1730 viewsPublished 2023-10Watch on YouTube ↗
About this talk
I Am The Cavalry, 17:00 Tuesday Water is life…and increasingly exposed to accidents and adversaries. There are over 150,000 water systems in the United States alone. Further, water is critical path for the resilient functioning of Health, fossil and nuclear power plants, food production, living populations. Dean will be discussing some of the existing security challenges of the water system, and how they can impact other critical infrastructure sectors. Dean Ford
Show transcript [en]

all right we're at the top of the hour we good video wise and we're good audio-wise and everybody smile and everybody's happy check check we even got a we got a mic for our speaker that is fabulous our presentation now is called water water everywhere the Krakens Kelpies and mermaids in today's water sector so yeah please join me in welcoming Mr Dean Ford uh Dean 25 over over 25 year career has included um has involved automation systems engineering and Consulting he serves as the managing principal engineer at luminary automation cyber security and Engineering LLC but here's the deal he develops and leads a dedicated and passionate staff of automation Engineers he is a licensed Control Systems engineer in 24 states of

these United States and partic participates in many standards committees and he's a member of a lot of associations but he is passionate passionate I say about informing folks about the issues related to the water system so with that Mr Ford take it away thanks David so I I can't sit down I there's just something in my brain when I talk I have to walk so um real quick so why the title Krakens Kelpies mermaids um I try to go with a little keeping with the theme of these conferences um you know Krakens down there on the lower right they're uh they're quite a a mean breed um and if anybody's been to Scotland and what Scotland water did

with the uh with the canals over there over the last 20 years or so there's uh these Kelpies are coming out the stainless steel Beautiful Creatures coming out of the ground um if you ever get into that area please stop by and then you know the mermaids let's you know we'll say that they're they're the good so the ugly bad and good good bad and ugly that's as uh about as creative as I get so uh today we'll go over a few things um trying to keep with the theme of of what Josh and the the calvaries about um so I wanted to kind of educate a little bit first on where does water come from um who and what uses water and

what are some of the threats that we're seeing in the water industry and then go over some questions so who is this guy David went through a few of those things um Key Parts to focus on I am not a hacker I'm not a cyber guy um and in three different separate personality survey study deals the results came back that I challenge people's basic assumptions so so there's some things I might say that might not work out today in your mind but uh just I'm challenging your assumptions and then I operate on on four really foundational principles um I don't believe in accidents I don't believe there is such thing as an accident somebody somewhere made a

decision that caused an incident uh I focus a tremendous amount on people and people how people work with technology how their brain works with technology the last 5 to 10 years technology has increased to a point where you can analyze how your brain works with that laptop that device uh the different parts and pieces in your world and really the destruction that we're doing to the human brain because of all this technology I look at Cyber as a great unifier um I see it as an opportunity for all these systems that have not been able to get money to label something with cyber so that the money froze freely um you know that's my world that

I live in is I got to get money to get stuff done and I got to talk to boards and I got to talk to funding sources and for some reason when you put cyber on it nobody wants to talk about it they just want to give you the money and then uh for us in the water industry specifically cyber is just one of the many many many many many risks that we have to manage um and it's not one of the higher risks that we have to manage as we'll get into um so audience participation time where does water come from motorc good answer that was fast where where what's that the ground the ground okay so underground Fair

answer it's stored in the cloud it's stored in the cloud I'm done all right well let's go through a few of those all right so um for human consumption water is is one of the Earth's most abundant resources right but for our consumption as a human we can only use 2.5% of that water it's only 2.5% of the water on the entire Earth that we can actually use we call that freshwater now of that 2.5% less than a third of it is what we can actually get today so think of glaciers which I guess in a few years will be gone um so far underground that we can't access it um there are reports now in California we have wells going so

deep that we're touching water that hasn't seen the light of day in over a million years um who knows what's growing in that um so there's a lot of other forms and things like that that we've got out there but just understand just because you know we as humans are 98% water the Earth is full of water there's not that much of it that we can actually use and it is not an Irreplaceable it's not a uh u a uh infinite resource it's a very limited resource the water cycle you saw this before didn't you so so real quick this is a a pretty basic water cycle without human intervention so you know it is a cycle

right it's it's somewhat of a closed loop if you will as well um this is a courtesy of the USGS you know there's a lot of things going on in the cycle uh snow melt you know you're going through this this constant piece the parts to think about are what we see down in the lower left with groundwater infiltration groundwater flow you know uh we need to get the groundw back into the ground after we pump it out right so there is processes for all this here's probably a more realistic view of it because it adds evaporation and then all the different ways we as humans can screw up water um there are very very few

resources that we use on the earth that we only use one time and water's one of them unfortunately all right so you flush the toilet we've only used that water once and then it goes into the waist system and back into the the cycle so you'll see we were just talking about um the gentleman here in this area about a lot of reuse cyphs that are going on right now so a lot of different opportunities are coming out for reuse and and um you know the concept of toilet to tap uh I know it sounds a little weird but most of you if anybody has a is getting their Source water from a river you're already

drinking the city's Upstream Wastewater so you know it's not that big of a deal to to think of it that way so where are some Supply sides underground we got aquafers and Wells um I encourage you if you if you're on any kind of a well system if your water supply is coming from any kind of a well um check out where your aquafer is your aquafer might be being fed three states away and so it doesn't really matter what your groundwater is doing or what you're polluting what the people in your area are polluting it matters what three states over polluting so those are some things to think about me I live in Maryland our groundwater comes through

Pennsylvania so when fracking turns into a thing we're very concerned in Maryland because there is no possible way that fracking can't contaminate the water supply because you're Drilling through the middle of it well all of our water's coming through there so pollution is a big deal surface water rivers lakes reservoirs rain water storm water and then we talked about reuse so believe it or not we actually treat when we send waste water out it's far more cleaner than we get it and most of it's drinkable it meets all the minimum requirements to make it drinkable um so it's a public issue that you don't feel like you should be drinking Wastewater well I think uh some

states are going to have to start getting over that so where do we differ from electricity um we had a lot of discussion about the electrical grid um so everybody thought 2,000 public utility electrical systems were were a lot in the US today there are over 151,000 public water systems 151,000 public water systems that is only serving 80% of the population so where's that other 20% coming from folks like me that have a single well out in their yard that's serving me there's 16,000 Wastewater systems serving about 75% guess where the other 25% to septic system of that 25% of the septic systems out there the overwhelming majority of those are failed or failing and dumping

dirty water back down into that cycle that we saw before right the other interesting thing is water systems are local there's not a massive grid so if my power plant goes down in my local next to my house the grid still stays up right well that doesn't work that way in your water system if your water plant goes down um it generally you're going to have to back feed from another water plant but if your entire water system goes down you're done we don't have a lot of cross connects between utilities there are some that do that but very few so small failures can lead to a very specific and local outage um and but you know electric we

get to dump the electrons after it goes through the light bulb they go back into the ground right through the grounding systems we don't get to do that with water we have to we have to treat that water and uh the oh she's not here she talked about the the cup of tea causing the big problem well in water we have something we call the big flush and it happens every year at the Super Bowl at halftime and every water utility across the states goes you can see the spike it's very Trend it's it's a massive Spike of water usage and then there's a slug that runs through the Wastewater system back to the plant it's uh it's

quite amazing who uses water we do what are some industries that that use it agriculture water agriculture cement concrete that's a good one I didn't think about that one Healthcare Ian again the data centers that's very good I got to start adding that in here y right so there's quite a few um you know to your point everybody does you know residential uh we we call that portable water right you want to be able to drink clean water uh commercial buildings what people what a lot of folks don't understand is a lot of commercial facilities are cooled with water there's a water component that's in that heating and ventil heating and cooling system industrial same cool and heat flushing

production transport uh Energy power generation will sew that in a little bit uh Public Safety you know fire hydrants um sanitation healthc care is a big big user food uh we talked a lot about the are kind of the egg side before but there's the the the production side right there's all the different parts that go into that Transportation uses it think of your waterways and then we all use it for recreation as well we don't want to go out to a burning River um and try boating on that right so we talk about freshwater withdrawals and this one still surprises me every time I see it but this is only from 2010 this number hasn't gotten any

better um but Thermo El electric power so basically power generation is our biggest user single user of water irrigation comes next and that's not you Watering your grass that's uh us making Green Fields out of desert and then um you know a lot of different things after that public Supply kind of is a residential space so uh again 2010 data this stuff doesn't come around very often they haven't updated one recently there's I think there's something from 2015 but the think of the Trends not the exact numbers and then there's a a huge discussion going on right now about the water energy food Nexus and so this group of focus folks got together and they researched

1400 55 articles and they just started pulling words together and I forgot what that's called some kind of a social area network or something like that but what stuck out at me out right away on this was so water is in the lower left and on the right is water and energy and energy so you got two giant lines going over there and then over on the left again is water energy and food so there's a a huge connection of those three resources if you will um and they really none of them can operate without the other again food production um relies a lot on surface water and groundwater so of all the the users we we in the US use

about 3 330 um how do they call it 330 million 330,000 million gallons per day so we do everything in the water industry a million gallons per day so it's 300,000 well for the rest of us that's 300 billion right so 300 billion gallons a day um and so this works out um to some some very very large numbers hospitals was a specific ask I have a client um in the South that uh that makes this statement I'm going to read this hospitals are the most critical customers that we serve even a few minutes without water is detrimental and presents a major life safety threat we'll go into that in a little bit our hospitals are some of the most

significant water users in our system this particular Health Care system at this utility is their largest customer I think they've got four hospitals there and then also each Hospital itself is all is one of the top 15 users on in out of that utility that utility serves 700,000 people almost 800,000 people to give you an idea of the amount of water we're talking about where does all that water go at a hospital so this is from uh Boston Massachusetts Water Resources Authority is Boston um so the sanitary number probably makes sense right the cooling HVAC probably a little shocking um medical processes the rest of these make a lot of sense that the HVAC number continues to stand out but

again en large building commercial space that HVAC systems are all based on water so if you see those in a generation facility or you're driving on the road on a on a day with a lot of fog or or it's hot you'll drive past those cooling towers and you're just spitting off right they're evaporating well that evaporation got to be made up and they do that obviously continuously power generation you know fossil fuels don't necessarily generate power directly they do in in Peak stations but in base loads we're creating steam steam is made of water water check in my my science here so both in in in fossil fuel and nuclear power the Heat that we

generate is transferred to steam the steam creates the spins a turbine right we generate power pretty straightforward right so that this was just recently there was a a small attack on a dam Russia attacked a dam in Ukraine blew the dam up there's also the the the Europe's largest nuclear power plant that Ed relies on that reservoir uh to cool so they rapidly started pumping water out of that Lake and and uh making some ponds and doing some other stuff so they they feel like they're in good for for good for a little while but that's an example of how these things can Cascade into you know your largest nuclear power plant shuts down uh

because of a simple thing as a a reservoir that it was pulling water out of right hydroelectric obviously that's kind of needs water right well not a big deal except your Hoover Dam I don't know if anybody's seen the picture of that lately it's frightening how low that level is uh even with all the snow melt and everything that only increased like 20 ft I think was the only only amount that it went up so it's still down to like 1,000 feet or something crazy so um you know these are all big Supply sides power generation consumption you know again it's just kind of shocking how much water something like that uses uh so 11,000 gallons per or almost 12,

gallons of water per megawatt hour um so in 2020 it was 47.5 trillion gallons of water a lot of that again is is the power generation industry is a lot better at at reusing but they are in this cycle of evaporation and you got to replenish that so um you know this is water that basically they boiled off almost 50 trillion gallons of water they turned it back into vapor and send it back up in the atmosphere I thought this was pretty interesting um by state where was all the the heavier users of of water for thermoelectric basically electric Generation Um so Texas and Florida are the largest which is kind of interesting to me but um I haven't dig I didn't dig

enough into this it was just fascinating how the the West isn't as where you would think it would be for uh for power generation residential use so our portable water right the little pipe that comes in um you know events like uh Flint um events like olmar oh um were not necessarily cyber incidents those were again decisions made by folks that that that should have done known better um but you know when we look at those you know we look at how much the reason I say we're we're we're fighting a lot of problems there's a trillion dollar crisis in the water industry right now there's so much buried infrastructure in pipes I mean in

some cities we still pull out wooden pipes out of the ground when we're replacing fixing a leak uh it's a little frightening to think about that but you know that pipe hasn't moved around a whole lot in in a 100 years and so there's a uh there's a real opportunity that that those things have to get replaced eventually so residential use you've got a lot of different things you know uh sanitary again a lot of this is single time use I don't think anybody's keeping a bucket of water that they're washing their hands in and then washing the dishes in later right so you know think think of how you do this when you're camping water gets a little bit more use

um and and it is a known fact that you don't actually have to flush the toilet every time you use it just throw just throwing that out there like if like if you go to the bathroom and your spouse comes in behind you and goes to the bathroom you probably don't need to shut flush a toilet the person after you should just throwing that out there again um some surprising numbers uh on the the amount of usage gallons per day per person um you know the the mid mid uh what is it Idaho Colorado or no Wyoming Utah really up there I want to understand that some more um on the trends that go on there

uh on why why those folks need so much more water than the rest of us but there might be a lot of uh Utah there might be a lot of people trying to grow grass in the desert so you know those sorts of things we we as a society need to get smarter about where we're living and when we live there we should actually adopt what's already there and not try and bring invasive species like grass in grass is an evasive species believe it or not it's not natural to the US I don't know if anybody knew that it's it's not a natural occurring thing so what are some of the threats um going through different areas um so I'll

get to the Cyber Parts F in a little bit but I I want to make sure that we're all thinking about this this particular industry in in a new light perhaps but scarcity is probably the largest threat and turning water into a global political crisis right so um there are entire countries that that you know water is not readily available right um or they got to carry it in a in a bucket on their head for for miles to get something they can drink um and the droughts that we've got in the in the southwest right now in in most areas right now in the US are in drought um I'm in Maryland we're in a drought um

what a lot of things that you'll see in those areas too are the the um and I should have shown that uh if you want to look at a very interesting graphic uh look at subidon in California and Virginia now you'd think Virginia they don't have a water problem Virginia's structure underneath um the Chesapeake Bay was caused by a large meteorite I'm sorry for those of you that believe the Earth is only 2,000 years old but it's not um there's a there's a the way that the structure works there is that water doesn't get replenished by the ocean it's it's isolated well they've pumped out so much water there that they've dropped the entire state has dropped a

couple of feet um now there's a few Wastewater utilities that are working to start regenerating that and pumping back down into that aquafer um but California's got a much larger problem I mean we're talking tens of feet that Central California has dropped it's amazing to see it and and that people haven't realized it but now I was it that last big rainstorm that created another Lake that hadn't been there in 50 years I can't remember the name of the lake yep so those are the things that are going to start happening and and that's going to become a lot more realistic because not only are we having much more severe weather events but we've also drop the elevation of all

this land and ground so there's you know water flows downhill that that physics law still applies so those sorts of things are going to start happening a lot more um I don't know what to tell you about that so again we only use water once um we're getting better at it there's cities like Anaheim California a lot of the California cities you'll see a purple pipe running around the purple pipe is reused Wastewater and they're using it to irrigate and probably Tucson and Phoenix and a lot of those sorts of cities um so it's becomes another another water source for non-potable non-drinking water I actually have a bigger concern from a threat standpoint if somebody

wanted to do something bad to our water supply for in Maryland all they got to do is drive across a little bridge and knock out most of the city of Baltimore because our largest Reservoir has three or four highways rolling over the top and toss a bag of something over that we don't test for if I don't test for it I can't find it and I don't know what kind of wacko stuff is out there right now and if you throw it in the water supply I'm not going to find it I'm not going to filter it out and again the Aging assets are a bigger concern for us um you know we just aren't replacing

pipe fast enough there's a lot of societal political things going on in the water industry um you know these some of these communities Jackson water as an example they just don't have enough money to maintain the system um I know there's a lot of commentary going on about Jackson water but the issue at Jackson water was it had nothing to do with storms and everything it was a lack of resources and the inability to hire qualified people because the pay rates were so low and that caused this series of events that occurred over about 5year period so so that when one pump failed they lost control of the system when you lose control of a system the water pressure

drops and creates a vacuum inside the pipes and you suck in water because every pipe leaks it it doesn't matter when it went in your pipes are going to leak so it's imperative that we put pressure on the system so what they found was is that when they got back into that system and they figured out that the the distribution system was not working the way it was supposed to I was putting pressure pressure in over here it should be coming out over here but it wasn't what they found was is over the years valves get turned on turned off to create spots to make fixes and if you don't go back and open up those valves

in that scenario then the water now no longer goes this way it has to go up and around and over well that whole system was doing that so that was a real cause of that failure in in Jackson regionalization private I should have put privatization I'm not a I don't believe in privatization I do not believe that the free market is going to fix water um the United Nations is is and most folks believe water is a right um and turn it over to a profit-driven company is not going to solve that problem that's the opinion of Dean Ford um happy to debate that anytime pollutions anybody heard of posos that's the water guy he knows is

that already uh I forgot what it stands for it's polyfloral blah blah blah blah blah I call it forever chemicals so when posos gets into you it never leaves there's no way for you to get rid of it guess what it's in every water source that we have at different levels it's like plastic today microplastics and fish uh mercury and fish um it's pollution they're forever chemicals we're looking at at billions of dollars to remove that from your water supply well nature didn't put that stuff in there Dupont and malra and other companies did they just settled with Dupont for a couple of billion to help solve this problem but again we can't expect a a a a water utility that

already is struggling to to do this to to produce water to just simply add filtration to take care of this problem as well storm water and sewer overflows uh it's the world of our climate change right now right we're going to have a lot more problems with these uh sanitary sewer is not supposed to get in or water and sanitary sewer is not supposed to get mixed storm water and sanitary sewers are not supposed to mix um there's a lot of stuff that goes on under the ground that we don't know about uh things get put in a certain way it might not be working that way for very long so sanitary sewers end up

overflowing into our waterways and contaminating waterways um that's a that's a big problem our people that work at the water facilities today in in our water utilities you know we're suffering from everybody else um we we've got the the gray hairs that are getting to the point where they're ready to go um we don't have good knowledge transfer so somebody that might know where all those valves are that are buried in the streets and roadways all around your town when he retires she retires we don't have that record anymore um and that you know unfortunately the pay is low that used to be an industry that had a lot of great pensions um a lot of great backup

Retirement Systems but as cities continue to struggle financially they continue to cut those programs as well so staff turnover and staff reduction is a big problem that leads us to funding so at most utilities it costs more money to make water than what they're charging you

for make that up volum right well yeah because most utilities have a 30% to 50% non-revenue water meaning that they make 50% more water than they can sell again if I have an aging infrastructure it's 100y old pipe it's leaking I'm got to keep pressure on that so I got to keep pushing water out of that old pipe so so yeah we make it up in volume yep so again I I go back to we have these buried pipes that we've got to get replaced is a is a very critical situation so unfortunately um our political environment is not letting us raise rates fast enough to cover that so let's get to the Cyber stuff so again

let's go back to um something around the the the the way that this the water facilities work uh so again I can I can attack a a a a local plant and affect whatever that plant feeds if I'm in a water utility that has multiple plants generally there's a way that they can pipe around that um so we're not going to impact a lot now if I start hitting multiple sites um more possibilities to cause failure if I hit the Enterprise level I'm really just hitting Enterprise level stuff like customer billing things like that uh the Water Utility is decades behind other Industries um so there's not much happening at the Enterprise level that's going to hurt us at the

operation level um back to our cost discussion so there are some things that are going on um that are being forced we talked a little bit about things about an electric where the comp there's a one piece that was used across or one vendor across multiple sites well that standardization effort is also a giant cost reduction effort so that's something that we're going to have to address that that I hadn't really thought about till till that was brought up but you know spare parts and keeping things running and I can only if I only have to train my team on one thing instead of 17 different platforms I'm going to take that risk um so standardization is a big component

of that so unified architecture is going to potentially open up more challenges that we're going to have to mitigate um fortunately I think we've got that problem solved so we just need to copy that if we can get over our our problem of not invented here syndrome but on the operational technology I think the thing that's keeping us the most safe is the gigantic technical debt that we've got um I'm still working on controllers out in the field that have been there for 30 years 40 years um these things don't even have letters in them to to hook up to the Internet to an ethernet cable I don't I would I don't even know what the protocol is

that I'd use to do it um so we're still pretty protected from that obsolescence however as things continue to move forward the automation industry in of itself is struggling with finding new talent and those and the talent that we are finding is is not needs to learn some some fundamental foundational issues around keeping things secure and that you can't just plug everything into the internet and and things of that nature that are going to potentially create this this problem this tsunami of a problem that we're trying to modernize our facilities and we're making them more less secure as we as we reduce our costs and so there are some challenges we're going to have to deal with there um there's also

a terrible Trend going on in our industry right now in the automation industry of migrating to newer platforms to fix security holes um so you know I can go take a 30-year-old controller and I can copy it and convert the code and put it in a brand new hardware platform that's modern but I didn't fix the code the code is still 30 years old and so the way we would do things today is not the way we would do things right not the way it was done before so one thing about most networks most utilities water utilities is we cover Geographic areas of thousands of square miles in one utility well we have to talk to all those sites and we collect

data off all those sites and that's done with old radios Old radios yeah um and and you count you pick throw somebody throw out a spectrum and I'll tell you utility is using it anywhere from you know I got some stuff down on the 100 megahertz and oh this is great it flies through trees you know no problem um all the way up to 450 900 um I'm desperately trying to get everybody switched over to mobile um um to pass off that that they can't they can't work on these systems anymore anyway so they're just as risk there then turn it over to Verizon and letting Verizon handle the Cyber component of it um most of the stuff is being fixed at

the component level they're finding Joe blow's garage it'll get his osilloscope out and figure out why the radio doesn't work anymore and replace that transistor um and of course every radio manufacturer's got a pathway to a new piece of equipment but it's still got the same problems right right and we are not going to pull fiber out to the middle of some field in the middle of nowhere um that's that cost is too great we also suffer suffer terribly from the shiny new object effect oh I have this problem let me Google it oh this piece of software and this piece of Hardware will solve that problem for me let's put it in I can buy that on my

credit [Laughter] card are you going talk about that talk about how that is not authenticated oh on those radios I don't think encryption existed when those radios were built they're old they are old oh yeah I can go out I can probably walk out here with a with a omnidirectional antenna and get on the Las Vegas's Network um it's scary but what are you going to do you know you got we were working with a Virginia utility that had 1,500 sites they were pulling data off of well so so the other thing about the the water industry that I've learned is they're they they like to bury their head a lot they like to claim

ignorance a lot so a lot of times when I'm delivering an assessment report I have good news and bad news you know good news is we're done with the project bad news is is now you know everything is wrong you can no longer claim ignorance and so the liabilities on this stuff and we talked a little bit about cyber insurance at the at the lunch break you know cyber insurance is becoming a real problem right now we have utilities that aren't able to get that insurance because they can't meet some of the minimum requirements like who's going to ensure you when you got this list of vulnerabilities like why don't I just write you a check now

yeah well true I I guess but you know regulations without any Authority or without any uh um ability to to manage and audit it's just a regulation I got I can go to 500 food I can go to 500 food companies today that still aren't still aren't applying the biot terrorism Act of 2001 what's the SEC got to do with a utility it's not a public it's not a public company so I don't know what they I'm going to look up that though I'm going to talk about that right these are these are city-owned municipal agencies there's there's billions of dollars of assets right buried underground and in these utilities um I'm hopeful you know cost

are driving standardization so I'm hopeful we're going to work something out there but I'm also is anybody familiar with the concept of digital twin so the one thing about operational technology is is that it's very predictive I can look at a network and tell you what's going on right it's very predictive and I'm hopeful that the digital twins are going to solve several problems for us one it's going to give us a training platform so like a flight simulator later right two it's going to give us the ability to look for Trends and find out where are things going where it should be happening how should this be working um and so I'm looking for a lot of those

things to start work to start falling out but it does cost money to make that stuff work and so I'm looking forward to that Trend coming soon so on to our mermaids um the good side our Trends uh in use as our population continues to go up our trends for use are is going down so we're getting smarter about how we're using water we have cities like uh I forgot your name already sorry Travis is working with cities to reduce that water use um conservation again that it's it's a really funny business because we want you in in California we mandate you to reduce your water consumption but that's how I make money so okay

okay I can't replace that Revenue I can't don't use the product that I'm trying to sell you so I can yeah yeah but I can't do that because the public the Public Utility Commission is not going to let me do that so um and then from a resilient standpoint so I don't want anybody to walk out of here screaming for the hills um you know the the good thing about water is that gravity is our friend it's a very resilient product um there is not a lot that you're going to do to a water system that can't be recovered from Fairly rapidly um production process is pretty simple we're going to take a bunch of water and we're going to put it

in a big vat and we're going to let it sit for a day and then it's bunch of stuff is going to settle out and then we're going to say yep we'll add a little chlorine to it just enough chlorine so that when it gets to you the chlorine is gone that's pretty much the process there's not a lot to it there's some little bit of filtration we want to get the contaminant out not contaminant but the uh the parts out that that coagulate the sediments but that's what I say we don't do a lot to water and so when we start putting stuff in water that we can't get out of water without a lot of

stuff that's why desalination hasn't really taken off is that the energy consumption to desalinate water is far too expensive now eventually maybe they'll figure it out there's some obviously along the west coast or some plants that are starting to work that out um the distribution process is just as simple it's gravity why do you think you see big towers of water towers up there's these things are all over the place we keep those full we keep your pressure on your lines now as we say we don't one of the things in the automation industry that we don't do a good job of is we don't do a good job of of cyber securing our instrument mation so can can devices be

hacked yes um have we planned for that when we design the system no is that a problem the other part to a lot of these things are that that we have a lot of physical things built into these systems that will prevent things from occurring that when things go really bad um overflows things like that but when you get a situation like uh was it Lewiston I think the beginning of the year that that topped a a dam and and knocked out a reservoir um you know again that wasn't a cyber incident that was a a dumb programming incident um you know things things that are connected to processes are supposed to change when things don't change there's

a problem so that has to be some programming that's in there and that's where my a lot of my concern on the newer Generations coming in that don't think through you know as I always say any idiot can program for normal um I don't need I don't need anybody special training for that um government I think is finally starting to recognize the criticality of water uh I know that sounds weird but you know for for decades we've we there's a group of us who've been talking about cyber security and water and it was just uh earlier in I think it was March this year the EPA came out with adding on to the sanitary Serv

survey that they were going to start actually measuring utilities on that the utilities all went crazy and said no you can't do that blah blah blah even the professional societies said problems with it I don't understand that sir so I like the idea getting into this you know government is recogniz and I just wanted to you know just mention that in Europe we getting we're getting regulation called this two and oh I'm sorry I forgot about that no it's right 15 minutes oh great perfect so in Europe we're getting NIS two which is a regulation where we're trying to look into supply chain dependencies for being able to deliver critical services and water is now being part of that

umbrella so being SE of something that you know you'll be regulated on that you weren't regulated on before but what is very interested is that other companies that depend on water in their supply chain like factories you know manufacturing Med medical blah blah blah they're going to be measured on that and that drives me back to a question you said you know we cannot increase the cost of water but I think we need to get to a point where we understand what is the dependency and how can that drive you know the cost the cost of securing that supply chain right because right now we're saying you know water can cost more than you know the dependency 2%

what's the un's 2% of I can't there's a number of whatever right but but we're not seeing that you know bringing that to you know the multibillion dollar manufacturing companies that are depending on water they you know we need someone to pay for that cost yeah are you volunteering yeah I got a little bit take up the take up the Passa hat um that's a great point and and so you know I look at things like that and say What's the value of our water that we're supplying you treatment treated water if you had to go out and build your own plant for that you should be paying a little bit of something different right

um so uh talking about government recognizing that we've had a water iack for a while um it's been horribly underfunded um and they really just did a relaunch I think um we were one of the first luminary is one of the first water ISC Champions um so I I I'm I'm hopeful about that there's a lot of effort going on in the industry in the water industry and in the automation industry um with different groups that are really driving out a lot of these things um and I think a regulatory agencies are finally starting to engage so we had talked about the EPA well that you know that got shut down by I think Missouri

Arkansas somebody else all the all the the really Progressive States um decided to to step in and shut that down so um you had a question yeah um thank you this is a a really good wide ranging talk they filled in a lot of blanks for me could you talk no really each one of these topics could be a you know its own conference we actually have those um could you talk about where chip fabrication fits in so miss semiconductors yeah so uh like Columbus Ohio has three water plants and Intel announced that they're going to build another Factory there and to supply and and like an average ship Fab or at least that size uses of a small City so

Columbus is going to builded about a fourth one just it's like 145 million gallons just for that plant and um and who's paying for that I I that up I wanted to ask you because there's the chips act yeah right and there is there is incentive there is money to do that and um and because there is such a focus there there isn't an option to build it someplace else yeah so like in Arizona tsmc is building their their huge thing and they're they're really pushing forward I think at least but this is what I want to hear from you on recycling their water they're like 98% so do you see reason not to do you see

that as an opportunity do are you optimistic about that there's there is this there's so many imperatives pushing it Forward some of the things you talked about it's not going to be rid of overnight but there's an opportunity for Innovation to kind of lead the way on some of this absolutely and that's where that's why I talk about this in such a broad range because this is kind of like the magical electricity where you just flip the switch and it just comes on right you turn on the tap and there's water there but water's got an additional Challengers you normally don't see it you have no idea that your water supply how many who knows where

your water is coming from well that's a terrible example bunch of educated people um but those are the things that that that the the public just isn't aware of and so we've got a huge um public relations problem because our public our users don't even understand how they get their water right but you know something that Columbus could be doing is maybe they double the size of the plant and shut down an old plant you know there's just stuff that could be done around that um that I would be looking at as well but the reuse the the concept of reuse has to be something that is looked at and we've just got to get over ourselves on that I mean nobody

wants to do it you have 10 minutes thank you um the other thing about the water industry is I have I have yet to meet a group of people are not more focused on making this place run than than the maintenance staff and The Operators inside of a water utility both on the oper you know it's it's like a personal mission that they've got and they have done things that are simply amazing with junk um there are Motors out there that are a hundred years old and they're just cranking away um these guys just make stuff work I can wait till you're done the good slide so so anyway kind of wrapping things up um on this slide so cyber is a

risk it's something that we have to be aware of it's not a big a risk today as it will be in the future so companies like me my company and and others are working really hard to make sure that that we put systems in that's what the topic on the S bombs was so interesting to me it was like I'm not going to be able to accept just more data and more problems it has to come with a tool that's going to monitor itself and so the systems that we put in today of water utilities are fully automated monitoring um you know so there's an opportunity for actually the watered industry to Lea frog most Industries because we aren't

going to have the staffs but we can spend money on technology and capitalize it and then the big part that I'm doing right now with a lot of our water utilities is there's a massive convergence opportunity that I see between the the operational technology the informational technology the engineering technology so in in a water utility there's a tremendous amount of engineering done around hydraulic models and just the documentation um and and then the what I call the security technology the St I've got physical security at all these sites most manufacturing facilities most water utilities security is its own little box over here but it still uses the same servers the same wires the same everything and I don't understand why

those aren't sharing I you know I've got I'm already doing a lot of work together so we're working on bringing and collapse and all that stuff together um I no longer I'm the of the the proponent of physically separated systems we have enough technology now and enough monitoring capability that I can load everything onto a couple of servers um and and monitor that traffic so I see tremendous opportunities for cost reduction and Manpower um better utilization of Manpower around those systems because I can have now I can finally get to a point where I only have one Cisco switch guy instead of I have a guy in in security that knows switches I have a guy in it that knows switches and

I have a guy over here in OT that knows switches Nobody Does it whole time nobody's really good at it we all hack into them and we create more vulnerabilities well if I can hire one person that does that all the time I'm going to be in a lot better shape so that's the good side um we'll wrap that up then with any other questions Josh um I've been stuck on a point you made that water is incredibly resilient I think for certain things like you know emptying egregious levels of lie into the reservoir can correct fairly soon but I would think and maybe you can make me sleep better at night that there are some harder to recover from

scenarios so something like um we saw quite a few large farm irrigation systems online on showan with hardcoded passwords like if you drain the supply in a in a scarce area or you overwater that could be a harder recovery time but also I was more concerned of pressure monitoring like could you introduce strain that could do damage lasting damage to underwater pipes or like are there any lasting there could be right so we've we've talked a lot about and back to the the Jackson Mississippi um concern um I've I've worked on algorithms where if I can if I can tighten up the control I mean understand we're dealing with some of these pipes are are 48 in Main right you're not

going to control that down to a half a pound it's just not physically possible so I can tighten up control but if I'm if I'm measuring with a yard stick I can't control that with a micrometer right um so so there are concerns about that but also a lot of these pumps are are uh centrifugal type pumps so I'm not going to be able to really overcome like there's there's a there's a minimum there's a maximum that I'm just the pump's just going to stop running or stop working right um and those are the types of like I said there's a lot of physical things that these systems are built around now stressing any kind of

an old pipe you know and that happens during the big flush and different scenarios that I just there's just physics that I can't overcome right I got a thousand miles of pipe and I got one source of injection that's why a lot of times we'll work off a gravity where we can take that that the mains will feed big tanks and the tanks will feed the distribution system so um so yes and and yes I mean there are going to be scenarios like you it unfortunately a lot of this material you can log on to any site and really get you know these folks don't keep track of where their contractors are storing their data you

know that sort of stuff down the supply chain of where all this information's sitting so somebody like me could open up a few documents and figure out how they're running something and and get in and and work it out and you know take care of some stuff but yeah okay we've got time for one brief question one brief question before we wrap it up so um in terms of the resiliency angle for water sector uh we've had conversations in the other sectors around how automation could be dialed back in an event of a business continuity event or a disaster recovery event how do you see that in the water sector are are we is that a pipe gream

or is it possible for these kinds of initiatives that knowledge transfer so the one thing when we design it always amazes me when I walk into a a water system a water plant I I walk into these control rooms and there's dials and there's buttons everywhere some of them work um but they always tell me oh you know if if we ever lose this here skat system we can go back and start running it I like I said well I'll go pull the plug you can show me nobody wants to do that because the problem is we don't exercise that right we don't exercise it so we don't write it down we have no idea there's nobody

left at the utility that's ever run doesn't even know what that switch is for because the label's gone so so yeah and he retired right so so can it be done yes and again that's where I'm excited about the digital twin concept is because we can start doing that knowledge transfer you know at the end of the day it's just a bunch of pipes and pumps and valves I can figure out how to establish flows you know if they if they shut the power off I'll bring her in a generator it's just it's not you know weather is a bigger risk um than than somebody attacking my system and I don't treat that very well so how the hell am I ever

going to spend time on the Cyber side of it but those sorts of things when I start to solve for tornadoes and hurricanes and other stuff I automatically solve for cyber so when I say cyber is a great unifier I can go get money for cyber dress it all up make it look pretty get a bunch of money but I'm actually solving a whole bunch of other problems so I look at I look at the Cyber um front as a great unifier for us to solve a lot of money problems inside these utilities because like I said I I'll write some kind of cyber function on it and the board doesn't even want to talk about it they don't want to know

they just how much do you need don't tell me I don't want to know yeah PL plausible deniability all right listen Dean this has been fabulous please join me in thanking Dean Ford for this really fascinating discussion