Securing the Last Mile: Resilient Branch Connectivity with SD-WAN and Satellite by Thomas Dang

Um, that I think I added to my slide this morning that like mostly when I'm not messing around on my computers, I'm trying to mostly push my computers into the river. Uh I think that is the most secure and resilient resilient way you can run them, right? Because if they're at the bottom of the riverbed, nobody's hacking them. We're very secure. Uh and nobody's compromising my data. But yeah, I mean that covers all those things there. We're a big or at YG. Um like like was mentioned, I'm Thomas, but the um YG, we're 5,500 employees, so like small for government, but still a fairly large size or um we have lots of remote campuses, over 250 buildings um that we

service. um every single one of those buildings needs connectivity of some sort. I'll mention this bit later, but when you work in a rural area like that in northern Canada where the connectivity is difficult, um that can mean many different things, right? When you say real connectivity, because I come down here and I laugh because I get 5G plus on my phone and I get, you know, 2, 300, 400 megabytes per second on my phone, megabytes, not even megabits, right? Like it's it's actually very seriously fast. Um and then I go home where they just brought 5G in in White Horse and I'm lucky if I get 70 megabits, right? Like so it's 5G but you

know u but yeah so it's one of those things where what happens when we have an organization in such a difficult remote connectivity area right um it means you're not using those you know very common down here in the south now people saying oh well some branch offices we're only going to go 5G right we're not even going to bring fiber to it because it costs more to bring fiber than just to run a 5G network right uh something like that that that's that's not the case up north we don't do that we do have some LTE uh backup sites but You'll find that typically um in some cases even for example the LTE was so

poor that we couldn't get the like uh firewall telemetry traffic to flow over it. It was it was just too slow, right? Like we couldn't even get the tunnel to come up, right? So so LTE isn't going to to work for us. 5G barely exists. Um so what happens? Well, what happens is it turns out we uh up until this year, last year uh only had a single fiber line out of the territory as well, right? So it followed the Alaska Highway uh went all the way up to to White Horse and and to our communities as well. Um but that single fire line was a single point of failure, right? And and anybody who's

built a large data center or designed a large data center knows you always bring in two providers, right? Or you or more providers even than that, right? Um when you have a single ISP with a single line of fiber, we're coming in. Um you can't do that, right? That that's just not an option. Um and that happens almost annually, you'll have some sort of major fiber outage in the Yukon, right? So, uh I I was joking for a while when I first got up there. said, "Oh, there hasn't been a fiber cut since I got up here, and we're going to build this dumpster fiber loop, and we're going to be redundant." Uh, and it's never going to

happen again. Um, and then literally, I think two weeks later, we had a we had a three-day outage from a fiber cut. Um, and there was wildfires, right? The was the big thing in northern BC. The wildfires actually burned right through the fiber line, right? And, um, the ISP, we're talking to them and they say, "Oh, we can't get in there because wildfire management has deemed it unsafe to get crews in there." Right? So, so you don't know when the connectivity is coming back. you don't know when things are going to uh be able to be even start fixed or you don't know how the extent of the damage. Um and you're sitting there as the government and and at the

time I think I was acting director of the the technology infrastructure and I was overseeing the network team and the security team and and you're sitting there going okay so 911's down because all the phones are routed elsewhere. Um we have no way to email anybody any of our employees. We have no way to call any of our employees. Cell phones don't work because they also back haul over the same fiber line. Right? There's literally no connectivity. There's no payment systems that work in town. So people can't even go and like buy groceries right now, right? So anybody that had $50 in their pocket was basically considered rich because they could go get gas, right? So So this is

the reality. And and and at that time because the fire was on the road, which is where the fiber was as well, um you didn't really had an option to drive away from this either, right? Because the road was closed, hence not being able to do repairs, right? So what happens when when you're cut off from the world essentially, right? Um what happens it turns out in in White Horse is they put up signs that say if you have a medical emergency go to the community center an ambulance will be waiting. Um peace officers will be driving around the community uh and flag them down if you need help. Uh and that was a solution that was in place at the

time, right? So this is a like we're talking about data centers down. Sure, that's bad and like there's some business impact to data centers being down, but beyond that there's real human impacts, right? When you're the government and you're talking about what happens when we lose everything, right? Um and and and those are alarmingly common, right? So it's it's not just the Netflix, it's not just, you know, tellahalth or or things that can be deferred. It's it's it's essential services that have to happen right now. Um when the satellite industry shortage has changed in the last few years, low Earth orbit, um I've got one with me now. This is a relatively new one, the

Starlink Mini. Um you'll see it's incredibly small. Uh fits in my backpack, right? I can carry it around. Powered by USBC. Uh uses like 50 watts max. Um, so super low power relatively speaking. Um, and this thing can pull I've pulled over 200 megabits per second down on it, right? Like uh and we're pushing 20 plus megabits up on it, right? So we're talking about a fundamental change to how we see connectivity in rural areas uh of all over the world, right? So um but particularly in the north northern Canada where we didn't have an option where we could even get 200 megs in some areas, right? Um and I think I have it on a future slide, but like some of our

sites as I was mentioning are are are literally running ADSL lines, right? We're talking 10 meg down, fivemeg down, um one 1.25meg up, right? So we're literally running ADSL on some of these sites, right? So it's one of those situations where this is a a game changer for us in terms of how we want to do connectivity, right? How we want to talk about if there's a health center um in a rural area and fiber is cut maybe just to that community, maybe to the whole territory, we're not one or the other. Um but if there's an outage for some reason, what can we do about it? Right? and and satellite links just open up that option for resilience,

right? So, if you're thinking about as a large or uh that's huge if you're the government, but if you're even if you're a smaller or a different type of or um who's saying like we want to make sure our business operations continue and it's going to cost us, you know, however many $100,000 or a million dollars for every day you're down. Um those are all impacts you can start to mitigate using the satellite technology, right? Um but yeah, like we have name a technology uh and then name some that you don't think are in use anymore. Uh and we probably have it at one of our sites, right? So, we're we're running MLDDS across many of

our sites uh for rural communities. Uh single provider fiber like I mentioned, we've got cable internet, we've got fiber internet, we've got ADSL, we got LTE, we got uh there I think there I pulled I pulled a hub out of a out of a site uh last year. Our team pulled a hub out of a site last year. It's a 10 meg hub and I said I didn't I'm shocked that this site is functional considering this piece of hardware is in here. Uh I'm shocked this piece of hardware has not failed in the last 10 years. You know, all sorts of things. But um the challenges are in rural areas, there's going to be extremely high costs too,

right? So when we talk about how do we connect communities, we have one community called Old Crow. Uh for those of you who don't know, Old Crow um is routinely one of the coldest sites in the world uh in the winter times. Um so it's winter here obviously summer in Antarctica, but it's it's the coldest site in the world uh that's measured by Environment Canada. Um often it'll be minus 50 for you know a week, two, three weeks at a time. It's a flyin only community. There's about 300 residents. Uh so previously we would pay five figures for a uh one web installation uh in that site right so we we bought our own satellites they're the size of you

know uh this whole row of chairs right and it's getting us you know 150 down 50 15 up 50 upright whatever so it was it was five figures very expensive latency wasn't that good um and it was fragile right like we we found we actually had issues with it where if it was really really cold we would lose connectivity um the provider said oh it's a problem with the install something with your wires. They sent somebody up, tried to fix it, didn't work. Sent somebody up again, tried to fix it, didn't work. But basically, every year we would have like several days where we'd lose connectivity to this whole community. Um, even though we had a satellite link

that we were paying tons and tons of money for, right? Um, and that's just the reality of of what we see in some of these very remote areas. And then I went up there um this summer to to go in and do our sterling installation which we did uh for very very low cost cuz these things are a few hundred dollars each instead of um thousands and thousands of dollars, tens of thousands of dollars. Uh and when I was up there, you know, every single house, every single residential house and the municipality all already had Starlink on all their every single house. Some houses had two sterling actually. Um like it was crazy to see the change in connectivity in

these areas that's happened in the last 6 to 12 months, right? Um, and there's going to be more providers and and we're excited about more providers, particularly as a government because we we want to be investing in Canadian technology if we can. So, we know like Teles's coming uh in the future that's invested by government of Canada and and based out of Quebec. Uh, but we we know that other technologies are also coming online in the next year or two. and and and we're hopeful that that's going to keep driving prices down but also be able to provide us more resiliency to things like geopolitical issues or um or just you know satellite arrays going

down for whatever reason right outages just like any other provider u but those satellite connections the previous ones that we were buying had higher latency right so they they're very latency sensitive the coverage wasn't as good right so if um uh for example the cellular service in or the phone service I guess in old crow is all run through the one web and uh Cband radio installations And at certain times of day, if you talk to the residents there, they'll tell you like you can't make a call between 2 p.m. and 6 p.m. because there's a bunk satellite that flies over um and we have to wait till the 6 p.m. one comes around and then and then and

then you can get a call again, right? So we we see less of those issues as we move to these alternative LEO providers um these LEO satellites and and we bring in things like VOIPE and we bring in things like Teams and we and we bring in more modern technologies that that just use IP transit, right? We don't need all this old legacy DSL stuff. We don't need all this old legacy um phone lines, copper, whatever, right? We're just going to do LEO. We're going to do IP connectivity and and make it work for the people that need the access, right? And that's our priority, right? So, I guess the the as the name of the talk

implies, let's talk about SDWAN, right? Softwaredefined WAN. Like what is it? Um it is actually in a sense quite simple, right? It's it's essentially an overlay network where we can dynamically steer this traffic based on source destination uh different protocols whatever uh and we can base based on like an SLA or a metric where we measure things like the latency the packet loss the jitter whatever right we can measure those things we can determine which one is the best link based on the things you've decided and then say okay so today Starlink's working really well we're going to send all of our VoIP traffic over Starlink next day oh the copper is working better we're going to send all

our traffic over copper right or whatever right next minute, next 30 seconds, whatever. Right? So, um there's many products out there that do this, right? There you got you got I mean there's like five vendors out there, I think, that do this in in the hallway here. Um but uh they're all some version of dynamic routing, right? Uh with a with an overlay network using typically some sort of tunnel and then an underlay network where it's your your transits, right? So having a fundamental understanding of networking and how these networks um work in general I think is really important when we think about these deployments because and I'll get to in a second here it's it's

supposed to be branch connectivity made simple and there's some air quotes there um and the the reason it's not quite so simple is because oh I didn't it's a slide next one but the reason it's not quite so simple is because it it takes care right like any other network it takes somebody who knows what they're doing uh to take care of it but I I mean securing the network edge is um something I wanted to touch on really briefly here as we get into this but it's big questions right like do you trust your ISP because when we use our MLS circuits obviously that runs over our ISP network when we we have a they

call it a eman a managed uh it's a managed municipal area network for us where they span some vlans across the whole municipality that's also over their hardware over their over their fiber over their network right do you trust your ISP in the in in the US just last year we saw that major telos Verizon AT&T lumen were all breached by the Chinese, right? So, so we can see APS are targeting Telos specifically and beyond that, do you trust your telco in terms of are they going to keep you up all the time? Is their uptime SLA as good as they say it is? Um, so, so there there's multiple factors to consider beyond just connectivity, but also the

um integrity of your network and the confidentiality of your data, right? Uh, I mean, people talk about zero trust and bit bit of a buzz word these days of course, but we really are serious when we say we should have encryption in transit, right? because encryption in transit prevents if there is something like a compromised ISP uh from being able to observe your traffic, right? So, what we decided is we went with a mesh VPN instead of like a a traditional hub and spoke for a few reasons, but um a lot of these providers will now be able to do um they call it ADVPN, autodiscocovery VPN. So, it's you can split your traffic uh to each site

directly uh and it will basically create a mesh of uh tunnels between all your sites. So if there's like site A is connecting to site B, it doesn't need to go through your hub first, right? So um that reduces I mean for a lot of our sites as as a government, most things are coming back to our data centers, but it does reduce that impact uh on on your on your links, right? And and when you're talking, especially for satellite links where you only got a couple hundred megs, that could be a big deal, right? You you push 10 megs off that that could be the difference between somebody's team's call stuttering or not, right? Um, but yeah, so when we

talk about these things, it's important to also consider we also do a lot of inspection on on these things. Uh, oh, my mouse is in a weird spot here. Oh, uh oh. Broke it. Broke it already. Can't see my notes. There we go. uh but we do at least like SNI and IPS on all our east west flows as well right so uh in some places we are doing full inspection it depends on the hardware we have available the risk of that site a a lot of things right we're not going to be uh we're not going to be overprovisioning in sites that only have one user let's say but but in some sites we're going to be doing full inspection

and that allows us to get enhanced visibility as well right so when you think about a traditional if you're if you're purchasing a man um from your ISP if you're thinking about purchasing an MLS circuit from your ISP um whatever between your sites. Traditionally, you basically put a router on each end and then it routes and the traffic flows and you have no idea what's going on, right? So, that's not very helpful if you have to do a forensic investigation after the fact. That's not very helpful if if you have some sort of um something going on and you're going like which computers were talking to what, what type of what type of uh intrusion were we seeing. Um, so

we we we do turn on all of our east west stuff uh in those sites that we're deploying the stuff now and and it provides us that actual forensic stuff, but it is one of those things where you you can at least if not limit the brass radius, you can at least go back and see what it was, right? So um that's helpful to think about as well. Um but like I alluded to, you don't just buy a bunch of hardware and expect it to work, right? So all the vendors will tell you that they you do. Um and my apologies to any vendors in the room. It's it's it's not true, right? Uh everything requires some work, right? We

did it all in house. Uh we have a we have a small but lean team in in in my organization and we have one or two people who very quickly got very familiar with the technologies and a lot of the shortcomings. Um but you can also go proer, right? You can also hire somebody go in and deploy it for you. Saves you a ton of time, saves you a ton of heartbreak when you break something and are on the phone with TAC at 2 a.m. trying to figure out why the house hunter's down. Um or or whatever. um risks of course that you'll have to you know either keep paying the provider to maintain it for you or do a transfer

whatever right but that's one of those things where u I like to think about in terms of do you understand your own network right and and that's a good question to to keep in mind because for example when our fiber was cut uh last year for those three days um I went and the first thing I did is I had um satellite at my house of course because I'm me um and I uh used Wi-Fi calling and I tried to call the knock, right? So, I said, "We have a knock line for the ISP and we're going to call them and be like, "What the heck? Why is my network down? Is it a cut? How bad is a

cut?" Right? All those things. Uh, immediately the phone line was dead because their knock was on their own network and their network was dead. So, I've like skimmed through all of my emails, which I could do because again, I had a satellite. Um, and it was Outlook 365. And I went through all my emails and I found somebody's cell phone number and I like tried five different phones before I finally got somebody else who must have had a cell phone on a Starlink or something um at home and I talked to said, "Oh yeah, our knock is down. All the phones all the phones in our knock are down. I can't even give you a ticket number." Like that that was

the that was the problem that our was running into at the time. But that's a situation where when you start to think, okay, what redundancies do I have in place? What resiliency do I have in place for my network? That's where you say, "Oh, if you're using a managed service provider and you can't reach them because your phones are down and you don't really know how your network works and you don't know how this SD1 thing is configured, that's where you're going to run into big problems, right?" Um, so we had a site where we did have a dual WAN system and that was the old crow site where there was a one web and

there was a backup Starlink. Um, but we had a managed service provider doing that work at the time, right? We we've since transitioned to bringing it in house. Um, and this is one of the reasons why is actually it's because when I finally got through to them, um, they said, "Oh, we think it should be up, uh, because it, you know, has the two Ws, so it it should be up, right?" Um, turns out a day later when I talked to them again, they said, "Actually, it's been down this whole time. We messed up a firewall rule." Uh, and we've been bl we've been dropping all the traffic at the second WAN, right? So, we had no way of knowing that. We

were cut off in the world. But if we had our own systems, if we had running it ourselves, we'd be able to see that immediately, right? You look at the logs, you can see the packets being dropped. we'd go in and we'd fix it and we'd have brought at least one site back online. Right? So those are the types of situations where it is important to think about yeah the upfront development and training time for your team might be higher but it is important to say what is the impact if your team has no idea what's going on and you're trying to phone three different vendors, four different vendors on the phone with TAC. All those things those hours add up in

the crisis time and and that's not the time you want two three four different people yelling yelling at you while you're on the TC call, right? Um but yeah like like I said we have all those connections and all those things. Um but the limitations we ran into are it's not set and forget and to expand on you need to know and understand your network. It's even things that you don't think about, right? Like, are you using internal DNS? Most organizations, if you historically are using an MLS circuit or a man or whatever, you're probably using your AD for DNS, right? So, when you decide to switch to something like this where you have a dual WAN uplink and

you're running multiple tunnels and things like that, what if your tunnels go down? So, you expect your your internet should continue to work, right? That's kind of the whole point. Um, the software defined wide area network, right? So, you're you're defining that the w the LAN the internet will continue to work for you even if your internal connectivity is down, right? That's kind of the point of the of the resiliency project. Well, what happens if you're using ADDNS and your main site's down because the tunnels are down? Well, now none of your clients have DNS and they can't get their emails or Google or whatever, right? Um because you pointed it at an internal server, right? So, you

better have a DC at that site or you better have them pointing at a at a backup uh DNS of some sort, right? That that's local to that to that site or internet DNS or whatever, right? So, so there's there's lots of gotchas that you don't think about until you deploy the stuff. something goes down and you say, "Well, it should have come back online. Why didn't it? I'm seeing traffic. How come the users are reporting no internet?" And then you realize, "Oh, we didn't change our DNS records, right? We didn't change the DHCP scope or whatever." Right? Like, so it it's things that you don't realize until you're trying in the middle of a

deployment to say, "Why didn't come back on our testing? Why didn't it come back during an outage? Uh why am I seeing traffic, but it's still users reporting down?" Those types of problems, right? Um and and and everyone will say, "Oh, okay. Well, you just need to know your network flows then, right? you need to understand what's going on in your network. Well, let me tell you, we're an org with like 12 departments, 15 departments or something. Um, every department's got admins. We've got central admins. We've got my team. We've got, you know, other departments that have their own teams um as well and their own service desks and everything. Uh, and now you're trying to figure out,

okay, so you're telling me you have these 10 applications, they all run on the internet. As long as the internet works, they're fine, right? That and then that's what they tell you, right? and and then you say, "Great, it'll work when we when we do the switch over." You do the switch over. It turns out they had some weird thing that used an IP based authentication um that needed your your main site's IP address uh and you didn't do that because you're like, "Well, I'm going to split tunnel this traffic because it's going to be uh way more efficient for us." And then now you've broken some critical workflow, right? So even if you think you've got

your net flows mapped out, you know where everything's going, you're going to find edge cases that are going to impact business uh when when you do the cut overs, right? like and and that's just reality of any large uh project like this and and and that's another thing that uh if we had to go back and do this again I think we would uh there wouldn't be an easy way to do it. Um, but it would involve, you know, being able to understand more closely like what are every single user actually doing? Like what what websites are they going to every single day, right? Like can we get a list of that stuff from the

departments, right? And and they're not going to be able to give you a good one. But it but it's much better to be able to say, well, you gave me a list and you told me what the requirements were and I met all those requirements and it still failed, so it's your fault. Than it is to go in and say, well, I thought it should work and you told me it would work. I told you it would work and it didn't work. Right? And then now you're trying to figure out who's to blame, right? So it's much better to try and get that inventoried ahead of time and we would have gone deeper on that. I

mean, we were in a time crunch because of people were uh very alarmed after the significant fiber cuts, right? That we had to move very quickly. Um but but those types of things, we had to go back and we had more time. We would we would do a much better job of inventorying some of those things, right? Um but going back to my understanding your network and understanding networking in general. Um the way we implement it was using Forigate uh Forinet uh SDWAND uh partially for licensing reasons, partially because it was uh what we were able to uh deploy very quickly and learn very quickly. Um but it requires a strong strong understanding of things

like BGP. It requires a strong understanding of things like routing protocols um and the specifics of your SD1 vendors uh implementations, right? Like how do they implement their their SLAs? How do they implement um routes that are going to be removed or added dynamically um from your hub site? Do you use BGP and then you have to figure out are we going to drop one of the sessions or whatever? Like you have to you have to figure all that stuff out um on on on on your side and understand how how that network is going to continue to function, right? And that's where if you offload it all to a third party and you say third party unlet for me, third

party run it for me. Third party do everything for me. Um then you're going to be hands tied when you have a real crisis, right? uh and you're going to be waiting for third party when you're when you're in a real crisis, right? So, I I think there are significant risks to that approach. I mean, I can see why if you need something fast, it would make more sense. Um but it is the the the in-house training there there's no replacement for having at least some good understanding of what's actually how your network actually runs, right? Um uh let me slide down my notes here. Yeah, I mean I think the the important thing to know is that you need to

document all this as well, right? because we have, you know, uh I got a picture here, uh, you know, 15 sites, whatever, that we're going to run this stuff to, right? Uh, and that number is only growing, right? Because we have these sites that we say, okay, we deployed it to 15 points of presence in every community. Um, the 10 points of presence in every community, but it turns out that the ADSL online cost more than just buying a stling would in an ongoing onm way anyways. So, like, why don't we just kill the ADSL and also put a straw link there? Uh, or why don't we kill the copper and put a straw link

there as well? uh and that number just for us it seems to be keep growing right people are saying well why are we paying for these legacy technologies that are slower that are not as uh that are not as reliable and causing us more grief right so um we still see sites go down all the time right our failover systems now kick in regularly more than once a month we see we see the sites go down usually the internet stays up because our system is working the way we designed it sometimes it doesn't right and that's when you have to go and say like did I mess up the DNS, did I mess up, whatever, right? Uh we usually begin

to like actively monitor when we see these links fail. So, we do we'll send an alert to our ticketing system. Um sometimes it's me that gets woken up or whatever and then, you know, you start opening a ticket with your providers and you start looking at the routes and and everything and saying what's going on, right? And figuring out what's gone down, what's staying up, right? But when the internet's not working or it's up, it doesn't always mean what you think either, right? Because right now, we we did a big project a couple years ago where we moved everybody to VoIP, right? most sites move to VoIP, right? So even if your VOIPE is up um but the

community has been have their network cut, right? So let's say I don't know if you're in I'm going to name a random community pharaoh in the Yukon or whatever, right? Let's say there's a fiber cut to that community. So let's say you were able to keep the whole government network up because you had the redundant satellites, you had the SUAN, everything worked exactly as expected, right? It was perfect deployment from the government side. um we're still going to run into problems because sure all the VoIP phones work but the cell service is down. So who are you calling, right? Like who who who's calling you or who's calling 911 and how are they going to getting through to to

the services, right? So so just because you're able to deploy perfectly on your side doesn't mean your clients, your customers are also going to be able to reach you, right? We we're looking at different options for how we deal with that. I mean our satellite links uh we don't have as much bandwidth as we would like and obviously in most these links so we can't run things like all the government websites over them. We also don't have BGP peering with Starlink right so we can't run our own IP addresses over over those uh types of things right now we're looking at some alternatives to that whether that's using something like a like a cloud uh

CDN or a WTF that will front our services for us or we're looking at um getting an ISP in in a different location that will uh peer over our tunnels to something like that. Um, but then there's limited bandwidth, right? So, if you're writing the whole Yukon.ca over, you know, your 200meg Starlink line, that's not going to work that well, right? Um, you're going to get some real traffic congestion issues. Um, but when we look at these types of things, it's not just a matter of your internal services, right? Because your internal services will stay up, but people still can't reach you, right? So, it's this thought of we can prioritize the things we need to prioritize and we

can do everything you want to get as resilient as you can, but there's still going to be factors outside of your control, right? Um when when we had a cut last year, they told us it's a once in aundred year event. They're never going to see a cut this bad again. What had happened is uh there was the primary fiber cut, there was a secondary fiber cut um up the line a bit as well, and then there was a third route that they sometimes use to flow traffic that also got cut, right? um that went to NWT and and then and then then that line also got cut. So they said, "Oh, like that's a one in a 100red-y year fire event.

It's never going to happen again." I got three emails this summer from the ISP saying all three routes are in at risk right now, right? So like, yeah, once in a 100 years as in they're going to tell me next year that that they're all at risk again, right? So there's always going to be things outside of your control. It doesn't really matter how many times you implement different solutions, how many times you uh put in these resilient systems. Like by some fluke of chance, we could lose Starlink on the same day that the fiber got cut, right? like and then we'd be screwed, right? We'd have we'd have no resiliency. Um but what happens is it

doesn't mean that um it doesn't mean you did anything wrong necessarily, right? Someone's going to be yelling at you and thinking you did something wrong, but um it means that we have to plan ahead for as many of these options as possible, right? So, so the lessons learned here is really that we can do everything right um and your department is still going to screw you when they didn't realize they had a specific workflow. You can do everything right and you're still going to be screwed because the clients can't phone you anyways. You can do everything right and you're still going to be screwed because both providers that you selected are going to have an outage at the same time. You can

do everything right. Um and Cloudflare could go down and your website's down anyways. Uh you can do everything right. You can get DDoS on the same day, right? So your your your 500meg links that you've aggregated or whatever suddenly you're saturated, right? Um you can do everything right and and more often than not Teams will have an outage, right? Like how often does Teams have an outage if you're a Microsoft shop here, right? So um that could happen any day of the week. Um and and those are the things that we just aren't able to account for. But the point of implementing something like this is to say business continuity is important. U we do everything we can

particularly as a government. We do everything we can to make sure it's going to be as stable as possible. Um and we we try to provide the best possible experience for our department clients and for the public, right? Um but that doesn't mean that we control, you know, acts of God, acts of nature. We don't control um all the outcomes but we do control how we react to them and we do control how we prepare for them right so as a business if you're thinking about that it should be how do we prepare for as many reasonable eventualities as we can right how do we prepare prepare for as many reasonable outages as we can we selected this

solution because of multiple reasons not not the least of which was cost right like um this is a much more cost-ffective solution I know uh my partners in one of the other territories um government of none of it uh went in on a full enterprise Starlink array, right? So they get a 10 gig symmetrical link from Starlink that they BGP peer to and it's their primary uplink. Um, but they're paying uh I think it's upwards of a million dollars a month for that, right? Like so it's it's they're paying significant amounts of money for for a link that we're not thinking is uh feasible for us when we have a fiber uplink, right? We're saying we only need

it in backup connectivity situations. So, we're not going to make this massive investment that's, you know, millions of dollars in hardware and then millions of dollars in ongoing maintenance um because it's not it doesn't meet the business need for us, right? The business need is to have uh resiliency for these brief outages that may last up to a few days, right? Um but when we talk about other organizations like Nunva who don't have terrestrial fiber to their to their uh sites, then that starts to make more sense where they're saying we don't before this they only had one web, they only had a few hundred megs um or maybe up to a gig. Now they're getting 10 gig, right? So, so

that's the type of decision that different organizations are going to have different answers to, but as like for us for as government, we have to make the best possible decision for for what resources we have and also what the the time uh the timelines will allow us to implement reasonably quickly as well, right? Because when there's an outage and suddenly every single minister is saying, well, how come my department went down? Well, then you have to have an answer for every single one of them, right? Um and and it's funny you you you'll get some interesting stories by right they'll say things like well um you'll get the argument that they say for liquor stores for example say well

liquor store payments going down doesn't necessarily sound like a critical service right you're saying that that can go lower on the list than than my getting the health centers online first right or whatever it is right nobody's going to die if the liquor store is down um and then you realize oh but like in terms of the business continuity reasons it's actually fairly high on the list because taking a liquor store offline for a day the revenues in that one day will offset the entire years installation for that site, right? So you think about the business implications um and and there's many things you have to consider when you when you're trying to deploy these

things, right? So so for for us it was a lot of lessons in how does all these disperate pieces of the network have to work together? How does it all have to um play into the objectives of what a government trying to deliver? Right? I guess I should make it clear. We have government run stores in the new fund. But the uh not like here in Alberta, but um when when you're looking at all these different pieces of of infrastructure, they're all competing priorities, right? And and and if you're a business, you're going to think about that too because you have to say, for example, for us, when we do our main site uplink backups,

uh at our data center, um which also services most of White Horse, uh the capital where most of the employees are and most of the citizens are, that's like um I think it's threequarters of the population lives in in the capital city there. Uh when we think about that, we we dep prioritize almost all traffic, right? When when there's an outage. So we we we basically say if there's an outage, you get Teams and Outlook. Uh because everything else is prioritized for phones, basically, right? Like we've only got a couple hundred megs. We're not going to give you Netflix. We're not going to give you Spotify. We're not going to give you all this other stuff

because the internal services are going to be up. You're going to be using our internal links, our dark fiber, our uh man circuits, whatever. Those are all going to continue working. But the things that we care about are going to be our VOIPE lines are going to be our um teams lines and and our ways to communicate with each other which is Outlook, right? So uh those are the types of things that we we have to make business decisions on what are we going to prioritize in the heat of that moment, right? And and that's a decision that takes a lot of DRP time, takes a lot of uh business continuity planning time, right? Um and you're going to have

every single stakeholder arguing that their thing needs to be prioritized over your thing. Um, and when the traffic is or the bandwidth is so limited, the reality is you're going to be able to meet three of those requests out of the dozens you're going to get, right? Um, and that's something that it's never fun to be the person saying no to people. Uh, but you're going to have to, right, if you if you're trying to implement this type of solution, right? So, so I mean, I think for us, bringing in an SD1 solution uh was mostly successful. We we got all the kinks worked out after a few months. Um, we we did have kinks

obviously. We had outages. We had issues where things weren't working as expected. We had um individual services that went down because they were using network routes we weren't expecting, things like that. Um, but overall, we now have a significantly increased resiliency from what we did a year ago, right? And so, I mean, is it worth the investment in this type of product? I mean I think if you're a company or a business or a government or something who has these types of critical services um for example I don't think I mentioned it earlier but like water treatment plants for example we did the same thing right we deployed uh dual WAN in every single water treatment plant that's

those that's where we use uh LTE plus satellite um and the LT is not that great in a lot of those sites but uh when you're when you're thinking about these types of facilities you got to think um is it worth it to spend the time and effort to learn how to do all this to run it to pay for two uplinks, all those things. Um, and if you're in a city like Edmonton, if your data center is probably never going to see an outage anyways, right? If you're if you're probably never going to see an outage um at your main business site, right? Um, maybe it is for you, maybe it's not, right? So, I mean, like what's the cost

of an hour of outage? What's the cost of 10 hours of outage, right? Those are the those are some of the questions um that we had to answer. And and and for us, the answer was, well, 10 hours of outage uh for the hospital could mean like someone dies, right? Like, so that was why we had to invest in some of these things, right? But if your 10 hours of outage is, well, if it happens at 9:00 p.m., then literally nothing happens to us, then that's a different conversation, right? Um, but that's uh that's the that's the the long and the short of it, I guess. Um, we're continuing to roll out more of these

deployments. I've got about 20 of them deployed right now. Uh, we're probably going to deploy another like, I don't know, 12 in the next two or three months and then we're going to have uh probably upwards of like 40 to 50 of these things by the end of the year, right? So it's it's it's a large deployment we're doing because of the way our connectivity is set up, right? Because of the uh immense cost to run rural connectivity and and um and critical services in rural areas. Um but it's something that I think if you don't do then you have to answer to somebody and tell them why you chose not to, right? And and having that prepared

is just as important as doing any of this technical roll out, hiring somebody to the pro services, whatever. It's just as important to tell your boss, I chose not to do this resiliency project because it would have cost us, you know, three full-time employees and and six months of their work, right? Or or whatever it is, right? So, so it's just as important if you when you when you make these types of decisions to to have the full picture and be able to justify the decision in both directions, right? And uh and we like to say at least in the government, we we always give the you know the uh three options, right? You give the option A, B, and C. You say

A, we're going to do this. is going to cost a ton of money and it's going to do all the all the things you want. B, you can do this which is going to cost some money and do most of the things you want. And C, we do nothing. Uh, and this is the risk if we do nothing, right? So, we we always try to present present options like that and and and I think for for business continuity reasons, if if you're going to have to answer your boss anyways, it's good to have something at least thought about like that. Uh, I think we got like few minutes left here, so maybe take some questions before my everyone goes to

sleep. Anyways, yeah, sorry, I talk really fast so my apologies. Um you had the list there of satellite providers uh one web starlink uh telesat I think a lot of us are putting uh critical infrastructure now on starlink um from a government side of it um is there any discussions about that as a national security challenge that we have to do something about it because even telesat it's all going to be launched by Elon Musk So it's not really all Canadian. >> Yeah. So, so what I will say is that we we do have a direction that we we don't deploy Starlinks anywhere non-essential, right? Um we we only we only purchase essential uh deployments. Um that's our

current policy uh direction from the government. Um I I will say yeah, I mean there's obviously always going to be some amount of risk you have to take on, right? Um every organization that you're going to do, every third party you deal with, whether that's Teles, Starlink, One Webb, every third party provider you deal with is going to have some third-party risk, right? Um for us it's about choosing an appropriate amount of third party risk uh with our cost right obviously the Amazon Cuper project is also going to come online um next year obviously also American um but the the risks that we have to consider are that at least in some cases let's say if it's

not an American provider directly such as one web or or telesat um then we have some control over data residency right so so those are things that we're considering as we as we select our next round of deployments when they become available Um when when we talk about the Starlink right now, I I I will say the the the challenge that we have as a government um it's at least in GCUN is um that it really is the only viable provider for us in a lot of these sites, right? Um the one web arrays are tens of thousands of dollars just for the hardware cost and then tens of thousands of dollars a month to maintain for us,

right? So so for us it's it's literally an order of magnitude difference in price. Um, and it's the only viable way if we're going to be deploying 50 of these things uh instead of just two or three, right? So that that's the the challenge that we're running into. Um, but that's also why I think I mentioned one of these next ones. Do you trust your ISP? Right? Like um I don't necessarily trust my ISP, whether my terrestrial one or my satellite one, right? That's why I have resiliency against both of them. That's why I have encryption against both of them. Um, and that's why I don't, you know, just rocket all my data into whatever random

cloud on the internet, right? like that's um you you mitigate these risks by taking the controls that you you should be taking anyways I guess right so yeah any more questions for Thomas

>> um yeah like in terms of um 911 automatic location determination um like kind of two two parts to this question like what what did you do to manage that? And also as a general consumer, should I just assume uh when I'm using my phone that 911 is not going to be able to determine my location accurately? >> Uh I I I I can't speak for for the 911 specifically um on that side. I I I didn't do the emergency services deployment. Um it's run by our community services division. Uh we only dealt with the network connectivity for them. Um but what I will say is that for for when we did our VoIP deployment for example

so on on landlines on phones um that are VoIP uh we do make sure that we select you know the appropriate address for all these things right so the operators do receive those on on your terrestrial um phone links um for cell phones I I can't speak too much on that um but when we did our VoIP deployment it was actually a big deal because one of the problems you'll run into when you do a VoIP deployment is depending on your provider um if you do a uh E911 service is called, right? You buy you a 911 service, they may first send you to a call center in Ottawa or or Ontario or whatever before they route

you to local 911, right? And that's actually a big problem because now you're spending 5 minutes trying to get this Ontario call center to transfer you to the White Horse EMS service because of the guy having a heart attack next to you, right? Like that that's a waste of time really. So, so one of the things that we actually uh worked with our ISP or sorry our telco to do which is also our ISP but um our telco to do was we actually um split our E911. So, so where necessary we do route it to the the E91 service the the call center but where we can we actually split directly to our telco. So so if you pick up a phone in

the government in Yukon most sites um it's a copper landline out to the local 911. Right. So, so that's that's a choice we made um because we felt that it was unnecessary risk to have this delay from from uh using centralized call centers. I have a quick question Thomas if I may. Um, was there any study done with regards to other governments in Canada who might be using these kind of services? Like you mentioned, Nunovet has a 10 gig link up symmetrical with Starlink. They're definitely paying a lot of money. Any other provincial governments who might be struggling and going towards Starlink or rural networks? Uh, sorry, Starling um radio networks, I guess. >> Yeah, I mean I I don't think there's a

like a formalized study. I mean we we talk to our partners all the time especially in the north um to see what other people are doing and I know um many jurisdictions uh are are piloting or or already have implemented for example Starlinks on top of ambulances. Um, so in some northern jurisdictions now, uh, they've put a Starlink on top of every ambulance and then that's how they're doing their connectivity to their dispatch, right? Uh, I I I know we're going to start piling that in in the near future here in the Yukon because one of the problems with some of this is uh, like the current modem system they have modem. If you know

anything about radio modems, it's that they're not fast, right? The opposite of real time. Um, but the uh one of the problems is that you're supposed to get when you get computerated dispatch, the dispatch is supposed to see on their screen where the ambulance is and say turn left in 100 meters, right? Uh, doesn't work over an ardium modem. Um, but that's why a lot of these providers are or sorry, these emergency services departments are switching to satellite uh directly on the vehicles combined with LTE. So when they get into town, they use LTE. When they go uh into rural areas, they're using satellite and and that's something we're going to be piloting um soon as well. But but I am

seeing it with my partners in other jurisdictions like um BC and the Northwest Territories I believe are doing it. >> Perfect. Any more questions for Thomas? >> All right. So in the absence of any further questions, let's give a big round of applause for our speaker here.