Reverse Engineering for Vintage Systems

besides DC would like to thank all of our sponsors and a special thank you to all of our speakers volunteers and organizers for making 2018 a success dobri autre good morning how are you my name is Dave Riley and I showed up early so I could set up in case there were any mishaps and there were so here we are like I said my name is Dave Riley I'm presenting on reverse engineering vintage systems although I only cover a sliver of everything that there is to be covered so there's certainly further reading to be had and lots to talk about you're welcome to contact me at any of those two locations if you want to talk

more so about me I'm a former ish embedded engineer in in past lives I've done a hardware software and FPGA work I still do occasionally but right now I'm an API programmer I'm also a computer necromancer I work with a lot of formerly dead computers and ones that really just want to take a very long nap and my preferred architectures generally to work with I like vintage Mac's a lot I grew up with them so I'm very fond of them and they also have a very interesting architecture I also work with a lot of pdp-11 s and other Dex stuff and 8-bit micros 6502 and 68 oh and I are my favorites so this presentation is going to be

biased a little bit toward those categories but there's many others when it comes to vintage machines definitely not qualified to instruct on this stuff who really is it's it's half archaeology half embedded systems and then the third half is magic I'm currently employed at fireEye but I submitted this talk before I was hired there and I'm an API developer there so if you don't like this presentation do not blame my employer that's all on me but I hope you enjoy it anyway so what kind of vintage computer are we talking about how vintage specifically there's religious wars on this subject what qualifies as vintage when I really got into the Hobby about ten years ago

the general gist was anything about 15 years old or sell and these days it's a little bit harder to say there's you know ten years later there's more computers that you can consider vintage and what really qualifies is again up in the air most people don't really think old pcs when they think vintage although they can qualify and more and more that's that's becoming a thing people reviving das games and that sort of thing certainly relevant for this topic for security but I'll mostly be talking about systems from the late 70s up through the early to mid 90s and so that covers things like late PDP sevens because the earlier ones usually aren't in in operation anymore

early 8-bit pcs Apple two Commodore 64 trash eighty all the way up through sort of the peak zenith of the VAX alpha spark mips NetWare time period also pre imac max power pcs before the new world that sort of thing one easy metric for whether you could consider a computer vintages is it old enough to vote that probably fits unfortunately now that includes computers made in 2000 and I don't really want to accept that but here we are and yes these machines are absolutely still around they're even around in places where you would really care about them failing and if any of you are ICS people probably about 80% of this presentation is going to be pretty

familiar to you because almost all ICS stuff involves a lot of vintage machinery so agenda for this talk why vintage systems going to talk about why we care about them why we should care about them even if we don't and why you might want to where to start in terms of finding out about them getting into them that sort of thing how do we break them so for the red team how do you get into them how do you penetrate them how do you convince people that these things need to be fixed or replaced how do I protect them so for the blue team what do I do for a machine that hasn't had an OS patch since 1998

is still controlling some important giant piece of machinery and then for the yellow team for lack of a better term how when do I replace them that that can be a deep topic it's not just hey this thing is old it needs to be replaced because oftentimes if it were that easy it would already be done so why do we care computers are usually still in service after 20 to 40 years for three main reasons number one they're very important maybe they're controlling something big and dangerous like trains or nuclear power plants number two often they're very forgotten someone has put a server in a closet somewhere and it's lived there since 1984 and it's still you know delivering

the mail or news groups or whatever and so as long as it's still running no one no one bothers with it number three somewhat important is lazy or cheap or both because it's easier to replace to not replace a computer than to replace it until it breaks down and these really are the ultimate unpatched systems and in a lot of cases unpatch Abell unless you're going to do the patch yourself which sometimes you have to do in something that's critical so just how important are we talking critical process control is a pretty typical area you'll see older things again like I said ICS big area for for vintage machinery there's a lot of multi bus

systems and vme systems still running machinery and trains and everything else out there in fact I think the London Underground still runs on multi bus or at least a big part of it does and there's lots more in industrial like steel mills our nuclear launch system still uses a old IBM system with eight inch floppies which you know is sort of good news because eight inch floppy drives are easy to repair but the media is not so you know not great the benefits of the replacement may not be worth the cost and the risk sometimes just in the someone's opinion sometimes it doesn't wash out so you know if you're talking about something like a

giant steel mill if you replace it it's going to cost a lot to do it right it's going to cost you know man years it's going to be mountains of documentation but you know you also want to weigh that against what happens if the old machinery breaks down and we can't replace it because these things haven't been made since the 80s and you know don't generally underestimate the value to a business of something that's known working there's the old if it ain't broke don't fix it you know again with critical things that's important because the cost of failure is extremely high also the original developers and maintain errs are almost certainly long gone on a lot of these things so the

ramifications of the replacement or upgrade may not be particularly unclear or it may not be particularly clear because the people who wrote it and maintained it before aren't around to give advice so it's a good reason why those are often not replaced as far as being forgotten about everyone either has or has seen server rooms that look like this this looks actually a lot like my basement even though it's not my basement things get forgotten and buried especially in let's say small to medium sized businesses that have survived long enough and these servers get set up they get put up you know under someone's desk that person leaves it becomes so the de facto server corner and you know there's

credible stories of NetWare and VMs servers being found behind drywall because someone put a server into a closet later the workers just patched up over that closet the machine was still running and nobody knew about it until the mail stopped coming one day and I'm like whoa what server handles that and I had to trace cables back through drop ceilings and eventually they found it behind a wall these things happen and of course those operating systems have fantastically long up times so mmm and of course if no one but facilities management goes into the wiring closet at your work you know how would you you know the third generation employee at this office necessarily know about the

old backs that's been running about running the accounting system that everyone has pretty much forgotten about so finding those can be a bit of sleuthing work and of course the third reason laziness or incompetence sometimes things don't get replaced for really dumb reasons especially in government because once you add bureaucracy to things it just takes a lot of time in effort to to replace things because everyone's afraid of getting fired no one wants to lose a government job because they well they'll always pay well but they're they're at least pretty static you know for example the Department of Defense will still pay big dollars for a particular micro VAX model or vac station because it's on the

checklist for a mission-critical app and so on eBay you know all the other members of that family are going for a hundred bucks and this one goes for three thousand because the DoD will still pay that for it and no one wants to get fired for purchasing the wrong thing or moving it to some emulation solution which will probably work but might not and then of course sometimes management just won't replace a system that's still working usually because technical debt is really poorly understood effect in a lot of circles you know the the understanding that it takes twice as much effort to replace something farther down the line is a little bit alien to folks and it's

difficult to explain that's why these things stick around and the places you'll find them so where do we start well if you want to start finding them again safety critical systems power plants heavy industry lots of ICS stuff trains anything nuclear basically anywhere you see computers controlling really big expensive machinery it's likely also long-lived inexpensive systems that have custom interfaces so a lot of hospitals you know are famously still running like Windows 3.1 for running their MRI machines a lot of electron microscopes have interfaces that were only available for old Max or pdp-11 or whatever and they still work and unless you're going to build your own interface you're not going to be able to use your electron microscope so

unless you're in the mood to buy a new one you keep things running lots of other big lab systems lab equipment is notorious for just kind of having hacked together interfaces to computers that may not be replaced very often less often you'll see them in everyday use so old servers that never migrated like you know your old mail or news group server that's still been chugging along until one day it doesn't everyone's gone to an autoparts store and you see them still clunking around on a VT 222 check out your brake pads or whatever they're still running an ancient POS system that works absolutely well enough for them so there's not much reason for them to change at least as

far as management's concerned old computer labs also you know universities frequently just don't clear out the room until they actually need the rooms so my my alma mater actually advertised that it was still running onix's SGI onyx is up until about two or three years ago when they revised the course catalog although I'm not sure they still were so often these things are physically obscured so like an old computer lab no one uses or a wiring closet or something that gets patched over they get forgotten because the people who maintain them leave and no one's cataloging them anymore so it's chalk another one up for asset inventories very important before you get into things always be cognizant of legal

things so there's the standard potential legal disclaimers even vintage software mace may still have licenses that are still in effect for bidding reverse engineering that may or may not matter depending on what you're trying to do with it it has the same legal hazards as reverse engineering modern software and hardware because people do get sued for that but it may be hard to find out who actually owns the IP now some of these companies have been bought and sold three times and you'll find that the company that that actually owns it now says well we don't have any record of that but we will still sue you if you try to take it apart and you know so

some companies especially defunct ones because they can't won't sue you if if you're not harming their commercial interest but of course some do anyway because they're jerks and some also have a funny idea of what might harm their commercial interests sometimes they're also right but a lot of times it's the lawyers told us to do it there's also physical risks of course and again we're talking about things controlling large industrial systems be careful when poking at things that you might break because you know you don't want to be destroying a ten million dollar piece of machinery because you brought something down to the ports can older HOF Hardware often didn't have as many safeguards on

it so you could configure things in ways that could actually damage the hardware that's true of new hardware - but it's less common and of course it's just old I mean you know old hard drive platters are just spinning rust they've been spinning for forty years maybe sometimes they just stop working and if you're really exercising them they may stop working quicker so with that said how do we break them how do we get into them this is this is actually the most reverse engineering part of this talk but we'll we'll get to it so what are we trying to do with these old systems if why would you need to know this as a red team member let's say

standard red team's stuff you want to get into a system you want to gain access you want to dump critical data you want to see if you could potentially break it and cause a lot of financial damage and so you'll want to try networks a lot of these things are networked some of them art many of them aren't some of them were pretty standard networking so there's lots of different interesting Network protocols that don't really exist anymore they may be physically isolated and that can still be important you know again if we're talking about things like ICS having physical accesses sometimes just something you have to assume an attacker has had and you want to be able to make

sure they're at least not easily able to get in and break things you might want to manipulate data if it's an accounting system you might want to be able to pay someone extra something like that or do s important things anything that you would want to do with the modern system it's pretty likely that you would want to do with a vintage system except maybe distribute malware over the web that's probably not going to happen short of that you know nothing changes under the Sun how do we get in so we talked about networks physical access RC ease abound there's lots of them because you know while a lot of these things went under code review a lot of them

didn't and the you know hacking community was not as distributed back in the day there are a lot of things that were never found privilege escalation on multi-user systems can be a big one so for example in VMs now called open VMs just in 2017 there was a privileged escalation the system found that went all the way back to VAX VMS 4 which was released in 1984 so that's a 33 year latent bug in a safety and security critical system that the Department of Defense and Guinness and other people use still so there's there are still lots of things to be found and lots of things that were never found that you know you can drop a zero-day on

a 33 year old system if you want there are Hardware firmware bugs so we're we're coming back again full circle to this now where we're finding out things like our network cards are actually just tiny computers that can also be hijacked well this was also true in the 80s for a different reason well the same and different reason the the host computers weren't as powerful the buses weren't very fast so for example a pdp-11 Ethernet card is a 68,000 computer that does the job of sending and receiving the data over Ethernet and then one that's got a bunch of stuff it dnase it into the hosts memory you can exploit that if you find a bug in that

68,000 computer's program and overwrite it so and nothing nothing really changes but these these things are out there and the nice thing is those are a little bit easier to disassemble than some custom weird microcontroller from Intel also lack of good access control and certainly especially if we go back to the 80s especially if we go back to early micros there may not have been any access control so if you have physical access game's over you can pretty much control it other ones especially early ones were very limited like very limited password length RS X 11 M on the PDP 11 I think it's an eight character password max a hint I'm not even sure the

case-sensitive because the entire OS is in case sensitive lots of old backdoors left around lots of default configs certainly a lot of earlier versions of VMs had default passwords for field service which was basically a system config' so and there's hardware debug options so a lot of them made it really easy to get any debug the hardware because if you were assembling a computer like this you probably needed to do some of that so those things are still left open it's just like leaving an open and unprotected jtag port on a modern router something like that a lot of ways to get into the older systems also there's not much encryption the older Hardware just didn't really

have the horsepower to encrypt much and do anything else at the same time you could you can do a YES on an Apple 2 if you want you just need to come back to it next Tuesday most networking also assumed fairly benign host networks obviously not all of it but if it was built at a time you know let's say it was in a university AI lab and it was only ever gonna be AI students accessing it they may not have really ever expected much to happen to it but the things change the again ICS has the same problem you know that's why they keep those networks separate at TLS also was not a standard till the 90s

when it was invented by Netscape so everything earlier is custom and a lot of places just either didn't do it or if they did do their own custom crypto it had the usual effects you would expect from custom hand-rolled crypto from people who didn't know what they were doing so encryption is usually not something you have to worry about overly much there's also of course weird networking protocols and hardware that you may not have heard of before that you may have hard times getting equipment for unusual hardware like arc net token ring isn't that unusual because I think IBM still uses it but it can be harder to get on a newer machine local talk for for older

Mac's Dec DD CMP which is just serial SDLC x.25 that's just a tiny sliver of the things that might be out there you may find yourself needing to cobble things together fortunately that's actually easier now than it used to be because it's easy to get things like FPGA evaluation boards that you can do with that or beaglebones that sort of thing there's also lots of protocols and Linux actually still supports a lot of the older ones which is nice and TCP dump understands even more of them so appletalk decnet and related px' those are still supported by Linux they're still supported by some of the BSD s and you can still listen to them with

TCP dump or Wireshark that sort of thing there are quite a lot especially if you get into the IBM world that are a little unusual but IBM actually keeps them maintained so you will if you're trying to go into a vintage network you may have to do some deep digging on a lot of these protocols to figure out how to actually talk them so without all all said what's already available for vintage research and reversing so the nice thing is the vintage computers and systems are often really well-documented compared to the modern systems so if you go and look up manuals on PDP 11 processors they will tell you exactly how every block of it works and part of

that was because every block of it was on a separate you know card about this big that you put into a rack for the early ones so if you had a broken one you needed to know how to fix it but 'men sever ething down to a bit level and there are a number of archiving organizations that are dedicated to preserving all this history so bit savers is a big one archive.org also serves up bit savers information but does its own archiving as well so a lot of service and operation manuals for these older machines are still around and available if you look for them unless they were incredibly proprietary and even then sometimes they are because

people just had them lying around in a binder that they took home from work one day sometimes even schematics and diagnostic guides especially if you get into service manuals and if you're lucky things like theory of operation guides and assembly listings so how this really works with comments which if you're trying to find a hole in something can be a Holy Grail because oh you didn't check the bounds on that great unfortunately of course it was a long time ago and it was a time that not everyone was really concerned about preserving history so a lot of documents were lost to accidents or careless storage HP just lost a huge trove of documents that were waiting to be

scanned because of a warehouse fire I think also lawyers again a lot of lawyers don't want things getting out if they don't have a reason to be out and we'll just forbid it and then the documentation disappears which is a shame but usually there's something available so if you're trying to get in do your homework up front do a thorough search to see if the thing you're trying to find is documented somewhere I talked about archive.org and bit savers a lot of that stuff is ocr'd so if you do searches for things that you're looking for it'll pop up on Google a lot of the time because it's ocr'd actually well enough for Google to understand it which

is good and you can find it there's no need to rediscover the wheel if it's already documented unless you really want to if you really want to find out how these things work the best way is to dive right in if you're just trying to do your job it's not going to be the fastest way and it might not be the most effective way someone else also might have solved your problem so you don't need to write a custom appletalk listener because tcp dump still understands it someone might already have written a program to route decnet over the internet in fact they have it's called heck net and you can use that to your advantage if you need to break into

things remotely there's lots of old tricks still out there Usenet archives are full of them because that was what people communicated on before the mailing lists and that lasts forever there's also now modern vintage computer group forums and mailing lists which if you're interested in this I highly suggest you join because you can ask questions of the people who actually use it at a time or the people who've been researching it in the mean time and have learned all the fun tricks you know worst case buy an old sysadmin who knows a thing or two about what you're looking for a beer and ask about their tricks and they will probably still remember it unless you've had too many beers

so for example talk about do your homework up front part of what you want to do is get familiar with what you're looking at so this is a scan of not actually mine but the same model of my pdp-11 scuzzy card it's a scuzzy card but it's also an 8086 computer that runs a specially formatted dos executable I found all this out because my scuzzy card broke because of a dumb thing I did and I needed to fix it and it turned out to be a bug in the firmware but taking just a look at the card if you're familiar with the older computers you can already see a couple things like you have what's clearly

under these stickers a CPU and a pair of eeproms which means it's a 16-bit computer and if you look at a couple of the surrounding chips which you certainly cannot see from the back you can see things that only ever existed on Intel 8086 unless someone was crazy so that was enough of an you know an intro to say well I can just pull those eeproms and dump them and I can probably disassemble them with any standard dos disassembler and figure out what's going on and I could and it did and it worked but you might not be able to spot it that quick if you hadn't already been looking at a lot of older computers so getting to

know the context in which you're going to be operating can be incredibly valuable if for no other reason then you can just say hey I've seen that so other things that might help lots of old systems were reconfigured by building from source so you know I'll actually still BSD UNIX like net BSD OpenBSD FreeBSD you can still install and uninstall drivers by rebuilding from source Linux is similar the whole system comes with commented C code even commercial systems like VMS and RS x11 M they they reconfigured by rebuilding themselves and they basically came with stripped and obfuscated assembly code or other code for for rebuilding when you changed your configuration that's a little bit easier to reverse engineer

from then trying to take a memory dump and and move outward from there of course anything based on UNIX really set the bar high on that because the whole operating system came with its or code obviously the proprietary unix's were slightly different but still compilers and CPUs also we're comparatively simple back then so there's not a lot of optimization in anything that's compiled so you're not going to have to be you know trying to wrap your head around stupid compiler tricks like you know declaring two variables in a single register that sort of thing because the CPUs didn't have multiple execution units for the most part pipelines were non-existent all and optimizer did was get rid of you know

garbage collected references things like that and so the code is usually very easy to follow unless it's intentionally obfuscated especially handwritten assembly because it was written by a human who needed to read it again and often the OS facilities were written with assembly programming ease of use in mind because they assumed everyone would be writing the programs and assembly so the way you interacted with it was based on that rather than a bunch of C function calls so it can make it a lot easier to read what's actually going on there more things that might help the hardware is generally a lot easier to physically probe if you need to then then modern hardware if you need a

dumper ROM you don't have to desolder a BGA package you can just pop it out of a socket a lot of the time or if it's soldered on it's easier to desolder number one but there's also clips that might work that that could override the signals while it's powered on and read it out dip packages are really easy to clip scope and logic analyzer probes on too much more than than surface mount packages there's also custom clips that exist for popular families like 68 oh 30 in QFP package you can get a clip that just sits right on that chip and you can plug it right into an logic analyzer and see what's going on on the bus and then also

the buses are usually narrow and slow enough to attach logic analyzers to so you get an HP 16700 family logic analyzer which you can get on eBay although the shipping is going to be a lot because they're heavy they're amazing and you can bypass the license manager on it to get the licenses for the inverse assemblers which will basically disassemble what's going across those buses live so if if your of options and you just really need to see what the processors are doing right now it's a great option the code sizes are also often very small so for example the pdp-11 s they only add a 16-bit virtual address space and same with 8-bit micros

UNIX on PDP 11 for the earlier machines which didn't have split instruction and data space those you know UNIX programs could only be 64 K in instruction and data doesn't leave a lot of room but it also means if you're trying to disassemble it you're not going to have to go too far for more advanced operating systems for systems of that area like RS x11 M there were complex overlay mechanisms that can make it harder to read but it's comparatively rare things to watch out for so even ancient code from the 80s still employed anti reverse engineering techniques because reverse engineering was still going on certainly there was a lot of espionage back then I know

digital put some fun things in the in the die for some of their chips so that they could see if they were being copied by the Soviets and they were I think one of the VAX dyes had when you care enough to steal the absolute best and it popped up in the Soviet clones so and sometimes there's clever techniques that that make reverse engineering harder to so unconventional branching methods like branching to the indirect target of some register can make it hard for disassembler to follow the flow especially if it's an older disassembler meant for that particular target branching into the middle of existing instructions sometimes that's for obfuscation sometimes it's just because you needed to cram something into a very

tiny space and that was your only option one of the boot roms for for pdp-11 does that because it only had I think 128 words of space for a very long boot process and it actually branches into the middle of an instruction because it was able to reuse that particular value as another instruction that's very hard to disassemble self-modifying code was actually fairly popular back then because number one it didn't break any kind of emulation or JIT compilers so no one really cared but number two you could do some things to save a few cycles or some space or confuse your competitors when they were trying to disassemble it and so unless you have some really fancy plugin for

Ida or something those are hard to do disassemble automatically packing into data sections same thing malware another pack word packers do now done a lot for compression two virtual machines and bytecode interpretation that's also used for compression the Apple two at one point had a 16-bit virtual machine interpreter in there that was incredibly slow but also was incredibly compact code and so you could use that but you'll see that in older code a lot I think actually all the original Microsoft Office programs for the Macintosh ran on a virtual machine as well and of course you'll see encrypted and encoded in compressed strings just like you do now so as far as how you get

in there are standard system tools so a lot of vintage machines have built in debug mode Zoar monitors hit a key combination or send a serial break condition or something like that and you'll pop into the monitor so on spark machines there's open prom which is the same as open firmware on the Macintosh on newer pdp-11 there's ODT which is basically just a serial version of the front panel console switches you can just go directly into memory and do whatever you want there VAX alpha system console was basically the next iteration on that you can get full access to everything if you have physical access to the machine and in some cases like I said just a serial break sequence will

dump you into those things sometimes especially if that was never turned off by the person who is setting up the system and since lots of vintage systems were written in assembler there are extensive machine code level debugging solutions in a lot of the operating systems meant to run on these things because that was the only way to debug them there weren't a lot of source level disassemblers back then and often they were very expensive and didn't come with the system for offline work of course there are lots of reversing tools some of them you probably know pretty well some of them you probably don't there are a lot of interactive disassemblers there are more these days

than ever because people keep writing new ones which is nice because it means Ida keeps getting cheaper but they all have benefits and drawbacks over each other especially when it comes to weird CPUs and vintage ones so I know of course is the big one it has a very long history in reverse engineering and as a result it also has a huge selection of CPUs and binary formats so it includes things like pdp-11 alpha 6502 especially if you have the pro version lots more doesn't have VAX for some reason but but it chances are 80 90 percent of the time you'll be able to find whatever target you're trying to disassembler a disassemble in Ida the only problem is

of course it's expensive if you're working for a company and you're part of the red team that may not be a big deal but if you're an independent consultant or a hobbyist it's maybe not so great and they have a free tier but it's x86 only so radar - which I never know how to pronounce properly newer open-source disassembler it is free it's not bad it's not spectacular a lot of times especially for non x86 architectures it's not bad for arm I found it pretty much unworkable in its current state for PowerPC but your mileage may vary it doesn't have as many CPUs supported as Ida but part of that's just it's newer and people are writing new ones all the

time you can always write your own just like you can for Ida and everything else so if if you have the time to devote to it you can always write a module and that may make your job easier and of course again it's free and it is improving rapidly it is getting better at a very nice pace if you're looking specifically for classic max stuff this is the definitive solution it runs on the classic Mac Steve jazz expose II which is the disassembler and the debugger which is the system level debugger it it gives you the the capabilities to go a lot deeper than Apple's development tools go and it is an interactive debugger very much in the

vein of IDA it works very well it has a very funky interface because he pretty much just wrote it however he wanted not keeping to the interface guidelines from Apple but it is the way to go if you're trying to get classic Mac stuff or really even any 68k stuff done because it does such a good job with it and it does things with classic Mac idiosyncrasy idiosyncrasies like resource forks that Ida really can't handle and radar and everything else doesn't work under the classic environment in OS 10 the debugger doesn't anyway Mac nosey does but it will work under emulators like sheep shaver other PowerPC emulators so if you find yourself needing to disassemble any

of these for whatever reason they're excellent tools and if you need a real machine find a vintage computer group you will get more than you want binary ninja it's a new kid on the block pretty good actually it's commercial it does a neat thing where it basically lifts things to a higher level intermediate language so you can examine the higher level program flow better which is nice and it's also fairly extensible it's cheaper than Ida Pro and actually it's more expensive than this now I think oh maybe it's not maybe maybe that's actually still correct but that is still cheaper than Ida for either version it's very scriptable it's a lot more friendly to modern you eyes if that's your thing

it doesn't have nearly as many processors and binary formats supported but it does support a lot of things that you'd need to see in a typical I see yeah yeah ICS environment like arm PowerPC that sort of thing and again you can always write more and there are community developed plugins that have more support as well it does support 6502 which is nice hopper is a new one it's Mac and Linux specific I haven't used it but people really seem to like it supports the things that that I would want to see in sort of a basic disassembler especially for for ICS stuff it's not a great candidate for vintage stuff but and again there are

plugins for things like 6502 and TI DSPs and that sort of thing and it is actually a lot cheaper binary em is another one I haven't even looked at yet but it looks like it has a very similar feature set that you might like so that's about it for getting in and I promise the rest of these slides that go quickly there will be a little bit of a question time before we're actually it's time to get out so we're going to talk about how we protect them so now this actually this slide is out of order this was supposed to be in the red teen side so how do I protect or protect against these systems number one of

course replace them or upgrade where it's possible we've already gone over the reasons why that doesn't happen there are lots of good reasons lots of bad reasons for that but when you can and wear it where it's feasible replace them if you can't isolate them as much as possible you know these systems you should treat them as fundamentally unpatch Abul so the trick is to put as many things in the way of them being exploited as you possibly can so sandboxing is an obvious one there is no reason you should be running you know a vintage Mac or something on Windows 3.1 on the same network as your production servers don't do it and you

know you may have to do things to route those protocols over whatever your gateway is but it will be worth it from a safety standpoint you know decnet doesn't route over modern IPs but Cisco routers often have ways still of routing that so place gateways firewalls what have you in the way of these things so that people can't just get full access to them also audit them you need to have a good inventory as with everything in security know what your computers are know where they connect you don't want someone saying oh yeah that Ethernet cable that just threads directly through the drywall there no one really knows where that goes it know where it all

goes and you know listen make sure that you have some sort of monitoring solution that is actually monitoring for non IP networking protocols you should be doing this anyway because in in an IP only network if someone starts doing that it's a sign that they're probably up to no good but definitely if you have any machines you need to be listening for them and logging and know what to look for and yeah what if it just won't die if if it's mission-critical and irreplaceable try to find a way to replace it again you can accomplish a lot with sandboxing you might have to do custom code to route stuff and firewall it and all that you

may even need unusual hardware like cisco routers again can still route some of those old protocols you may need to get an older router and older interface cards for them but it can work you should hire the cranky old people who have big opinions and pay them well enough that they will stay because they will actually know the pitfalls that you should be looking for on these networks better than you know someone just out of college who wasn't born when the computer was built which doesn't mean they can't know it it's a likelihood thing and again make sure you have visibility you may have to write your own logging and monitoring code using you know BPF and and other things to

find custom or unusual protocols and make sure that your seam knows what to look for what to flag and that your your sock also knows what to be looking for coming from the seam you don't want lots of false alarms but you also don't want total ignorance so the last part how and when do I replace them so how to replace these things you need to fully examine the cost and risks of replacement so obvious again for big ones like ICS if you're talking about replacing something that controls a steel mill that could kill people if it goes wrong that's a big cost and you need to absolutely make sure that the thing that you're

replacing works correctly before you do it which means a lot of time a lot of documentation and a lot of research some of the subtle things can be easy to miss you know what if there's this really subtle race condition that causes some terrible cascading chain that ends up blowing something up it happens do you really remember everything that this was connected to is there some modem Bank running somewhere that some system maintenance thing out in Peoria needs to connect to or it's going to you know stop billing your important customers and don't forget OSI layer eight which is politics there's always going to be someone attached to the existing system and there's always gonna be someone who

wants to get rid of it so figure out how to correctly leverage those things and satisfy those players and you'll get a lot farther than just fighting make backup plans for if something doesn't work like you expected these replacements aren't easy things break and consider partial solutions emulation is very popular a lot of nuclear power plants are still running on pdp-11 code but not physical pdp-11 machines because they're not very maintainable emulation works well I think parts of our traffic control system work on emulated HP minis as well and a lot of them can even interface with old hardware be very wary of off-the-shelf silver-bullet something it says oh you got an old backs put this

box in your network and it'll replace it sometimes they work a lot of times they're snake-oil so you have to be super careful about what you trust with those things when do you replace these things well if they're well-maintained they should evolve naturally because you should be replacing things when there are a couple years old and starting to get a o L rather than 20 years old and oh my god we have no idea how to replace it which happens a lot addressing long-held technical debt is always a difficult sell to management because it costs more the longer you wait and they're even less likely to want to do it the if it ain't broke mentality is

pervasive but it doesn't consider whether it's going to be a catastrophe if it does break like you know a bridge the bridge breaks it's bad news and you should have fixed it beforehand you it can help to account for the long term financial benefits of upgrading and replacing compared to the cost so like lower power consumption longer uptime lower breach risk lower risk of catastrophic failure etc if the system is officially EOL you should be trying to find a good replacement you shouldn't be trying to hang on to it unless you have an extremely good reason that concludes my talk we have like three minutes for for questions if anyone has any but if you want to contact me

there's the info if you want to learn more about vintage computers there's some resources to connect to I've seen at least one VCF member already here so questions yes speak loudly I know many III have a long time ago I meant know many people who do you mean the tablet right yeah yeah we have VCF members who actually have whole collections good machines you almost certainly it's not in my domain but I'm sure you could get easy pointers on any of these lists for it though other questions so 3b one yep it never goes away I didn't even touch IBM mainframes which are still running code from the 60s in some cases but iBM has

made a significant investment in making sure that stuff still works right so yes any other questions nope thank you for coming to my talk I appreciate it