I know someday I will be punished for some of the evil, uncalled-for cruelty I have inflicted on poor Microsoft Access when it was all I had available to me. In my defense, I was on an informal development team (ahem) in a company with a dysfunctional tech culture, and we made do with what we had while fighting to get our hands on more appropriate tools.
You think you dislike Access? What if I told you that me losing my virginity was delayed solely because of MS Access? Now think of the printer scene from Office Space!
I'll weigh in outside of Twitter. Back when I worked for a large telecom, we decided to move a datacenter from a mid-west, barely staffed site out to one of our larger datacenters.
This was a year-long several-hundred rack problem that culminated in a hellish weekend of running cables and powering things up. It was "all hands" in IT and didn't matter that I worked in a capacity that kept me far away from hardware most of the time -- I had past expertise in that area, I was called upon.
Everything went relatively smoothly until about three months later when I received a phone call from our department VP. Apparently a vendor that we billed for Cost of Access[0] had not paid us since we moved DCs; they hadn't received any bills and were happy not informing us of the problem. A long investigation resulted in the discovery that the bills were submitted electronically, by modem (this was 2008, not 1998). The modem phone line was moved and located at the new datacenter, hooked up to a USRobotics 56K modem, which faithfully dialed a number, connected at 1200 baud (?!) and submitted the bill. And we're talking a bill that's big enough to be noticed on quarterly reports.
The offending computer had been brought to my office building in the suite downstairs, and my VP said "You've got some experience with old hardware, can you just take a stab at it?" It wouldn't boot; the CMOS battery was bad. I noticed three numeric values printed on a label affixed to the case and realized, immediately, that these were IDE (ATA) drive geometry settings. So the CMOS battery failed a long time ago, and the engineer solved it by printing the numbers on the front of the case.
And what a PC it was ... an IBM NetVista circa somewhere near the year 2000. Running PC-DOS. It sat in a rack on a shelf surrounded by servers, completely ignored since it was hooked up to UPS, not accessible to the corporate network, and was never patched or rebooted[1]. It was an accidental somewhere-near-3/4-million-dollar-desktop-server.
[0] Cost of access for the kind of telecom that we were was, I think the largest non-staff expense the company had, but it also worked the other way -- we billed people CoA for our network.
[1] The device could not be accessed on our network except from one other host, via one port; it was as air-gapped as it could be and other than the few minutes spent "sending the bill" each month, it didn't carry any data of value.
I too have a need to keep old computers online and it's frustrating when the CMOS battery dies and all those super important motherboard settings that make it boot from a hard drive in the first place are lost, over and over on every reboot.
Not everyone knows that old computers do not necessarily boot from the hard drive by default.
You can even replace most CMOS batteries on a running machine without causing any issues. The power from the system will keep the BIOS set, and it's a 1 minute job for many servers, slide out the rack, pop the top, thin phillips screwdriver to the CMOS battery to pop the old one out, pop the new one in and button her up.
Heck, you can buy 2 CMOS batteries for a buck at the dollar tree. They'll last a year or more with the light usage that would be.
In machines of this age, I don't think anyone would have considered soldering something in like that. That didn't happen until halfway through the 2000s.
This model (and every other that I've seen, though, admittedly it's been a while since I've looked at a typical rack mount server) had an easily removed battery.
I'm not sure what that engineer's reason specifically was, but I'll chock it up to laziness.
The CMOS batteries, themselves, have a shelf life and for whatever reason, we probably had none of the specific size required on hand. Being that the device was on battery backup in a rack, it probably led to the engineer thinking "it'll never be rebooted, a label will do"
Nice! Somewhat disappointed that the solution wasn't dialing with the correct area code instead of local (or somehow acquiring the local line matching that number and setting up call forwarding), but recognizing drive geometry settings is even more awesome!
Here in the OT world and where systems need to be air gaped [1], modems are absolutely used to provide separated, secured links and non-IP links. They are not going away any time soon.
I should have clarified -- the modem was really obsolete for the purposes it was being used.
The carrier involved was one that everyone has heard of. All of our other carrier billing was handled using a variety of authenticated links/services to the carrier. And this particular carrier had other, better, ways of doing it, too. For whatever reason, we had never altered that process. It was permanently fixed a few months after we got that host working again.
I would have said Excel, but that would be a lie. You can do a lot of stuff with Excel when done right. So I'd say the following:
Convoluted Excel sheets without documentation doing things the ERP system that is in place is supposed to do. Especially ones that are not feeding data back to said ERP.
I'm working on porting a foxpro database from 1988 that's running an active business which I won't talk about much before it's done, and have actually encountered in Costa Rica an auto parts shop in one of the sketchiest parts of the capital city of this country with a green on black phosphor screen running what looked like dBase III for what they were doing on old IBM PCs.
It's pretty crazy what's out there still. I think the one everyone here is familiar with but might not know is really ancient is the travel booking systems for your plane tickets and accommodations, dating back to the 1960s:
Karsten Nohl - Where in the World Is Carmen Sandiego? (33c3) [1]
>I'm working on porting a foxpro database from 1988 that's running an active business which I won't talk about much before it's done, and have actually encountered in Costa Rica an auto parts shop in one of the sketchiest parts of the capital city of this country with a green on black phosphor screen running what looked like dBase III for what they were doing on old IBM PCs.
This is amazing value for money. A simple system for a small business working reliably for 30-40 years. In our world of tool churn (often seemingly for its own sake), it's unfathomable. But this is often what the customer really wants from us, they're just afraid to ask out loud for fear of being laughed at.
I work in software for manufacturing. We were doing a customer panel and someone remarked they weren’t in the market because they “just upgraded their software”. Follow up question revealed that happened 8 or 9 years ago, so you’re pretty spot on.
This basically only runs on A: old hardware + operating system (no go, need to do other workstation tasks at that company), or B: a windows 7 virtual machine with DOS compat set up, anything else isn't working, windows 7 validation servers were recently taken down leaving things in a precarious state of support. That's basically what I would call held together by duct tape at this point
Oh yeah, man, FoxPro... for a year or so in 2008 I was responsible for a FoxPro database of music venues and artists for an agency. Felt really interesing to maintain a system that's older than me.
When we modernized all desktops in the office, I set up Win2k in Qemu on each of them, which loaded the FoxPro thing from a shared network mount. I'm just realizing 19-year-old me never checked if FoxPro supported simultaneous access. It surely did, at least I hope so :-)
edit: However, 19-year-old me was smart enough NOT to touch that Solaris or whatever-it-was server hosting the FoxPro thing.
Speaking of booking systems, here's an interesting video about a campground in Germany which has been using an Atari ST with custom booking software every day since 1985:
A buddy of mine works for a company in a similar situation that rebuilds transmissions for industrial equipment. They were not allowed to upgrade their computers because the one piece of software that makes the whole company "work" is some ancient dBase system that is accessed through DOS. The bids they received to "modernize" it were astronomical mainly to nobody being around that actuall understood the bizLogic involved in a way to rebuild it.
I asked what happens if the PSU/HDD/etc of the machine running it dies? I was told that everyone retires. It was an actual conversation that had been given serious thought. At that time, the company was 30+ years old, privately owned and no employee had been there less than 15 years.
Sounds like something that could be exported to a virtual machine image and hosted with remote access, unless it runs some proprietary hardware with a PCI interface or something.
No need to hang your whole company on a single box's longevity.
re: FoxPro - It's Visual FoxPro for Windows, which makes it a lot less ancient than 1988, but a significant number of voter registrations in Ohio are handled by a piece of software written in Visual FoxPro.
Amusingly the vendor had a mandate to add "2FA" to the application last year. They added TOTP to the "login". All the data is stored in DBF files that all users have read/write access to anyway (because the database engine runs in the user's context on the client PC-- the server is just a dumb disk drive).
I'm working right this second on an application that originated in the mid-late 1980s in dBase II, then moved to FoxBase, then FoxPro and is currently Visual Foxpro. (and I've been developing/supporting this all along, since 1988) Might finally get phased out in 2022... maybe. It does what it does very, very well.
When I was at Siemens, who had an astounding multi-national PKI on smart cards that integrated with everything, we had to use some software from Hitachi to change our passwords. It rarely worked IME.
Hitachi is a name that I think of for power tools and heavy equipment, but not for software.
I've been on a kick of reading books about computer history of the 80s/90s, and one of the weird things that I didn't know before I started was the fact that Hitachi got pretty deep into mainframes and they were a major IBM competitor for a time.
IIRC, they have reorganized a lot but are still around and have followed IBM's path into consulting and selling other people's hardware.
SEGA built a lot of hardware around Hitachi's SH CPUs. First time I visited SEGA I was surprised to see that they also had...Hitachi elevators in the building. I noted that I was giving my talk using a Hitachi OHP and Hitachi whiteboards and furniture. I looked up to see how much time I had and observed I was using Hitachi time.
As it happens SEGA was never our customer though we did so much work for them; Hitachi paid our bills.
Web sites running on Mainframe, OS390. If any line of code/HTML is longer than 80 chars, it only reads the first line then stops. Because punch cards only took 80 chars.
I love quirky old tech-debt that people still work around. It means there's either a sysadmin that is super into it and knows everything from inside-out, or they're a sysadmin that's pulled all their hair out but damned if they don't keep it running!
Fortran had a 72-column limit on punch cards (the last 8 columns were used to number the cards in the deck). Even when we got VT100's there was some caution about going past column 72 (ISTR some of the VDU's would have a marker at column 72, possibly the compiler gave a warning too).
Oof. As someone who has never participated, every time a post like this shows up I'm reminded of how weird the twitterverse is. Such a microcosm of weirdness, the handles, the academic references (I don't care if you went to MIT for undergrad), the ego jockeying, the shared language of "almost-memes" like 'not all heroes wear capes', 'this is why we can't have nice things', and 'this is what peak performance is' etc et al. And tech bros navigate it as a game of sorts trying to embody some online peak performance version of themselves to signal "yes I'm part of the tech master race".
But! External cultural tear downs aside, I'd like to show some internal cultural self awareness:
Being mostly an academic but having done stints 'in industry' or collaborating with 'real' coders I can say without a doubt that all the jankiest tech lives, and thrives, in academia. Scripts that were written by some professor with no coding background in the early 90s continuing to output data for analysis today. Entire mega projects written by single-minded individuals trying to be the academic version of a 10x developer but they're creating 100x spaghetti code (I'm looking at you RHIC/CERN). Single matlab scripts that are 10s of thousands of lines long that take voltages and currents and output a number with an "uncertainty" that nobody has really ever bothered to externally validate or quality control besides the grad student who never finished his thesis 10 years ago (from an optics lab I worked in).
The list goes on and on. Hilariously, the most beautiful pieces of tech/code I've ever seen have also been in academia that will never see the light of day. There was this ~1000 line piece of OO code that ran a Kalman filter in near-real time on the all of the particles produced in Gold-Gold atom collisions (and simulations!) at RHIC that was probably the most elegant piece of work I've ever seen, written by a coffee chugging burnt out grad student that sat alone in the corner day in and day out.
Speaking of BNL/CERN, there is the code SixTrack which is/was used for both RHIC and LHC for collimation studies and long-term tracking. That's the worst I've ever seen. There were a few files of "normal" length but the main sixtrack source file was fifty-five thousand lines long and featured a custom-written preprocessor's directives throughout (since Fortran has no preprocessor, or for some other erason) that allowed different features to be enabled at compile time. And this was used to design the LHC collimation system and to check the machine's dynamic aperture. I should also add that this is recent/still the case.
P.S. agreed about Twitter, I always wonder what these people are like in real life.
I imagined some people would crop up with experience at RHIC/CERN. I never touched the SixTrack code directly but I've heard legend. The compile time features was a constant on all the projects I touched there and was such a hilarious relic. "The code is flexible! Just know these completely undocumented flags or spend months grokking the dense/obtuse/scattered source files and you can make the code do anything you want, all these hard coded constants will change".
As ugly as things like SixTrack were, I vaguely trust them just because there were usually a lot of eyes on them... even if they were hideous. The more obscure chunks of code that were somewhere in the analysis chain were the ones that really terrified me. I worked on a lot of the code for simulating detector upgrades (I won't get more specific than that) and that was a terrifying place. Designing entire multi-billion dollar detectors systems based on hundreds of different pieces of code and output data that were glued together with no over-arching design or integration. The code 'review' usually came down to "that output looks good"... or not...
Related to this, I now work in the atmospheric sciences, and a completely-unrelated-but-oddly-similar problem exists in the various modeling communities. So many fragmented pieces of code glued together and untouched or touched with no real computing oversight. So many wasted computing cycles. So many failure points.
RHIC/CERN seems like the perfect storm of incredibly intelligent and motivated people, coupled with time pressures and a lack of software development knowledge. Add in the fact that their "business" logic requires a PhD, and you've got an extremely limited subset of software developers who even could work on the code.
Most places, you'd figure people just wouldn't be smart enough to Rube Goldberg their way into a functional program. But someone who does high energy physics as their primary job is assuredly capable of torture a compiler as thoroughly as they want.
> remove gamma_e from calculation of kick by elens. the gamma_e factor comes from moving from the electron frame to the lab frame. It is actually compensated by the lorentz transformation applied to the electron current density
Our department's sysadmin was the real hero. He kept people's software running for years after they left the department. We were involved in some global-level public health data, so this wasn't trivial use-cases.
Ooooooooooooooooooo. The health research sphere is something I've always wondered about. I imagine in some ways it could be the absolute worst, being a terrifying nexus of privacy, academic, and health concerns.
Ha! I know a current coffee jugging grad student who is in a similar position.
They have some survey software in a game engine which was written by a previous grad student. They re implemented textboxes using a bunch of key capture commands and if statements, in one giant file with almost no functions.
Another program he showed me cleans survey data from excel and is one giant script.
Coming from the other side, as a professional SWE that helps my friends in academia with reviewing/debugging their code sometimes... You're absolutely right, some of the most mindfucking code I've seen was from academia. Some are pretty decent and elegant but mostly it's a janky mess of FORTRAN libraries being called in weird ways, code that isn't written at all to be read later on, etc.
The original original digitalocean was a mechanical turk of Perl and bash scrips with cronjobs and a giant mysql db. It would randomly shout things like "RUN THE ARIN REPORTS" (Not a slight btw, it was janky sure, but it worked..) :)
I know what the phrase means and chose to use it. :) His name is Jeff Carr, I very much doubt he nor anyone else from then would object too much to me calling him the mechanical turk of digitalocean, guy almost literally didn't sleep for 2/3 years, probably the most unique, loving and amazing human I've ever met, ever.
It used to be simpler. Subjectively, I think it used to be less janky. Or, at least fewer things used to run on cloud, so the jankiness was a smaller problem.
Not sure, maybe my comment is a case of "grass used to be greener" ;-)
A pentium 4 laptop hooked-up to a makita battery and a couple of PCMCIA serial ports of which a few were connected to 'thin client' serial terminals (axel branded?) and a variety of chemical sampling/testing machines connected to the rest, including the parallel port for a results printer (with tractor paper).
Fortune 1000 company, our mail transfer agent for sending out certain mass mailings were four Macintoshes. Worse, they were never upgraded to MacOS X, because they were so legacy they couldn't be. They were also not officially supported by either desktop support or server support. Although server support had to get involved when our IP address space was reassigned (they got their IP addresses pre-NAT).
At a Fortune 100 company I had to spend a day debugging a print job sent from an IBM mainframe in New York City to a Unix print server, out to a printer in Mumbai.
At one VC funded startup, the first server the company ever bought had no backup system and could not be turned off. We had to rearrange the server room once and someone had to hold the server in the air due to the way it was plugged in as the room was rearranged. It took years to get the budget for the server to upgrade it to, the day the new server came in it was hijacked by the analytics group.
A business critical application that was continuously pulling data from Novell file shares, DCOM remote controlling Excel and really long .BAT files to manipulate it, and FTPing it to other places with some abandonware FTP app. All on 3 shitty old PCs with "don't touch me" post-it notes on the front and hard monitor burn-in. In a break room next to a copier.
I remember the first time I saw a BAT file that had a conditional in it. Mother of god. I needed eye bleach.
If your BAT file has a conditional, you are using the wrong tool. BAT files should be completely top to bottom with no branching. Anything else is madness.
I do have to give MS credit for PowerShell. There are things I don't like, but it's clearly well thought out and purpose made to fill a gap that existed for a long time.
Microsoft Equation Editor 3.0. The piece of add-in for MS Word that has been there since 1995, and finally removed in 2018 due to a security problem. Yet, they are still everywhere for work related to high school maths.
And there aren’t better replacements. I could type in the old equation editor blindfolded. Every keystroke has predictable behaviour. Ctrl-F for fractions, Ctrl-R for square roots. No other equation editors offered that. The new equation editor is not the same, I need to press so many unnecessary arrow keys just to get what I could have done before in half of the time.
Once upon a time, I worked on a small project that involved updating a chunk of Fortran code that did some sort of engineering calculation. The company hired some crusty old engineering professor somewhere who had no clue how to use any kind of source control to do the actual code update. My part of the project was to fix/update the part of this tool where the DLL built from this Fortran code was used by an Excel VBA script - you entered the input data in Excel, then some VBA fed it into the DLL, and stuck the output in some more cells. Getting all that to work right was an adventure.
Around 1999 I did a bit of work for a small shop with DOS 3.3 machines sending data to CNC machines over SCSI cables. They'd bought this stuff when it was end of life upgraded by the previous owners, and gotten the technician that knew the system along with it... a decade before that.
The tech had died several years ago and they were fully aware that when they had a problem with thsese machines that they couldn't fix, the business was done. They made parts for the fuses of artillery shells; so the actual business of their business was trivial compared to the connections and compliance paperwork, anyway.
"that company don't exist anymore" kind of software stack and also they were running everything to death explicitly. If they could have upgraded the electronics to modern, they wouldn't have bothered.
Would be fun gear to get surplus now. I'm pretty sure there's easy cheap hardware to make your Pi either side of a SCSI bus now and then one could poke at the hardware capabilities.
In the server room of a state government agency, sitting on top of a server rack was an old, yellowing AST 128 desktop PC (pentium 128). It sat there with it's little green light glowing and no one paid much attention to it. One day, a newer employee unplugged it and put in the excess equipment pile. Later that day, people were trying to track down a state-wide outage of the business license issuance process. They traced it back to the shared IBM mainframe, and then to an RPC service, and finally back to the AST PC that had been generating the unique license numbers on the HP Non-Stop and sending them to the IBM.
To prevent such an outage from occurring again, a yellow sticky saying "don't turn off" was attached to the AST.
Hmm. Makes a good story, but it sounds a bit apocryphal. What kind of new employee unplugs anything in a server room without significant vetting? Perhaps they hired an idiot, but most new employees wouldn't be that aggressive.
The new employee working under the direction of the slightly less new manager working to meet the demands of the recently appointed director to "clean out that server room, it looks unprofessional!"
To be fair, if the entire World Wide Web had gone offline then, it would have probably affected Tim Berners-Lee, Robert Cailliau, and maybe one person who was reading http://info.cern.ch/hypertext/WWW/TheProject.html
Have you ever tried to connect an IBM mainframe to an HP Non-Stop (Tandem) mainframe? Some developer figured out a way to do it without spending millions on new mainframe software. The key piece of the kludge was a windows interface to the IBM RPC service that both had a USB dongle and wouldn't run on anything newer than Windows 2000.
This was something she had done as a maintenance request and no one else really knew anything about it, until it failed 10 years later. The server people did ask everyone who should know about the PC, so it wasn't just a plug pull.
I remember it was nearly impossible to download the free version of RealPlayer back in the late 90s. They'd do everything they could to hide the link or obscure its location in an attempt to upsell you. Funny that the beeb would rely on the free one, too!
I worked for a company a long time ago that traded with companies all around the world. A lot of their customers were small businesses in rural Africa, South America, etc. and they in turn had local banks they used to facilitate payments to and from.
A large number of these banks did not have a reliable internet connection, or if they did, they most assuredly didn't use it for payment instructions. Instead, I had to jerry rig a system that used TELEX -- yes, TELEX -- connected to IBM's Lotus Domino/Notes suite of mail transport software. That way we could "email" domino and it'd connect to some fantastically expensive TELEX hardware which then went over the wire on whatever the hell network they'd use for that in the 2010s.
We had carefully crafted email addresses with the TELEX numbers embedded (ever wonder why email address specs are so complicated? So people like me can do dumb shit like this!)
We'd then send them very carefully formatted instructions in uppercase ascii, to conform with (I forget which, SWIFT perhaps?) payment instructions that were clearly invented in the typewriter era and then hoisted, kicking and screaming, into the digital age with nary a change.
Fun times. Worked really well and tied in with all the other stuff they'd hang off of Lotus Notes. If you ever wonder why a lot of businesses struggled to move away from it? There's your answer.
> instructions that were clearly invented in the typewriter era and then hoisted, kicking and screaming, into the digital age with nary a change
Based on experience, any time I see a static number of fields of a specific type on a digital record, I assume it's because that's how many blanks there were on the preceding paper form.
It's also because COBOL and languages of that era had fixed widths for most of the "records" they had, for good reasons that many modern programmers are rediscovering every day...
A long time ago I worked as the infrastructure & systems manager of an biostatistical institution at a medical university. The institution had a large staff of highly qualified professors and researchers and others, doing cutting edge work on huge datasets, even by today's standards. It was a really well funded research operation.
One day the phone starting ringing off the hook, the mail blew up and I had a large group of people outside my office (no open work space there!) being very upset all of the sudden. It seemed that almost all of them were relying on this quirky internal service for some input of sorts to what they were coding on/with, and all the computations and calculations and modelling broke without it. And apparently, it had broken. So ofc I start looking into it.
All I got was an IP address, and my system lists and IP registers showed nothing. I went on to look in an old patch panel registry, and found a reference that might be something to look into regarding where it physically might be located. But ofc the patch panel wasn't in use anymore, but I knew they sort of moved it 1:1 to a new panel in the far corner of a basement. Got a new lead on where the patch terminated, and went there only to find an empty room. A lone network cable ran from the connection on the wall, through a hole in the back wall to the next room. The other room had no marking, and my key pass didn't work. I called the maintenance guy who came running, and his keys didn't work either. So we took the decision to simply remove the lock by force.
Once inside, I come upon a very strange sight. Again an empty room, with only a very, VERY dusty desk and chair, with an ancient Unix machine and monitor. No one had touched that thing in many, MANY moons. It was disgusting. Someone had set it up to do its work, and then left the building, without notice or documentation, and it had been forgotten. It was a very Tron: Legacy kind of moment. I had a look at it and it said that the raid was downgraded, but still working but somehow halted the machine. I took a chance and rebooted it and after a while it came back online. All the researchers were happy again! I eventually took the liberty to move all the source code off the machine and got help from a co-worker to set it up in a new Linux environment. It worked almost out of the box, my co-worker made some minor fixes to make it compile. For all that I know, it's still running.
This is a old war story that I never will forget, very fun to talk about :)
174 comments
[ 2.8 ms ] story [ 229 ms ] threadThis was a year-long several-hundred rack problem that culminated in a hellish weekend of running cables and powering things up. It was "all hands" in IT and didn't matter that I worked in a capacity that kept me far away from hardware most of the time -- I had past expertise in that area, I was called upon.
Everything went relatively smoothly until about three months later when I received a phone call from our department VP. Apparently a vendor that we billed for Cost of Access[0] had not paid us since we moved DCs; they hadn't received any bills and were happy not informing us of the problem. A long investigation resulted in the discovery that the bills were submitted electronically, by modem (this was 2008, not 1998). The modem phone line was moved and located at the new datacenter, hooked up to a USRobotics 56K modem, which faithfully dialed a number, connected at 1200 baud (?!) and submitted the bill. And we're talking a bill that's big enough to be noticed on quarterly reports.
The offending computer had been brought to my office building in the suite downstairs, and my VP said "You've got some experience with old hardware, can you just take a stab at it?" It wouldn't boot; the CMOS battery was bad. I noticed three numeric values printed on a label affixed to the case and realized, immediately, that these were IDE (ATA) drive geometry settings. So the CMOS battery failed a long time ago, and the engineer solved it by printing the numbers on the front of the case.
And what a PC it was ... an IBM NetVista circa somewhere near the year 2000. Running PC-DOS. It sat in a rack on a shelf surrounded by servers, completely ignored since it was hooked up to UPS, not accessible to the corporate network, and was never patched or rebooted[1]. It was an accidental somewhere-near-3/4-million-dollar-desktop-server.
[0] Cost of access for the kind of telecom that we were was, I think the largest non-staff expense the company had, but it also worked the other way -- we billed people CoA for our network.
[1] The device could not be accessed on our network except from one other host, via one port; it was as air-gapped as it could be and other than the few minutes spent "sending the bill" each month, it didn't carry any data of value.
Not everyone knows that old computers do not necessarily boot from the hard drive by default.
Heck, you can buy 2 CMOS batteries for a buck at the dollar tree. They'll last a year or more with the light usage that would be.
[1] https://www.classic-computers.org.nz/blog/2009-10-10-renovat...
The CMOS batteries, themselves, have a shelf life and for whatever reason, we probably had none of the specific size required on hand. Being that the device was on battery backup in a rack, it probably led to the engineer thinking "it'll never be rebooted, a label will do"
Yes, we've soldered airplane batteries to 486's to keep them running longer
I'm mildly amused you consider modems obsolete.
Here in the OT world and where systems need to be air gaped [1], modems are absolutely used to provide separated, secured links and non-IP links. They are not going away any time soon.
[1] Example, the dial-up time servers provided by the NPL for offline systems https://www.npl.co.uk/getattachment/products-and-services/Ti...
The carrier involved was one that everyone has heard of. All of our other carrier billing was handled using a variety of authenticated links/services to the carrier. And this particular carrier had other, better, ways of doing it, too. For whatever reason, we had never altered that process. It was permanently fixed a few months after we got that host working again.
Convoluted Excel sheets without documentation doing things the ERP system that is in place is supposed to do. Especially ones that are not feeding data back to said ERP.
I'm working on porting a foxpro database from 1988 that's running an active business which I won't talk about much before it's done, and have actually encountered in Costa Rica an auto parts shop in one of the sketchiest parts of the capital city of this country with a green on black phosphor screen running what looked like dBase III for what they were doing on old IBM PCs.
It's pretty crazy what's out there still. I think the one everyone here is familiar with but might not know is really ancient is the travel booking systems for your plane tickets and accommodations, dating back to the 1960s:
Karsten Nohl - Where in the World Is Carmen Sandiego? (33c3) [1]
[1] https://www.youtube.com/watch?v=vjRkpQever4
This is amazing value for money. A simple system for a small business working reliably for 30-40 years. In our world of tool churn (often seemingly for its own sake), it's unfathomable. But this is often what the customer really wants from us, they're just afraid to ask out loud for fear of being laughed at.
Marketing: “With this app, you can now buy pizza on your phone!”
Snark: *slaps a yellow pages on the desk along with a rotary dial landline*
I'm not saying their aren't horror-shows out there, but technologists drink far too much of their own kool aid, as class.
Docker: rebuilds virtualized operating system
When we modernized all desktops in the office, I set up Win2k in Qemu on each of them, which loaded the FoxPro thing from a shared network mount. I'm just realizing 19-year-old me never checked if FoxPro supported simultaneous access. It surely did, at least I hope so :-)
edit: However, 19-year-old me was smart enough NOT to touch that Solaris or whatever-it-was server hosting the FoxPro thing.
https://www.youtube.com/watch?v=6LxPEz9x2fs
I asked what happens if the PSU/HDD/etc of the machine running it dies? I was told that everyone retires. It was an actual conversation that had been given serious thought. At that time, the company was 30+ years old, privately owned and no employee had been there less than 15 years.
No need to hang your whole company on a single box's longevity.
Amusingly the vendor had a mandate to add "2FA" to the application last year. They added TOTP to the "login". All the data is stored in DBF files that all users have read/write access to anyway (because the database engine runs in the user's context on the client PC-- the server is just a dumb disk drive).
Hitachi is a name that I think of for power tools and heavy equipment, but not for software.
IIRC, they have reorganized a lot but are still around and have followed IBM's path into consulting and selling other people's hardware.
https://www.renesas.com/us/en
SEGA built a lot of hardware around Hitachi's SH CPUs. First time I visited SEGA I was surprised to see that they also had...Hitachi elevators in the building. I noted that I was giving my talk using a Hitachi OHP and Hitachi whiteboards and furniture. I looked up to see how much time I had and observed I was using Hitachi time.
As it happens SEGA was never our customer though we did so much work for them; Hitachi paid our bills.
I love quirky old tech-debt that people still work around. It means there's either a sysadmin that is super into it and knows everything from inside-out, or they're a sysadmin that's pulled all their hair out but damned if they don't keep it running!
But! External cultural tear downs aside, I'd like to show some internal cultural self awareness:
Being mostly an academic but having done stints 'in industry' or collaborating with 'real' coders I can say without a doubt that all the jankiest tech lives, and thrives, in academia. Scripts that were written by some professor with no coding background in the early 90s continuing to output data for analysis today. Entire mega projects written by single-minded individuals trying to be the academic version of a 10x developer but they're creating 100x spaghetti code (I'm looking at you RHIC/CERN). Single matlab scripts that are 10s of thousands of lines long that take voltages and currents and output a number with an "uncertainty" that nobody has really ever bothered to externally validate or quality control besides the grad student who never finished his thesis 10 years ago (from an optics lab I worked in).
The list goes on and on. Hilariously, the most beautiful pieces of tech/code I've ever seen have also been in academia that will never see the light of day. There was this ~1000 line piece of OO code that ran a Kalman filter in near-real time on the all of the particles produced in Gold-Gold atom collisions (and simulations!) at RHIC that was probably the most elegant piece of work I've ever seen, written by a coffee chugging burnt out grad student that sat alone in the corner day in and day out.
P.S. agreed about Twitter, I always wonder what these people are like in real life.
As ugly as things like SixTrack were, I vaguely trust them just because there were usually a lot of eyes on them... even if they were hideous. The more obscure chunks of code that were somewhere in the analysis chain were the ones that really terrified me. I worked on a lot of the code for simulating detector upgrades (I won't get more specific than that) and that was a terrifying place. Designing entire multi-billion dollar detectors systems based on hundreds of different pieces of code and output data that were glued together with no over-arching design or integration. The code 'review' usually came down to "that output looks good"... or not...
Related to this, I now work in the atmospheric sciences, and a completely-unrelated-but-oddly-similar problem exists in the various modeling communities. So many fragmented pieces of code glued together and untouched or touched with no real computing oversight. So many wasted computing cycles. So many failure points.
Most places, you'd figure people just wouldn't be smart enough to Rube Goldberg their way into a functional program. But someone who does high energy physics as their primary job is assuredly capable of torture a compiler as thoroughly as they want.
F.ex. look at the issues: https://github.com/SixTrack/SixTrack/issues
> remove gamma_e from calculation of kick by elens. the gamma_e factor comes from moving from the electron frame to the lab frame. It is actually compensated by the lorentz transformation applied to the electron current density
You know you're in for fun when the docs start with "SixTrack is wonderful, but it is bloody complicated." https://twiki.cern.ch/twiki/bin/view/LHCAtHome/SixTrackDoc
Our department's sysadmin was the real hero. He kept people's software running for years after they left the department. We were involved in some global-level public health data, so this wasn't trivial use-cases.
They have some survey software in a game engine which was written by a previous grad student. They re implemented textboxes using a bunch of key capture commands and if statements, in one giant file with almost no functions.
Another program he showed me cleans survey data from excel and is one giant script.
I pray for him every day.
I don't think that means what you think it means.
(a fake machine, operated by human)
Not sure, maybe my comment is a case of "grass used to be greener" ;-)
At a Fortune 100 company I had to spend a day debugging a print job sent from an IBM mainframe in New York City to a Unix print server, out to a printer in Mumbai.
At one VC funded startup, the first server the company ever bought had no backup system and could not be turned off. We had to rearrange the server room once and someone had to hold the server in the air due to the way it was plugged in as the room was rearranged. It took years to get the budget for the server to upgrade it to, the day the new server came in it was hijacked by the analytics group.
And for better or for worse this is why teams love using the cloud. It goes a long way toward bypassing “old school” IT departments :-)
If your BAT file has a conditional, you are using the wrong tool. BAT files should be completely top to bottom with no branching. Anything else is madness.
And there aren’t better replacements. I could type in the old equation editor blindfolded. Every keystroke has predictable behaviour. Ctrl-F for fractions, Ctrl-R for square roots. No other equation editors offered that. The new equation editor is not the same, I need to press so many unnecessary arrow keys just to get what I could have done before in half of the time.
[1] https://blog.0patch.com/2017/11/did-microsoft-just-manually-...
The tech had died several years ago and they were fully aware that when they had a problem with thsese machines that they couldn't fix, the business was done. They made parts for the fuses of artillery shells; so the actual business of their business was trivial compared to the connections and compliance paperwork, anyway.
Would be fun gear to get surplus now. I'm pretty sure there's easy cheap hardware to make your Pi either side of a SCSI bus now and then one could poke at the hardware capabilities.
To prevent such an outage from occurring again, a yellow sticky saying "don't turn off" was attached to the AST.
Nobody knows anything, and documentation is rare.
That’s a way to build knowledge I guess :)
https://commons.wikimedia.org/wiki/File:First_Web_Server.jpg
This was something she had done as a maintenance request and no one else really knew anything about it, until it failed 10 years later. The server people did ask everyone who should know about the PC, so it wasn't just a plug pull.
I worked for a company a long time ago that traded with companies all around the world. A lot of their customers were small businesses in rural Africa, South America, etc. and they in turn had local banks they used to facilitate payments to and from.
A large number of these banks did not have a reliable internet connection, or if they did, they most assuredly didn't use it for payment instructions. Instead, I had to jerry rig a system that used TELEX -- yes, TELEX -- connected to IBM's Lotus Domino/Notes suite of mail transport software. That way we could "email" domino and it'd connect to some fantastically expensive TELEX hardware which then went over the wire on whatever the hell network they'd use for that in the 2010s.
We had carefully crafted email addresses with the TELEX numbers embedded (ever wonder why email address specs are so complicated? So people like me can do dumb shit like this!)
We'd then send them very carefully formatted instructions in uppercase ascii, to conform with (I forget which, SWIFT perhaps?) payment instructions that were clearly invented in the typewriter era and then hoisted, kicking and screaming, into the digital age with nary a change.
Fun times. Worked really well and tied in with all the other stuff they'd hang off of Lotus Notes. If you ever wonder why a lot of businesses struggled to move away from it? There's your answer.
Based on experience, any time I see a static number of fields of a specific type on a digital record, I assume it's because that's how many blanks there were on the preceding paper form.
One day the phone starting ringing off the hook, the mail blew up and I had a large group of people outside my office (no open work space there!) being very upset all of the sudden. It seemed that almost all of them were relying on this quirky internal service for some input of sorts to what they were coding on/with, and all the computations and calculations and modelling broke without it. And apparently, it had broken. So ofc I start looking into it.
All I got was an IP address, and my system lists and IP registers showed nothing. I went on to look in an old patch panel registry, and found a reference that might be something to look into regarding where it physically might be located. But ofc the patch panel wasn't in use anymore, but I knew they sort of moved it 1:1 to a new panel in the far corner of a basement. Got a new lead on where the patch terminated, and went there only to find an empty room. A lone network cable ran from the connection on the wall, through a hole in the back wall to the next room. The other room had no marking, and my key pass didn't work. I called the maintenance guy who came running, and his keys didn't work either. So we took the decision to simply remove the lock by force.
Once inside, I come upon a very strange sight. Again an empty room, with only a very, VERY dusty desk and chair, with an ancient Unix machine and monitor. No one had touched that thing in many, MANY moons. It was disgusting. Someone had set it up to do its work, and then left the building, without notice or documentation, and it had been forgotten. It was a very Tron: Legacy kind of moment. I had a look at it and it said that the raid was downgraded, but still working but somehow halted the machine. I took a chance and rebooted it and after a while it came back online. All the researchers were happy again! I eventually took the liberty to move all the source code off the machine and got help from a co-worker to set it up in a new Linux environment. It worked almost out of the box, my co-worker made some minor fixes to make it compile. For all that I know, it's still running.
This is a old war story that I never will forget, very fun to talk about :)
https://www.imdb.com/title/tt0609852/