107 comments

[ 0.21 ms ] story [ 41.3 ms ] thread
We use to run Stratus VOS on Continuum series hardware at a lottery/poker machine company I worked at in the mid 2000's. It handled the jackpot system which decided which of 13,500 poker machines in our state would jackpot at any given time.

A reliable beast - never failed once in the entire time I worked there. Based on PA-RISC if I recall correctly.

Great story, thanks. I wish it said more about the architecture and also VOS.

Speaking of reliable beasts, I own(ed) a HP PA-RISC workstation that never crashed in six years, and it was even purchased refurbished (second hand, and educational rebate, and still the cost of a car in 1996 - but worth every penny).

I'd add "R.I.P. HP 9000 715/75" but of course she is STILL working now, twenty years later, and she is sitting right next to me (also got a second 715/100XC for spare parts in case on day they are needed).

The old workstations were always pretty damn reliable. I’ve got a Sun Ultra 30 in storage. I may grab it and boot it up and see if it still works. Suspect the NVRAM and power supply caps are duff though.
what are you using it with? hp unix? netbsd?
What is she doing next to you?
IIRC, Stratus was M68K (like 12 per cpu board organized into 3 logical cpu's for fault checking)
The OG Stratus FT was mc68k; what a beast. They moved to Intel i860 in the early 90s (XA series...I had a processor board from one hanging in my conference room back in the day), then HP-PA (Continuum series) in the 2000s and these days it's apparently virtualized on x86. Still very popular in the credit card processing world.
Ohh...I guess I had moved off them before cpu switches. I heard they ported Unix to Stratus as well - Did you ever get a chance to play with it ? was it any good ?

I think VOS being PL/1 based pretty cool - Stratus (and Intel fab PL/M stuff) was the only time I've ever gotten to work with PL/1...

The Stratus shops I was at (one financial, one transportation, one communications) considered the Stratus platform to be the holy of holies and mere mortals not on the Stratus group didn't get stick time on the actual systems. I was doing GRC type work with those guys, so I was definitely not on the Cool Kids Team.

That said, from what I gather from working with those guys, there are a couple of generations of unix-y things on Stratus:

1) The older FTX Unix was apparently a pretty bog standard SVR4 port from the user perspective, with the machdep stuff and scaffolding to deal with the fault tolerance layered on a VOS based abstraction layer in the kernel. I knew a lot about SVR4 at one time, long ago, so they chatted me up quite a bit about that.

2) Later they added a POSIX layer to VOS and ported a bunch of Unix tools (e.g. GCC). I dunno exactly what the business case is here or why people who want Unix wouldn't just live in the Unix environment. Neither did anyone else I talked to.

3) These days, I understand people are running Linux (RHEL) as well as windows as on top of VOS. All the worlds virtualized these days.

I obviously didn't get to code anything on the Stratus, but agree PL/1 is interesting as a systems language (the other big user being IBM of course). I understand 'Stratus PL/1' is a lot like 'Burroughs Large Systems Algol': they started out there and have kept adding vendor specific extensions to the point it's not really the base language any more. At least IBM gives it new names when what the programmers use drifts far enough from the base language definition (e.g. PL.8).

IIRC correctly, the stratus PL/1 thing went back to the main hardware designer being big on PL/1 at IBM (PL/1 compiler group maybe?). Also, fun trivia - the Stratus was apparently the first (only?) mini/mainframe that IBM rebadged (they sold it as the model 88)
We were lucky on our site, our team leader was a Stratus guru, and was a great teacher, so we had plenty of time on there if we wanted to practice. We were lucky enough to do just about everything in midrange - Solaris, OpenVMS, Stratus and a bit of Linux here and there in the latter days.
The poker machines are connected? Why wouldn't they just jackpot locally and indepdendently?
Can't have several machines "jackpotting" at the same time or similar time. The prizing algorithm is very strict.
Is this widely known? Presumably the jackpots are sufficiently geographically distributed so that a player who observes one wouldn't be incentivised to pause playing?
I don't think it is widely known how much casino gaming machines are allowed to control outcome. Indeed, in some cases they are required by law to control outcome.
These are probably the kind of machines you sometimes see in gas stations and bars, there's only a few of them in any location at most so there's never too many players nearby to be dissuaded from playing. Plus, a jackpot isn't the only way to win money.
Networked jackpots are a user visible and aparently user desirable feature.

When the jackpot is fed by all the machine in the state, they can grow much faster.

toast0's answer below is the correct one. Plus in Australia, gaming/gambling is heavily regulated - my state in particular didn't even allow poker machines until 1991.

So we never started with a network of disconnected poker machines, the whole thing was built from the ground up with a centralised jackpot system. I don't know what hardware the original jackpot platform ran on but I know it used hardware RNG units. By the time I started there ~2003 it had been migrated to Stratus.

> "This system runs an older version Stratus proprietary VOS operating system, which Hogan believes hasn’t been updated since the early 2000s. “It’s been extremely stable,’ he said."

That's not particularly surprising to be honest. If it's a server operating system, on hardware that doesn't change, there's not really a reason to update the OS unless there's a security problem found

It's not really a security strategy, but running an obscure proprietary operating system on a system which probably isn't connected to the internet means that you're only susceptible to highly targeted attacks by a dedicated foe.

Apparently the latest VOS release was in 2025, which means that somewhere there's a team of developers maintaining it. That's really the thing I find the most interesting, that there exists teams developing operating systems like VOS, strange DOS clones or UnixWare (perhaps less so these days). Must be a strange anonymous life.

> Must be a strange anonymous life.

This perfectly characterizes most embedded systems work. Competent software development using bespoke or off the shelf components that are not mainstream. Nobody ever realizes they’re using your computer software if you’ve done your job right. One day someone throws your appliance away because it’s obsolete.

Well Great Lakes Works is still operating, and you can still buy Stratus ftServers, how nice is that after all this time?
[flagged]
That's 3.15 seconds of downtime a year, or about 104 seconds since 1993. Quite impressive for a steel products maker.
When your swinging a crucible of 2500F molten steel, you really don't want to hear "hey y'all, one sec...I gotta reboot the server...".
Story time: I got handed a ticket to go change a UPS in a “server room”.

I was in Fairbanks visiting family, I had some downtime, so I decided to do some “slumming” to break up the monotony and relive some of the good old days before I was more boardroom than server room.

Besides, a cool grand for what I figured was 2 hours work appealed to me on some “miss the struggles” level, and it was better than walking around the tourist traps or doing video calls at some depressingly fecund coffee shop.

So, I drive up in my borrowed SUV with peeling paint, grab my trusty tool bag I leave with family for adventures like this, and walk boldly through the doors of this apparently normal retail establishment, clipboard in hand. It was 2019, and the smell of denim and factory pressed shirts wafted through my sensorium as I walked through the awkwardly desynchronized automatic doors.

It was a perfectly normal, somewhat 90s nostalgic retail store.., but walking up the stairs (the elevator was out of service) to the second floor, I could just sense that something was a little off.

The office had a smell. I can’t say what it was, it was businesslike, and not unpleasant, but it spoke to me in an unknown language and I was at that moment curiously heartened that my contract was being paid by a 3rd party. It smelled of stagnant ideas, missed opportunities, and a fossilized board of directors.

I’m not just being colorful here in prose. All of these thoughts were projected into my forebrain by some lizard sense I had developed in 40 years of skulking the halls of enterprises large and small. I was involuntarily imagining a half used tin of bengay on the boardroom table, right by the shiny beige speakerphone with the long stroke buttons and the filthy woven grill.

Curious by now about the flood of intuitions, I treaded the slightly dingy burnt orange carpet tiles up to the POC office, checked in, and was shown the door tot the server room. Through there she said, gesturing at a plain flat golden pine door with the elbow thingy at the top. “Through there, straight through, the door at the back. You can’t miss it. I think it’s unlocked.”

Unconcerned with the casual lack of escort (pretty common really), I opened the door and was immediately transported to surreal alternative world. It was dark, there were tens of flickering screens across to the right and one half hanging, flashing fluorescent fixture dangling precariously from the uni-strut above. The suspended ceiling tiles were about 50 percent in attendance.

Fragments littered the floor, and the door closed behind me with a loud thump. for just a moment I was touched by a singular thought….. (run). I shook it off, by now, far too curious to turn back, come what may.

Briefly thinking of my heavy tool bag as a potential weapon, then realizing the ridiculousness of the notion, I let my eyes adjust as I took in the scene with what I confess could only be described as some kind of unhealthy delight.

This was a large room, easily 800 square feet, and a was clearly dedicated to loss prevention surveillance. There was an epic workstation with 2 giant flat screen plasma monitors front and center, and ten smaller monitors on either side. Most of them were dark, and most of the others were flickering with static.a singular monitor had a poor but recognizable image of a small nondescript room with a small desk and two metal chairs, one on each side of the desk in an overtly adversarial position. The image smelled of stale sweat and musty cardboard.

There was seating for two pilots at the helm of this derelict craft, replete with an impressive array of buttons, strange industrial joysticks, and an oddly out of place black dell workstation keyboard with the cord ripped in half. As I made my way up to admire the helm, I nearly tripped on an extension cord, only catching myself on a water filled trashcan with plastic sheeting aspiring to the heavens up through the dark abyss of the half-intact ceiling, which I only now noticed was comprised entirely (if s...

Great story! Was expecting it to be Bernie Madoff's, not Sears lol
> The office had a smell. I can’t say what it was, it was businesslike, and not unpleasant

Timber that was very expensive when it was fitted, and carpet tiles that have repeatedly got damp and not dried out properly. Very old cigarette smoke from when people were allowed to smoke in the office that is now embedded in the blinds and suspended ceiling tiles.

I was recently in a large suite of government offices, fitted out in the 1970s and largely unchanged since, and that was the exact smell. We're imagining the same smell right now.

Great story! This reminded me of the urban explorer videos taken inside of the last Sears headquarters building. It was (is?) a pretty amazing place and it's somewhat tragic that it was abandoned and left to rot.
The real marvel is that you tortured both the English language and your readers that much just to describe a UPS swap.
No pain, no gain lol
It's not often I feel disappointed that I've reached the end of a comment, wishing I had more of it to read.
I started working on Stratus machines right out of high school... They took me from backwoods New England to Wall St.

The were Awesome machines - don't let those filthy hobbitsis...err..tandem users cloud your judgement. Stratus was pure hardware FT, and tandem... wasn't :-P

Vos was pretty nice too - decent macro language - not all the utils you expect with a unix box, but in terms of the core OS and shell it was pretty good machine .. 40 years later I still love those machines and the career they helped get me into :-D

Oh Tandem, you don’t hear that much anymore. They still have plenty of them still running banks and what not under the name HP non-stop.
yeah - the big difference was Tandem was software (+ hardware) fault tolerant while Stratus was full hardware level fault tolerant. This gave Tandem a real price edge.

Both were really popular with banks and brokerage houses (back when I worked with stratus at brokerages, HFT was just emerging and down time was umm... frowned upon :-D

I spent six months programming on VOS after a gig on a VAX. Yes, I'll be outside telling the kids to get off my lawn as soon as I find my reading glasses and have a nap.

VOS was intuitive and did what I needed. Nothing flashy. Just useful. I miss that sometimes. Give me a tool that works; if it also happens to be great, lovely.

I didn't work much with the Stratus hardware, but reading about it was fascinating. I wish I remembered more of it.

Hey, I worked there in high school as an intern! But just on the intranet, which for its time (‘96?) was really quite good.
When you think about it, 24 years is not such a long time for a machine to keep doing what it's supposed to be doing. It says something about our profession that we find a 24-year service life surprising.

There are airliners that have been in service for over 50 years.

(comment deleted)
I think it just speaks to how quickly the tech has developed. Airliners work more or less the same as they did 50 years ago, computers are vastly different (and vastly more powerful).
Airliners may look the same but they don't work as they did 50 years ago when the was no fly by wire in these planes, no high-bypass engines, etc.
Computers are many thousands of times faster and more efficient than they were 50 years ago. Airplanes still fly at the same speeds, and with only marginal safety improvements. Fuel efficiency has doubled however. Overall significantly less changes.
Faster but for the rest basically the same. More megahertz, more megabytes, more transistors per cm2 of silicon, more processors but that's basically it. Von Neumann architecture, processing separated from memory, block/file-based storage, input and output through keyboards and displays.

Yes, this drastically understates the change in computing but the actual technology hasn't changed that much, it just got denser and faster.

Eh, sorta. Fifty years ago was 1976 (I know, right?!). The 747 debuted in 1970 with high bypass engines. The A320 came out in 1988 with fly-by-wire and all Airbus aircraft since have been the same.

The biggest change from then to now is probably the amount of solid state electronics onboard. That's ridden roughly the same curve as other industrial and commercial applications.

It's not so much surprising that the machine hasn't broken, but that someone apparently considered it worthwhile to keep it around, having about the compute of a throwaway vape.
Airliners are up against physical limitations that don't incentivize upgrading. If there was an airliner that could fit millions of people and cost less than the 1980s version, I don't think you'd see many 80s aircraft around anymore, except in niece applications such as this computer. Perhaps now that we're at the end of Moore's law, you might see a larger amount of long-lived computers. But on the other hand, we'll probably find new ways to innovate in computing power.
Computers are still evolving rapidly. Instead of raw GHz increases, it will be things like tensor cores or other extremely-wide/GPU-type processing units. And of course, more RAM and more caches to feed them.
yabut... airliners require constant maintenance and periodic rebuilds.
Not to mention service disruptions and unplanned downtime.
(comment deleted)
One of the major differences between like, your PC, and a high-availability server is that high-availability servers require those things as well—and get them, piecewise, while still running. It's like doing a complete overhaul while the plane is flying.
I wonder how many datacenters bother with those sorts of things anymore, given how many workloads are no longer tied to individual nodes. Seems like it would just be easier to accept that a certain percentage of nodes will be down at any one time, and go through and repair/replace them as needed.
Cloudslop doesn't provide high availability the way high-availability platforms provide high availability. IBM is still doing brisk mainframe business. You won't find mainframes in many data centers because you need nearly a data center's worth of hardware to provide the volume and availability of one mainframe, and that can fit in one cabinet. (Mainframes also benefited from VLSI; new ones aren't dinosaurs that require three-phase power, their own floor of a building, and their own aircon infra.)
That may well be the case, but such a long period still seems extraordinary
There are industrial machines such as steam hammers that have been in constant use for more than a century.
This is the most dwarven thing I've read today
You raised in my mind that we are looking at this all wrong, the server is not actually one component, it’s really a system of systems, and just like an airliner is a system of systems too, the conflict arises in that the comparison is at the wrong level.

It seems when people think of a server today, they think of a single computer, if not some software server somewhere in the cloud. These subject kinds of servers are far closer to complex systems of separate components, more like a network system, it’s why components of what are really separate networked computers can and need to be replaced.

An airliner is of course also made up of components that are also systems, however at least in my mind, the difference is the actual expected use case. An airliner is not a good comparison because it is never expected to remain in continuous operation, e.g., that at least one engine is always running even when it is being overhauled or repaired, to satisfy a requirement of continuous operation.

FTA, this is a fault-tolerant server where all hardware is redundant and can be hot swapped while running. The servers you’re thinking about are redundant at the software level so rebooting one server won’t cause the service to go down.

What is remarkable is that apparently that VOS thing has never crashed. I wonder if they also do software redundancy under the hood to keep uptime going during reboots. If the CPU is hotswappable, it must have something.

Probably uses formally verified code along with plenty of housekeeping processes to keep any failures from shitting the whole bed. In critical system design, you build in redundancies that work in parallel such that any one failure will not interrupt the system.

The main computer system in the Space Shuttle is an excellent example of this. It had 5 identical IBM System/4 Pi machines. Three of them ran identical code and handled the same work. The fourth ran a completely different codebase to handle the same work, preventing a bug in the main code from killing the whole system. A fifth computer handled other tasks but could be swapped over to the critical role if needed. You could lose 2/5 computers and still have insurance against a cosmic ray flipping a bit.

VOS is a parallel lockstep OS. You drop nodes and replace them to keep the whole operational.
Those aircraft have maintainance windows/refurbishment/refits and aren't flying 24/7 though.
This server also had planned downtime maintenance. The article specifically mentions "no unplanned downtime."
>There are airliners that have been in service for over 50 years.

Are there any airliners that have flown for 24 years non-stop?

this is why I find comparing mostly-solid-state stuff with big mechanical contraptions a frustrating exercise.

it's more impressive to me that the electric fans within the Stratus server still operate than it is that the chips still work, and if we're going with the plane metaphor it's the silicon that puts the thing into the air, not the simple fans.

So really it just boils down to "well, the planes work is harder." as to why it fails, which is of course abstract and unsatisfying; comparing the comparative lifetime workloads of a chip that has shifted trillions of bits versus some jet engines that have moved millions of kilograms around the world.

and then another layer comes and makes it even weirder : the vast majority of hardware around the world that has been EoLd and replaced has been in that situation due to software, not the hardware itself; a paradigm that really doesn't exist in the physical engineering world in the same was as it does CS.

Mechanical integrity in industrial processes is often divided into two categories: static equipment (tanks, pipes, heat exchangers...) and rotating equipment.

These two have very distinct failure modes. The failure mechanisms for solid state electronics are much more similar to those for static equipment (corrosion, thermal fatigue, creep, ...)

The chips themselves aren't surprising, but there are other electronic components that may fail earlier like capacitors
Yeah, it’s not a good comparison. Give me endless parts and a maintenance window of 6 months every so often and I can make any server last a century.
Was going to say, there's servers that last several decades, they receive routine maintenance, most people buy a computer, and rarely replace parts, and if enough years pass, you opt to buy a new one instead.
I'm not sure, but I think there are a few submarines that have remained underwater for that long.
Absolutely not. They come up for shift changes at the very least, and more.
Naval vessels are some of the most maintained machines in the world. They get hauled out for hull repairs routinely, or put into long term maintenance facilities for months while they are repaired. Probably second only to airplanes and space craft.
> There are airliners that have been in service for over 50 years.

...with a very aggressive maintenance cycle (typical part life is 1000-10000 hrs of service). Most airliners of that vintage resemble the "Ship of Theseus" in real life.

Isn't mostly capacitors that limit the lifespan? With a proper maintenance plan (as airliners follow) you could probably keep servers running for a long time.
Those are getting regular maintenance and overhauls while not flying, it's not really comparable.

On other side only moving element would be hard drives and only wear element really would be the electrolytic caps.

It is interesting how long server uptime used to be considered a good thing but now the shorter the average runtime the better as it indicates the system was designed to tolerate failures.
No, a long uptime is evidence of the crime of not sacrificing sufficiently on the altar of security. You must be constantly interrupting service for security patches, firmware updates, release upgrades, etc. so your users are constantly reminded that security is your paramount concern.
Patching vulnerabilities IS important and well designed systems can keep running while one server reboots.
Linux may get a kexec handover protocol some years from now. You can reboot the kernel and your userland daemon seperately :)
this attitude towards just generally keeping shit secure will always be absolutely baffling to me.
I wonder if doing this to hardware would have overall been cheaper than doing the baroque Rube Goldberg shit we do to software to make it somewhat kind of fault tolerant… or at least convince ourselves it is.

(You can write real fault tolerant software but a wad of tangled crap and hidden dependencies crammed into Kubernetes is not it.)

For a compare-and-contrast, Tandem is the Stratus competitor that went more toward software FT (though it does have redundant hardware). You can also look at various n-of-m voter redundancy setup that are very common in aerospace computers. Definitely not rube goldberg, but also not cheap if for no other reason than you need 3+ of everything (e.g. the primary Space Shuttle computer ran 4 synchronized voters).
Fault tolerance is expensive, even when it's a single box. This kind of hardware is the wrong tool for the job because most use cases can tolerate some downtime and it's not worth paying 10x cost to get a few nines you don't need.
Same applies to most IBM and Unisys, mainframe and micros.
This is really calming. Is the music generated fresh for each city, or do cities share a set of loops? I'd love a rainy Seoul alley at 2am as a preset.
Definitely cool about the uptime, but as I have no experience on systems like this, I'm also curious about the economics of this as a business strategy.

How do the maintenance, development, and operations costs of a system like this compare to - for sake of argument - a typical line-of-business Java system running on commodity servers you own in a DC somewhere?

Can anyone with experience in both comment?

Stratus 'servers' were originally 'super minis' - 19" racks full of gear with multi-million dollar price tags - looking at the picture in the article, I was surprised to see what look liked racks of desktops ?
I know a guy who is very proud of the fact that he has a Flask based web app, fully exposed on the public web, running with >99.9% uptime on a cheap hosted VM with zero updates to either the OS or application code since 2016.
You can find random PDP, VAX, even X86 systems that have lifespans 20+ years. As for a single boot probably a much smaller corpus, maybe some forgotten Sun SPARC in a telco on -48V DC power bus.

A VAXCluster or Sysplex(S/390) would I guess be more analogous but those can be a bit Ship of Theseus since you can change the hardware out over time.

For VMSClusters 8 to 10 years seems to be a pretty common uptime for long lived clusters. I worked a gig a few years ago that had a VMSCluster that was up for 19 years at that point. Started on alpha, migrated to itanium, then I helped migrate it to x86. It was the oldest I've personally seen, though I have heard legends of much older.

My current contract has a PDP-11/70 running RSX-11 that's been up since 1992. It's uptime is longer than mine! Poor thing is getting retired next year though.

>Windows NT 3.1 and Jurassic Park were both released

A horror about the unchecked madness of human technological development, and a movie about dinosaurs.