Well, at least planning for the future seems prudent. Otherwise if such a system breaks permanently, then you are scrambling to get something else up while the whole system is down (even ignoring the question of backups and data migration to a new system).
Well the article says "If the computer stopped working tomorrow, a staff person would have to turn each building’s climate control systems on and off by hand" - I know that's not exactly a tiny job across 19 schools but it doesn't seem to be a hugely tough task
There can be dozens of air conditioners on the roof of a school. And they have to be cycled on and off all day long as they cool the building, else it'll turn into winter in there. Definitely a fulltime job for somebody. That's why we have thermostats.
As has been said multiple times before, it's not $2 million to keep this kid's school project running for another 30 years, it's $2 million for a better HVAC automation system. They've probably pissed away multiples of $2 million in energy costs not having a better system in place the past 30 years.
No kidding, but that's not how the article was written. My second paragraph was a comment on that.
Every budget ever written stuffs things that management thinks can be put off until later into funding for things that absolutely need to be done today.
It's not just the software. They're controlling these systems via radios that operate on the same frequencies as other radios in use. It's (at this point) clumsy and error prone. To be fair, they probably saved the tax payers a lot of money by using this system for as long as they have.
> To be fair, they probably saved the tax payers a lot of money by using this system for as long as they have.
Indeed. The major annoyance I see is the radios interfering with each other. The centralized control architecture is also something I'd like to change. Each building does not require anything much smarter than an Arduino to be completely standalone and, if you want to be fancy, an RPi to send collected data to a centralized location.
Another viable approach could be to setup Amiga emulators on old/discarded PCs and repurpose the existing software using wired instead of wireless connections.
The 2M number is probably a major overhaul of every part of the system. Utility bills alone will probably more than cover the difference.
I suggested a US$ 200 PC because many of those have no moving parts and more storage space than any Amiga ever dreamed of. They should be able to easily last 30 years or more.
> The only problem is that the computer operates on the same frequency as some of the walkie-talkies used by the maintenance department.
It's awesome that it's been functional for this long, but given the press attention I hope they don't receive unwanted attention given that this system was implemented in the 1980s, before computer security was as large an issue as it has become.[1]
With the amount of publicity this article has generated, I bet someone with an SDR is going to drive over and analyze the RF signals, and then hijack the HVAC systems for the lulz.
As a Grand Rapids resident who knows quite well how to use a SDR, I'm going to choose a positive interpretation of this comment and assume you're referring to the phenomena of "West Michigan nice", where hacking a public system for lulz would be absolutely unconscionable.
That being said, I'm actually rather tempted to at least figure out what the air interface is. The article mentions a 1200 baud modem - does a standard UHF/VHF voice channel have enough bandwidth to run PSK/FSK V.22 or V.23? That'd be quite a hack.
I meant that its far the heck away from everywhere and the population is low. Thus, a low incidence of griefers of any kind. But your take is good too.
Very likely it's Bell 202 AFSK. This is specifically designed to run at audio frequencies with 1200 Hz and 2200 Hz tones. You'd have to figure out how the higher protocol levels are implemented but it's probably going to fall within the realm of what is done with AX.25.
Often it's dealing with interfaces. A couple years ago I worked at a place with a laser cutter, like an old one. You sent instructions to a program that talked to an interface card, which controlled the cutter. The program ran under DOS. It wasn't possible to replace the OS or machine easily, because it was an ISA card and had custom interface software that relied on having direct access.
$2m might cover design fees for swapping out mechanical systems across 19 schools but probably not considering that this would be on the high end of the typical public project fee curve because of how messy HVAC renovations are for the sort of water distributed systems usually installed in large 40 year public buildings. It's not going to be craning in and out some package units from the rooftop.
Bringing Stocking Elementary out of moth balls, replacing boilers and roofs, and removing asbestos were just some of the projects GRPS put on the Warm, Safe and Dry list before the Commodore computer.
It seems they've already replaced boilers. They could have fixed any leaking radiators while they did that. ISTM they really are just talking about the control system. Perhaps it made sense to build a custom system in 1982, although I really doubt it. Nowadays, Honeywell certainly sells units that can just be dropped in, at each site. Unless they have steam tunnels running between the 19 buildings, centralization of control seems unnecessary. Sure, holidays, snow days, and terms move around slightly, but schools are on the internet now so staff at district HQ can manage each site's schedules when they need to do so. It would make sense to have centralized reporting, but just run that over the internet instead of the walkie-talkie bands.
Current best practice [and perhaps state regulations for public schools] require commissioning the controls. And RFQ's, RFP's, sealed bids, etc. The project requirements are discontinuous with a homeowner thermostat replacement in the same way that database cluster requirements are discontinuous with a Comcast residential gateway.
That seems orthogonal to centralization? Sure, an HVAC tech should regularly inspect each site, and should be on call for any issues. That doesn't require split-second coordination of mechanical operation among separate sites. We're talking about heating some schools, not operating a nuclear enrichment facility.
Neither is there anything about the "commissioning" process that mandates that control systems may only be replaced when mechanicals are also replaced. That would be ridiculous. Please note that I am not quibbling about the $2M cost. I merely observe that TFA discusses the control system only.
Commissioning etc. are orthogonal to the system configuration, and we should just agree to agree that a centralized control point is not necessarily necessary. OTOH IMPO, the control system requirements should be specified by a HVAC PE.
The proposal wouldn't be for replacing a computer and a 1200buad modem. It's for replacing a distributed HVAC control system at 19 locations. If you're doing that, you're going to want to bring the system up to current standards and that includes things like commissioning [1] the system for energy performance and replacing controls and logic at both ends...or maybe going away from a hub and spoke architecture. And it all comes with RFP's for design, public bids, performance bonds, insurance, warranties and all sorts of things that grandma's Wordpress site doesn't typically require.
[edit] To put it in perspective, getting the controls right dwarfs energy use. Let's call the cost $1.9 million and the number of sites 19 giving a cost per site of $100,000. Let's assume that the energy cost per site is $100,000/month and that the controls are capable of +10%. That puts the payback period at 10 months, so let's call it a year. Even if the efficiency improvement is only 1%, the payback period is well before the 30 year life cycle of the building (which not by coincidence matches the 30 year maturity of typical bonds and the bond financed control system built around an Amiga).
$100k per month per site for energy? That number seems high to me, but I can't seem to find a link suggesting for/otherwise. Is that really what schools pay?
Doesn't seem high to me. I recall that my school district replaced the HVAC system while I was in high school and the ROI paid for the new multimillion dollar system in something like 5 years.
That 800,000 sq ft is for an entire district. With a little searching, I found that Saline High School is the single largest school in Michigan at 480,000 sq ft. Most large schools seem to be in the low hundreds of thousands.
So the $100k/mo figure is probably high by a few factors - maybe an order of magnitude at most.
You might have noticed the trend of "de-malling," transforming indoor malls into open air shopping centers. While there are many drivers for this, the ongoing cost HVAC is a big one. So it doesn't surprise me that schools would pay so much.
Ah! PDP-11! I mis-spent my youth on one. I remember seeing one standing out in the rain in Los Alamos at an electronics surplus store. Didn't even rate getting inside the building under cover. Now that's melancholy.
This legacy system issue doesn't just happen to remote/low budget operations either. While they've now moved to emulating the system on what are presumably VMs, Columbia University uses what amounts to a 40 year old system to post final grades to the student administration portal. Official transcript grades are updated once daily, at some time in the wee hours of the morning. From a Columbia CS professor:
"Columbia processes grades by using a system that was first released in 1972. In order to have kept it running for more than 40 years, it has consistently run special-purpose emulators that make its otherwise state-of-the-art systems think that they are stuck in the 70s and using an operating system called “CP/CMS”:
The grading system is written in a programming language called “Focus,” which in 1975 was one of the very first database languages developed and released:
But because of this, grades are processed only once per day, in a batch job that runs at about midnight every night. I am not making this up.
The university, recognizing that it is time for it to upgrade, does have plans for replacing the grading system. The upgrade is scheduled for the year 2020. I am not making this up either."
I imagine these schools will eventually move to something similar: an emulated system with specialized hardware to interface with the mechanical equipment.
Many universities have systems like that. Banks too. Probably true of many early adopters: they built systems that worked and didn't rewrite them simply for the sake of having something new.
When I interned at IBM from 2003-2006, we submitted our timecards using a terminal application accessed over telnet. I don't know exactly what system it was, but the key commands certainly evoked thoughts of early mainframes.
It's not for the sake of having something new. New technologies do often provide obvious benefits. It's the cost of a new version (both in software development and reorganization of processes) that may be too high to justify these benefits.
At my first job, we built I/O cards that did stuff like the Amiga in the article does. The cards used adapters so they could run on anything from TRS80's to PC's because at the time the system was conceived, the IBM PC hadn't emerged as the dominant desktop yet.
The company is long out of business, but every so often someone finds some of their products at a garage sale or industrial auction or it comes in a box of other things on eBay. When they search for manuals on the product they often find me because by coincidence someone asked about those products on a forum I frequent and that conversation tends to come up in the first page on google.
It's amazing how many really old systems are still running after decades without change and no one gives it a thought until something breaks. I've often wished I had a list of their old customers so I could contact them and offer my services to upgrade old controls.
Yes and please do blog about it regularly if you feel like it - this whole thread on legacy systems is the most interesting read on HN for a long time IMHO.
Most of my credit cards will not update the balance immediately after I make a payment. It takes a day (sometimes more, depending on timing). Clearly batch processes are involved.
There's nothing wrong with batch processes. They are often easier to reason about than having a system where everything can change in real time.
Sometimes you want real time. Sometimes (many times) a daily batch is just fine.
You can have delays for that purpose in real-time systems too. And they'd actually be consistent, instead of having variable lengths to cancel the transaction.
Credit card processing does seem to be moving towards real-time. Lately I've found more and more merchants where I get a notification on my phone instantly after my Amex is swiped.
Why should the computer have to deal with what is obviously a poor policy decision on the part of the teachers and administrators?
Enter the grades when you're working - i.e. during your work day.
Oh .. I know, teachers work way more than they are given credit, and don't make nearly enough for the work they do, given that they're up until 4am doing some paperwork or other in order that a few hundred students get their results .. just playing devils advocate here. This is a 40-year old system, still working. If it ain't broke, why fix it?
Seriously though, my personal nerd-ego says that anyone who can keep a system running in emulation over 4 decades deserves a special kind of award for the high status of their accomplishment. This is something few of us are capable of doing these days, alas ..
It doesn't really have anything to do with policy or working odd hours. Unless everyone presses enter at the same time, grades will be input at different times. (I'd love to see administrators trying to corral senior faculty into doing that, though.)
As to whether a system should process grades more than once a day, that's obviously a judgement call that Columbia has made. And as someone who worked in the administration of similar university, a small part of me is cheering on whoever at Columbia has held fast against the demand for up-to-the-minute GPAs.
Actually, so you're saying you think it is all the result of smart policy, and I agree with you now. I see that up-to-the-minute GPA's are a real burden. Funny though, that they all have to hit enter at the same time, didn't realize it was that brilliant. ;)
Grades on the Columbia equivalent of the Moodle/Blackboard student portal can and do update instantly as soon as teachers hit submit. This is a recently updated, fairly modern system. However, these are not necessarily the official grades that go on your transcript. In my experience, teachers often never post the (possibly curved) final letter grade on this system. I assume that transcript grade system is a separate portal that teachers have to input the final letter grades into. This system, from the POV of the student, only updates once per day. Teachers put their grades in at 5pm? You get your "grade has been posted email" at like 2 am, and practically speaking get your grade whenever you wake up.
I do not miss those days of waking up early in the morning after finals to find those emails. I would nervously read them in hopes of not being disappointed by my grade
Is that really that big of a deal, though? There would be effectively zero benefit to giving the students their new grades on demand instead of after a batch process. It's not a reason to upgrade the system, it just shows that the system was made when batch jobs were considered acceptable.
I know what you mean comparing it to Columbia University, but Grand Rapids isn't exactly a remote location. It's the second largest city in Michigan. We're not talking about northern Alaska.
It's not remote, but the school district hasn't exactly been flush with cash over the past few decades. If this occurred in East Grand Rapids, Forest Hills, etc., I would be more surprised.
Even modern software would work like this. Many universities use a product called Appworx that schedules jobs... and your grades are only going to be upgraded once per day.
That Peoplesoft system dominates a lot of the schools in the US. It spread like the plague on promises of being tailored to every school's business logic. Unfortunately it did so via vomit-inducing over-abstraction to the point where interfacing with it just to lookup a student record takes hundreds of lines of XML/SOAP wrangling.
IBM mainframes incorporate VM technology in both senses of the word: the "abstract ISA" sense and the "simulated hardware" sense. They knew they were building systems that may have to be online for decades; and the exact same object code that ran on the original System/360 runs bit-identical on a brand new POWER-based mainframe (though obviously faster).
They aren't replacing a computer. They're replacing a (at this point) hard to maintain system for controlling their HVAC system and other aspects of their HVAC system. For 19 schools, this seems cheap to me. More modern HVAC and HVAC controls will also result in reduced energy costs saving the district money further down the line.
Virii were mostly spread over floppy (boot block or trojan) back then. While I agree that it's very unlikely that this system will get infected, don't think that times were better back then :)
Richard Karsmakers used "virii" to describe a plural of computer virus in the 16-bit era, largely on the basis that it was shorter. It isn't correct Latin, but it is a VX scene thing, so is absolutely correct in this context.
If your first thought is that $2M to replace an Amiga is ridiculous, stop.
They wouldn't be replacing the Amiga for $2M. They'll be replacing a 19 site remote HVAC control system, certainly including new radio systems, local controllers, and the central controller. They will want something with a warranty and a service contract for the next 20-30 years.
Welcome to West Michigan local media (though I'm sure this phenomenon isn't unique to this area). While it can probably be explained by bad writing, they've also learned that anything that provokes the ire of tea-partiers is golden for driving pageviews.
I don't see how anyone could have thought that it cost $2 million to replace this computer.
It became very clean that it was for a revamp of the entire system.
I see no reason for it not to. Keep in mind that this system has also required maintenance over the years (just like any system). The cost savings from improved efficiency (more granular control, reacting to weather & building occupancy, etc.) will also be a huge improvement.
I tend to be very cynical these days. Far too many things are not designed for longevity. But, I can certainly see where a newer HVAC system would be more efficient.
Modern climate control systems are quite nice if setup correctly and do a very good job. Probably save them quite a bit of money over the 30+ year lifespan of the new system.
Yep, you want the whole thing to act like a system not a bunch of parts. Energy savings and a comfortable environment really depends on this. A poorly setup system can be a nightmare for the maintenance staff.
And so this is why, in our office with a thermostat and ducts in every room, it's typically freezing in 1/3rd of the conference rooms, another 1/3 are way too hot, and there is zero meaningful control using the thermostats in the rooms over this?
Not saying it's not a good idea, just saying that in reality I've seen these systems work very poorly together in most office environments I've been in.
Well, if you have thermostats in every room, I would imagine that it would be impossible to get a consistent temperature. All the commands and one system to sort it out and move air cannot be good.
If I ever teach a CompSci again, I would think about making a climate controller a problem set. My first choice is a storage building controller, but this would be interesting.
A big reason is cost-savings. This prevents each building occupant from setting the AC too cold, or forgetting to turn it up/off when they leave at the end of the day and for breaks.
At 20 years, $2m means you can hire 3 people for $33k a year to do the job. At 30 years, you can only hire 2 people for $33k/year. That figure does not include benefits. That may or may not be viable for 19 schools (depending on how far apart they are, etc).
The $2 million doesn't have to sit around for 20 years in a non-interest bearing checking account. Put $1.5million in a 5 year CD and get another ~$50k/year. Give them other maintenance work to do while they are at the school turning on and off the heat, and it becomes more viable.
The radio system made sense back in the 80s before cheap, reliable internet access was widely available, but I feel like IP would be a much more practical communications solution today.. especially considering the fact that the 2-way radios used by the maintenance staff interferes with the existing system.
Though your point is still totally valid, I'm just thinking out loud.
> before cheap, reliable internet access was widely available
Cheap and reliable Internet access is still not widely available. You don't want to have your HVAC stop working because ISP botched something up. Frankly, IMO there should be zero reason for any of the HVAC control loop to extend beyond the building. Sending data from your controller to your device next to it through half the world and back is insane, and yet it is what's happening with IoT right now.
Well as far as the cheap part goes, my thinking was that these buildings definitely already have internet service in the first place, may as well just utilize that.. I supposed you make a point about reliability, but then again they're putting up with the whole 2-way radio interference, seems like the bar is already set low :P
I don't know, again I'm just thinking out loud, I'm no HVAC professional. We do use the internet to manage off site devices in my company and we tend to have more issues with power outages than we do with our ISP uptime, it's been a good solution for us.
You couldn't be more right. To this day, I encourage the use of radio, dial-up, closed-circuits, cheap leased lines... anything but the Internet... for mission-critical transmission. Just way too many issues from reliability to security. Using simple lines and protocols with little to no routing let's one use equally simple hardware. If resources are left over, one can use them for boosting manageability, reliability, or security.
If it is a really critical system, the proper way would be a T1 circuit between the two buildings. The circuit would only go from one building to the local telephone office, back to the other office. They tend to be pretty reliable, but expensive. You could then run a lan over this.
I still don't completely see why it'll be $2M, unless that's the total cost including the service contract for the system's lifetime. I mean, the hardware and installation labor can't be that expensive, right?
The GRPS website says that they operate 67 buildings, which leads me to speculate that the $2 million includes upgrading other parts of the HVAC system at each school.
Not all that much pops out of a few searches, but there are discussions of projects that make me think that $100,000 per building isn't a lot to spend on such a thing.
It would mostly be all the contractor and subcontractor costs. I'd say they need at least a couple of electricians, which are usually unionized and charge plenty (I have no issues with unions, it just costs more). The plumbers, because you can't hack into a wall without hitting a water pipe. Then the HVAC people, who have different sub specialties. Then the people who do the cement/wood/boarding. Not including the inspectors...
"That's probably because he wasn't sick. He was skipping school. Wake up and smell the coffee, Mrs. Bueller. It's a fool's paradise. He is just leading you down the primrose path.... I've got it right here in front of me. He has missed nine days...." eight, seven, six..
My company, an ASX30 corporation, still runs its operations through an emulated IBM 3270 terminal sitting on top of a DB2 v8 mainframe with literally thousands of tables in it. 4 character transaction codes and ctrl as 'enter command' for all!
We're currently developing a replacement SAP solution. I'm yet to decide which is more obtuse.
edit: The SAP solution was spurred not because the old system was breaking but rather that when the COO asked for some daily reports and the IT propellerheads replied that the data crunching to generate said 24 hour reports took more than 24 hours to run.
> A Kentwood High School student programmed it when it was installed in the 1980s. Whenever the district has a problem with it, they go back to the original programmer who still lives in the area.
I thought that. Then living in the UK our first 32-bit ARM personal computers turned up in 1987. They even ran rings around high end Unix kit. And here I am in 2015 typing this on a 32-bit ARM A7...
What's the realistic life expectancy of modern hardware?
I ran a Ras. Pi system for monitoring my sprinklers and such but after about 9 months it has had some funkiness, I suspect power related and then the flash has had a couple little glitches. I couldn't see it lasting 5 years. I have an odroid-c1 that seems to be more robust in terms of power and such (it has a dedicated transformer, not some cheap USB that seems to "work") I do worry about the flash crapping out... Although if the policy is to burn a few flashes and just keep some spares, you should have quick recovery.
Maybe some good server grade hardware, in a data center, you can expect some decent life.
Most embedded system failures are due to power and I/O.
If you have robust power supplies from a reputable manufacturer (I like TDK/Lambda) with good filtering on the input (maybe external powerline filter modules), and you protect your external inputs and outputs, then the next thing to worry about is flash lifetime which should be measured in decades if you're not writing to it.
Problem with a Rasp Pi/Beaglebone is that it's running Linux and there are likely disk writes occurring, so your lifetime drops. If you can both limit disk writes and get long lifetime SSDs, I don't see why it shouldn't last at least 10 years.
"What's the realistic life expectancy of modern hardware?"
Lower that it used to be. The design life for the Ford EEC-IV engine control system from the 1980s was 30 years. The program is mask-programmed onto the chip, and the parameter table is in a fuse-blowing type ROM. Many of those are still on the road.
With newer hardware, lifetimes are shorter. This is a big problem for long-lived military systems.[1] Electromigration becomes more of a problem as IC features get smaller.
There are embedded systems where 30-50 years of operation is needed. Pumping stations, HVAC, railroad signals, etc. have equipment which can run for decades with occasional maintenance. The NYC subway is still using century-old technology for signaling. It's bulky, but works well.
I read somewhere, actually I think it was an interview with Google's Saul Griffith, who said no smart phone was designed to run longer than 6 or 7 years in ideal circumstances. Realistically I imagine 5 years as unusually long regardless.
The flash survival rate heavily depends on how the system is configured. If you move every writable directory (/tmp, /var/log, /var/run, etc) to a tmpfs filesystem and make the main filesystem readonly, it'll live much longer (and better survive hard reboots).
I would say that's an assumption we as developers need to teach society, but we should really work on teaching it to ourselves first. I can't tell you how many times I've heard "We should just start over" rather than try and understand what we have.
The software would only be a small part of this project. The issues they discussed in the clip (risk of sudden failure, radio interference) were hardware issues.
This. Nobody today would even consider trying to design something so simple it could last three decades. And certainly nobody would want to support it. Everyone wants shiny and new...
How right you are. On the other hand I'm continually surprised that my paycheck in 2015 comes (more or less) from a descendent of an operating system I learned in the 80's.
Nah, somebody would say it need json-serialized config files, a flash-file system, a rails-based admin login running on a few amazon instances for fail over....
"Simple" Ha.
There's this whole coprocessor in the ethernet shield with its own proprietary implementation of TCP/IP and who knows when someone will find a serious bug in it. Then in 30 years you have a TCP/IP stack with a serious security issue and no way to patch it!
Agreed. It ran for 30 years and hasn't had problems (other than the radio interference issue). Anything that has lasted for 30 years has had nearly every bug worked out or worked around. The only things that survive for 30 years are systems that do the job.
Exactly! That's the ultimate problem with these kinds of systems. It's why I advocate incrementally re-engineering a compatible system onto modern hardware and standards. Not even anything fancy necessarily: just easy to maintain, portable, and fast enough. Companies like Semantic Designs even make source-to-source compilers that do bug-for-bug translations. I'm sure we could do dynamic translation of binaries for the dependencies, too, although I'm not sure anyone does that.
Its not the code that's concerning, its the hardware. The idea that they never bothered to virtualize or emulate this in 30 years says a lot about how IT is run at that school.
A single developer could emulate whatever IO goes to that radio and have it controlled by a simple computer with redundancy. I wonder what they're paying the original developer to maintain this turkey. I imagine he's charging enough to where the sunk cost fallacy is obvious here.
What I find interesting is all the places where FOSS is lacking. There doesn't seem to be a popular and well managed FOSS climate control system. Is anyone making standardized controllers, thermostats, etc? This stuff should be a commodity by now. I'm guessing its hard to work with all the proprietary HVAC stuff and its also an unsexy problem to solve. I'd love to replace my thermostat with a raspberry pi and have a lot of fun modern features. There's a hackaday article about someone doing this, but its a pretty primitive project.
There are some projects to start bring FOSS ideas/ideals to building system controls and decision making like this, but I think there are significant difficulties in the variety of proprietary systems and reluctance to change due to capital costs and/or maintenance contracts.
I think they got very lucky that the original programmer didn't move away or decide he didn't want to maintain this thing anymore, whenever some dingus from the school district decided to bother him. I'm seeing a lot of people ignore that aspect of the situation. There's a reason we talk about the bus number.
It is true that old code is not necessarily bad code. However, HVAC system controls have had very large improvements in efficiency over the last 30 years. HVAC systems are the single largest use of energy in a commercial building and the improvements that modern controls can offer in efficiency can greatly offset the costs of installing those new systems.
I'm mostly a back-end guy... does it really matter if you you use != vs !== when you know that both operands have the same type in javascript? In this case, you've got string literals, so (flying spaghetti monster, I hope there won't be any type conversions going on)
Pedantic, but if it's an Amiga 2000 (which it appears to be from the video) then this can't have been running since 1985, as the Amiga 2000 was released in 1987. But the guy does say the computer came from eBay, so perhaps this is a replacement for their original Amiga.
I'm not sure how the RF interface works, but it it uses a zorro card, the A2000 would be the first computer that could run it. If it used serial, the even a A1000 would be a possible system.
The GRPS director of communications just gave a presentation on the bond proposal last night at our neighborhood association meeting. Like any urban district, GRPS has faced a very challenging few decades with declining enrollment, competition from charters and suburban districts, and a largely impoverished student population. They've made huge strides over the past few years, since the current superintendent took over in 2012. It's been really exciting to watch.
At the presentation, the spokesman stressed that the previous bond was strictly to shore things up—anything that could be delayed, was. I can see that he wasn't kidding.
The record for obsolete computing probably goes to Sparkler Filters, which still uses an IBM 402 tabulator from 1948 for their accounting. This machine reads information from punch cards, adds it up, and prints reports. That wouldn't be too bad, except this tabulator is before the vacuum tube generation. It is electromechanical, using mechanical counter wheels for addition.
It is programmed by plugging dozens of wires into a control panel; each wire indicates something like this card column goes to this adder and that printer column. To change the program, you pull out the panel and replace it with a different panel. The "software library" consists of shelves of panels wired up for each task.
This system actually surprisingly sophisticated considering the lack of technology. For instance, there's a mechanical mechanism to suppress leading zeros in numbers. It can print text as well as numbers and supports three levels of subtotals as well as conditionals. It It's also fairly fast, processing 150 cards per minute.
There are still several forums full of current Amiga users [1] [2] [3] [4] being the most prominent English language ones. The most amazing thing is that there is still new hardware expansion being designed and manufactured for the original machines...
Those kind of results are why I encourage copying past designs for new hardware projects in safety-critical space. Especially NonStop architecture, which is patent-free now. Take one of older ones, make it 32-64 bit, put modern interfaces on it, put it on an older SOI process node for reliability, optional lockstep, and the result should run very well for a very long time.
Was hoping it would be an Amiga, and wasn't disappointed. Though it seems like this particular system is ripe for replacement because of the issues mentioned, I have a lot of respect for staying with old soft- and hardware instead of chasing something new and shiny when the old system still gets the job done.
A few years back a seller came to my workplace to ask if we wanted to advertise at our local cinema. We didn't, but I had a chat with the guy and was pleasantly surprised to discover that he was still creating and showing the ads using an A1200 with a genlock card running Scala (25 years old this year, real popular in the nineties with cable companies and broadcasters as big as CNN and BBC). Reason for not upgrading? "It still works great." Put a big smile on my face for the rest of the day.
The first version of Scala was released in 1990 if I'm not mistaken, he probably used a later version as he was using a A1200 which had the AGA-chipset that improved graphics considerably. He'd been using it "for about twenty years", and as this was two or three years ago and the A1200 was released in 1992 he probably had a 22 year run on the same hardware and software. He didn't specify the card he used, but I think it's safe to say it was the Video Toaster given its popularity at the time.
Polish Railways(state owned) had an Odra 1305 machine, manufactured in 1973, still operating on one of their stations until 2010. It's fascinating really.
272 comments
[ 3.0 ms ] story [ 208 ms ] threadThat said, I'm pretty sure I could get an Amiga emulator running on a Raspberry Pi for a lot less than $1.5 million.
Every budget ever written stuffs things that management thinks can be put off until later into funding for things that absolutely need to be done today.
And, probably, they could run their software under an emulator, on any US$200 desktop PC.
Indeed. The major annoyance I see is the radios interfering with each other. The centralized control architecture is also something I'd like to change. Each building does not require anything much smarter than an Arduino to be completely standalone and, if you want to be fancy, an RPi to send collected data to a centralized location.
I suggested a US$ 200 PC because many of those have no moving parts and more storage space than any Amiga ever dreamed of. They should be able to easily last 30 years or more.
It's awesome that it's been functional for this long, but given the press attention I hope they don't receive unwanted attention given that this system was implemented in the 1980s, before computer security was as large an issue as it has become.[1]
With the amount of publicity this article has generated, I bet someone with an SDR is going to drive over and analyze the RF signals, and then hijack the HVAC systems for the lulz.
[1] https://en.wikipedia.org/wiki/Notable_computer_viruses_and_w...
That being said, I'm actually rather tempted to at least figure out what the air interface is. The article mentions a 1200 baud modem - does a standard UHF/VHF voice channel have enough bandwidth to run PSK/FSK V.22 or V.23? That'd be quite a hack.
https://en.wikipedia.org/wiki/List_of_U.S._states_and_territ...
How on earth! And where can I apply?
Bringing Stocking Elementary out of moth balls, replacing boilers and roofs, and removing asbestos were just some of the projects GRPS put on the Warm, Safe and Dry list before the Commodore computer.
It seems they've already replaced boilers. They could have fixed any leaking radiators while they did that. ISTM they really are just talking about the control system. Perhaps it made sense to build a custom system in 1982, although I really doubt it. Nowadays, Honeywell certainly sells units that can just be dropped in, at each site. Unless they have steam tunnels running between the 19 buildings, centralization of control seems unnecessary. Sure, holidays, snow days, and terms move around slightly, but schools are on the internet now so staff at district HQ can manage each site's schedules when they need to do so. It would make sense to have centralized reporting, but just run that over the internet instead of the walkie-talkie bands.
IANACivilEngineer!
Neither is there anything about the "commissioning" process that mandates that control systems may only be replaced when mechanicals are also replaced. That would be ridiculous. Please note that I am not quibbling about the $2M cost. I merely observe that TFA discusses the control system only.
[edit] To put it in perspective, getting the controls right dwarfs energy use. Let's call the cost $1.9 million and the number of sites 19 giving a cost per site of $100,000. Let's assume that the energy cost per site is $100,000/month and that the controls are capable of +10%. That puts the payback period at 10 months, so let's call it a year. Even if the efficiency improvement is only 1%, the payback period is well before the 30 year life cycle of the building (which not by coincidence matches the 30 year maturity of typical bonds and the bond financed control system built around an Amiga).
http://www.schoolenergysaving.com/schoolEnergyFacts.php
"A mid-sized school district with 800,000 square feet of space pays more than $1M annually for energy."
"Space heating, cooling, and lighting together account for nearly 70% of school total energy use."
So the $100k/mo figure is probably high by a few factors - maybe an order of magnitude at most.
To me, it looks like a very optimistic guess.
"Columbia processes grades by using a system that was first released in 1972. In order to have kept it running for more than 40 years, it has consistently run special-purpose emulators that make its otherwise state-of-the-art systems think that they are stuck in the 70s and using an operating system called “CP/CMS”:
http://en.wikipedia.org/wiki/VM_(operating_system)
The grading system is written in a programming language called “Focus,” which in 1975 was one of the very first database languages developed and released:
http://en.wikipedia.org/wiki/FOCUS
But because of this, grades are processed only once per day, in a batch job that runs at about midnight every night. I am not making this up.
The university, recognizing that it is time for it to upgrade, does have plans for replacing the grading system. The upgrade is scheduled for the year 2020. I am not making this up either."
http://bwog.com/2014/01/06/why-your-grades-take-so-long-to-s...
I imagine these schools will eventually move to something similar: an emulated system with specialized hardware to interface with the mechanical equipment.
The company is long out of business, but every so often someone finds some of their products at a garage sale or industrial auction or it comes in a box of other things on eBay. When they search for manuals on the product they often find me because by coincidence someone asked about those products on a forum I frequent and that conversation tends to come up in the first page on google.
It's amazing how many really old systems are still running after decades without change and no one gives it a thought until something breaks. I've often wished I had a list of their old customers so I could contact them and offer my services to upgrade old controls.
[sigh] another business opportunity missed.
There's nothing wrong with batch processes. They are often easier to reason about than having a system where everything can change in real time.
Sometimes you want real time. Sometimes (many times) a daily batch is just fine.
Playing devil's advocate here...
Do grades change during the day?
Enter the grades when you're working - i.e. during your work day.
Oh .. I know, teachers work way more than they are given credit, and don't make nearly enough for the work they do, given that they're up until 4am doing some paperwork or other in order that a few hundred students get their results .. just playing devils advocate here. This is a 40-year old system, still working. If it ain't broke, why fix it?
Seriously though, my personal nerd-ego says that anyone who can keep a system running in emulation over 4 decades deserves a special kind of award for the high status of their accomplishment. This is something few of us are capable of doing these days, alas ..
As to whether a system should process grades more than once a day, that's obviously a judgement call that Columbia has made. And as someone who worked in the administration of similar university, a small part of me is cheering on whoever at Columbia has held fast against the demand for up-to-the-minute GPAs.
The vast majority of them wait til the last minute to do it, right at the end of term.
http://www.teyko.com/View.aspx?id=346&name=Saddam+Virus
...which was a rather brilliant little thing, but a personal hatred as it messed up many a .s files (.s stood for "source code" or "Seka", I guess).
For reference, 1.2 million in 2014 dollars is around $418K in 1980 dollars.
They wouldn't be replacing the Amiga for $2M. They'll be replacing a 19 site remote HVAC control system, certainly including new radio systems, local controllers, and the central controller. They will want something with a warranty and a service contract for the next 20-30 years.
Not saying it's not a good idea, just saying that in reality I've seen these systems work very poorly together in most office environments I've been in.
If I ever teach a CompSci again, I would think about making a climate controller a problem set. My first choice is a storage building controller, but this would be interesting.
Even if they do install local controls, a central monitoring system makes some sense for buildings that go unoccupied for days at a time.
Though your point is still totally valid, I'm just thinking out loud.
Cheap and reliable Internet access is still not widely available. You don't want to have your HVAC stop working because ISP botched something up. Frankly, IMO there should be zero reason for any of the HVAC control loop to extend beyond the building. Sending data from your controller to your device next to it through half the world and back is insane, and yet it is what's happening with IoT right now.
1) Reliable internet isn't everywhere, it cuts out a bit.
2) A lot of internet enabled thermostats are just not that great at reconnecting when the eventually loose their connection.
Though to be fair the thermostats generally will just continue on the normal schedule if they loose connectivity.
The company ended up using on a different wireless protocol to talk from the base station to the thermostats. (z-wave or zigbee fwiw).
Ironically we had a 80s error programmable thermostat in the office, which worked pretty well.
I don't know, again I'm just thinking out loud, I'm no HVAC professional. We do use the internet to manage off site devices in my company and we tend to have more issues with power outages than we do with our ISP uptime, it's been a good solution for us.
Not all that much pops out of a few searches, but there are discussions of projects that make me think that $100,000 per building isn't a lot to spend on such a thing.
The point I was trying to make was that they likely already have other systems in the remaining buildings.
We're currently developing a replacement SAP solution. I'm yet to decide which is more obtuse.
edit: The SAP solution was spurred not because the old system was breaking but rather that when the COO asked for some daily reports and the IT propellerheads replied that the data crunching to generate said 24 hour reports took more than 24 hours to run.
Now that's job security!
I knew when I saw the MC68k the first time that it had the Schwartz to rule them all — and decades later it's still true :)
I ran a Ras. Pi system for monitoring my sprinklers and such but after about 9 months it has had some funkiness, I suspect power related and then the flash has had a couple little glitches. I couldn't see it lasting 5 years. I have an odroid-c1 that seems to be more robust in terms of power and such (it has a dedicated transformer, not some cheap USB that seems to "work") I do worry about the flash crapping out... Although if the policy is to burn a few flashes and just keep some spares, you should have quick recovery.
Maybe some good server grade hardware, in a data center, you can expect some decent life.
If you have robust power supplies from a reputable manufacturer (I like TDK/Lambda) with good filtering on the input (maybe external powerline filter modules), and you protect your external inputs and outputs, then the next thing to worry about is flash lifetime which should be measured in decades if you're not writing to it.
Problem with a Rasp Pi/Beaglebone is that it's running Linux and there are likely disk writes occurring, so your lifetime drops. If you can both limit disk writes and get long lifetime SSDs, I don't see why it shouldn't last at least 10 years.
Lower that it used to be. The design life for the Ford EEC-IV engine control system from the 1980s was 30 years. The program is mask-programmed onto the chip, and the parameter table is in a fuse-blowing type ROM. Many of those are still on the road.
With newer hardware, lifetimes are shorter. This is a big problem for long-lived military systems.[1] Electromigration becomes more of a problem as IC features get smaller.
There are embedded systems where 30-50 years of operation is needed. Pumping stations, HVAC, railroad signals, etc. have equipment which can run for decades with occasional maintenance. The NYC subway is still using century-old technology for signaling. It's bulky, but works well.
[1] http://mil-embedded.com/articles/obsolete-cycle-costing-you/
The Debian Wiki has an overview of what you might need to change to achieve it: https://wiki.debian.org/ReadonlyRoot
I would say that's an assumption we as developers need to teach society, but we should really work on teaching it to ourselves first. I can't tell you how many times I've heard "We should just start over" rather than try and understand what we have.
There will be backwards compatibility options for an awfully long time.
A single developer could emulate whatever IO goes to that radio and have it controlled by a simple computer with redundancy. I wonder what they're paying the original developer to maintain this turkey. I imagine he's charging enough to where the sunk cost fallacy is obvious here.
What I find interesting is all the places where FOSS is lacking. There doesn't seem to be a popular and well managed FOSS climate control system. Is anyone making standardized controllers, thermostats, etc? This stuff should be a commodity by now. I'm guessing its hard to work with all the proprietary HVAC stuff and its also an unsexy problem to solve. I'd love to replace my thermostat with a raspberry pi and have a lot of fun modern features. There's a hackaday article about someone doing this, but its a pretty primitive project.
https://github.com/VOLTTRON/volttron from Pacific Northwest National Lab is one example.
http://www.bacnet.org/
http://www.textfiles.com/bitsavers/
They also have artifacts like MacPaint source code in their collection: http://www.computerhistory.org/collections/catalog/102658076
At the presentation, the spokesman stressed that the previous bond was strictly to shore things up—anything that could be delayed, was. I can see that he wasn't kidding.
It is programmed by plugging dozens of wires into a control panel; each wire indicates something like this card column goes to this adder and that printer column. To change the program, you pull out the panel and replace it with a different panel. The "software library" consists of shelves of panels wired up for each task.
This system actually surprisingly sophisticated considering the lack of technology. For instance, there's a mechanical mechanism to suppress leading zeros in numbers. It can print text as well as numbers and supports three levels of subtotals as well as conditionals. It It's also fairly fast, processing 150 cards per minute.
Links: http://www.pcworld.com/article/249951/if_it_aint_broke_dont_... https://en.wikipedia.org/wiki/IBM_402 http://bitsavers.informatik.uni-stuttgart.de/pdf/ibm/punched...
[1] http://www.amiga.org
[2] http://www.amigaworld.net
[3] http://www.amigans.net
[4] http://eab.abime.net/
A few years back a seller came to my workplace to ask if we wanted to advertise at our local cinema. We didn't, but I had a chat with the guy and was pleasantly surprised to discover that he was still creating and showing the ads using an A1200 with a genlock card running Scala (25 years old this year, real popular in the nineties with cable companies and broadcasters as big as CNN and BBC). Reason for not upgrading? "It still works great." Put a big smile on my face for the rest of the day.
And was the genlock card a Video Toaster by any chance?
https://en.wikipedia.org/wiki/Odra_(computer)
[1]: https://en.wikipedia.org/wiki/IBM_RPG
[2]: https://stackoverflow.com/questions/tagged/ibm-midrange